REE segfaults when Rails application has too many localisation files

We ran into an interesting problem – at some point of time our Rails application started to fail occaionally because of REE segfaults on startup. Even starting the console with ‘script/console production’ was occasionally failing with REE segfault. Application was growing, new features were added and segfaults started happening more and more often. There was no one single place where crashes occurred, so there was no clear understanding how to tackle this problem.

Examples of crashes we observed:

/vendor/rails/actionpack/lib/action_controller/routing/route.rb:205):2:
   [BUG] Segmentation fault
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/yaml.rb:133: 
   [BUG] Segmentation fault
/vendor/rails/activesupport/lib/active_support/vendor/i18n-0.3.7/i18n/
   backend/base.rb:257: [BUG] Segmentation fault
/vendor/rails/actionpack/lib/action_view/template.rb:226: [BUG] Segmentation fault
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/gems/1.8/gems/pauldix-sax-machine-0.0.14/
   lib/sax-machine/sax_document.rb:30: [BUG] Segmentation fault
/vendor/rails/activesupport/lib/active_support/memoizable.rb:32: [BUG] Segmentation fault

After banging my head against the wall for a week I found a solution (even two) and what might seem to be a likely reason for the segfaults. Two “suspects” – lack of available memory and incorrect version of libxml were ruled out. What seems to be the actual reason is the total size of the localisation files in config/locales loaded upon startup:

$ du -shb config/locales
1665858    config/locales
$ cd config/locales
$ find . -type f | wc -l
805

So ~1.6Mb in 805 files give occasional segfaults. Adding 200Kb of localisation files more started giving 100% segfaults on script/console startup.

Now I’ve found two workarounds for this problem.

1. Recompile REE with –no-tcmalloc flag

./ruby-enterprise-1.8.7-2011.03/installer --no-tcmalloc

Note that on 64-bit platforms tcmalloc is disabled by default.

2. Enable large pages feature in tcmalloc

This is described in tcmalloc documentation as: “Internally, tcmalloc divides its memory into “pages.”  The default page size is chosen to minimize memory use by reducing fragmentation. The cost is that keeping track of these pages can cost tcmalloc time. We’ve added a new, experimental flag to tcmalloc that enables a larger page size.  In general, this will increase the memory needs of applications using tcmalloc.  However, in many cases it will speed up the applications as well, particularly if they allocate and free a lot of memory.  We’ve seen average speedups of 3-5% on Google applications.”

There’s a warning – “this feature is still very experimental”, but it works to solve the problem with too many localisation files.

To compile REE with tcmalloc with large pages enables I just edited ruby-enterprise-1.8.7-2011.03/source/distro/google-perftools-1.7/src/common.h – replaced

#if defined(TCMALLOC_LARGE_PAGES)
static const size_t kPageShift  = 15;
static const size_t kNumClasses = 95;
static const size_t kMaxThreadCacheSize = 4 << 20;
#else
static const size_t kPageShift  = 12;
static const size_t kNumClasses = 61;
static const size_t kMaxThreadCacheSize = 2 << 20;
#endif

with

static const size_t kPageShift  = 15;
static const size_t kNumClasses = 95;
static const size_t kMaxThreadCacheSize = 4 << 20;

On production servers I opted for no tcmalloc for now – but I hope there’ll be a better way to deal with this issue soon.

Share

Pitfalls of Rails fragment caching with memcached

Fragment caching is a powerful technique for improving performance of your web application. Rails site describes in detail how to apply this technique.

Rails are providing developers with really excellent abstractions, but it’s always good to know what’s under the hood and how it all works.

There are a few things that might potentially cause bugs in your code, or waste your time (speaking from my own experience). So here goes:

1. Beware of globally keyed fragments

Let’s take example from Rails tutorial:

<% cache do %>
  All available products:
  <% Product.all.each do |p| %>
    <%= link_to p.name, product_url(p) %>
  <% end %>
<% end %>

Now if you need to deal with a multi-language site you might want to make cache fragment language dependent. What might seem a convenient solution:

<%- cache([user.locale.to_s]) do -%>

will turn into a source of very interesting problems. While calling the cache method without parameters will automatically create a controller/action specific cache key, calling it with a key will make this fragment a globally keyed fragment. Cache key in the first case is going to look like “views/localhost:3000/controller-name”, and in the other case “views/en” – this is not as unique identifier any more.

While automatic cache key naming provided by rails is very convenient, it is very easy to run into a problem with duplicate cache key names used in different places.

2. Another pitfall of automatic cache key naming is that you shall never assume that when creating a cache with global key you can later find it using e.g. telnet interface to memcache. Example – add

<%- cache('unique_cache_key') do -%>
<%- end -%>

in your view and then try to read directly from memcache:

$ telnet localhost 11211
GET unique_cache_key
END

At the same time

GET views/unique_cache_key

will work. It’s easy to make this mistake trying to check or delete cache keys directly from memcache when using Rails cache methods.

3. delete_matched is not supported by memcached (see rails/activesupport/lib/active_support/cache/mem_cache_store.rb)

In practice that means that if you’re using memcached as Rails cache engine and trying to delete or expire fragment cache using standard Rails methods and regexp – you’ll fail.

expire_fragment(/base\/xyz.*/)

will fail miserably. Ideal solution is not to use explicit cache expiration, but rather create cache keys in such a way that doesn’t require expiration. Alternatively it’s possible to use extensions implementing delete_matched for memcached (haven’t tried it myself though).

Tip: one very useful tool for checking memcached is peep by Evan Weaver – allows you to peek into the cache and see what’s really cached and how it is used.

Share

Notes from Gothenburg – Nordic Ruby 2011 conference

Here are my notes from Nordic Ruby conference in Göteborg, Sweden.

I’d like to say big thanks to the organisers of the conference (especially CJ @cjkihlbom) – everything went really smooth, even though there’s been 150 people attending this year compared to 90 last year.
Some points that I’d really like to highlight are:

  • a lot of time to meet people and discuss: 30 minutes talks followed by 30 minutes breaks, no q&a – those who had questions had an opportunity to talk to the speakers during the breaks
  • venue was great (of course, the boat 🙂 – there was enough space for everyone to move around, but at the same time it was compact enough not to get lost also everyone had an opportunity to have lunch and dinner together
  • “job board” a huge white board where anyone can post information about open positions in their companies – it got filled withing firts few hours – job market is really hot
  • lightning talks that any participant can give – 5 minute talks in the end of the day – it was really great
  • real coffee 🙂 espresso, latte, cappuccino, americano – you name it – professional baristas were at your service
  • 5K Nordic Ruby run organised on the second day’s morning

Continue reading “Notes from Gothenburg – Nordic Ruby 2011 conference”

Share

Call to undefined function: imagecreatefromjpeg()

While installing new Joomla modules I came across this PHP error (yep, still have to deal with PHP occasionally). I had PHP compiled from source on Ubuntu 10.04 as per earlier instructions. Quick check of phpinfo() indicated that while gd module was compiled in, it didn’t have JPEG support:

GD Support         enabled
GD Version         bundled (2.0.34 compatible)
GIF Read Support   enabled
GIF Create Support enabled
PNG Support        enabled
WBMP Support       enabled
XBM Support        enabled

Making sure that JPEG libraries are installed

sudo aptitude install libjpeg libjpeg-dev

and reconfiguring PHP with –with-jpeg-dir flag (the rest of the compilation process remains the same as here)

./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib 
--enable-mbstring --with-openssl --with-mysql --with-mysql-sock 
--with-gd --without-sqlite --disable-pdo --with-jpeg-dir=/usr/lib

and then restarting nginx

sudo /etc/init.d/nginx restart

helped to solve the problem.

Share

Setting up your own git server on Ubuntu

This will create a new user ‘gitosis’ and prepare a structure for repositories in /srv/gitosis. Now let’s initialize a gitosis-admin repo – it is used for managing repositories and access

Of course there’s always an option to use github. And if you’re working on an open source project, or want to concentrate on coding and not system administration, github is a lot better option than setting up and managing your own git server (I’ve been so impressed by @defunkt‘s presentation on #frozenrails, that started recommending github to everyone 🙂 But if you already pay for a virtual machine somewhere (like Linode), then setting up your own git server might be a viable option, especially that it is sooo easy.

The following instructions have been verified on Ubuntu 10.04 Lucid Lynx, but should work at least on Ubuntu 9.04 and 9.10 just as well.

Continue reading “Setting up your own git server on Ubuntu”

Share

Git – revert or amend last commit

Since we moved from SVN to git in HeiaHeia I had to revert or amend changes I accidentally committed or committed and pushed to git repository. This is not the most common operation, so I have to browse the documentation every time I do that. This is more of a memo to myself, which hopefully will be useful to others too.

Committed some changes, didn’t push them, and need to amend the commit:

git commit --amend -a -m "Commit message"

Committed some changes, pushed them, and need to amend the commit, do the revert operation instead, since someone might’ve already used your changes.

Committed some changes, didn’t push them, and need to undo the commit:

git reset --hard HEAD^

This will just toss away the last commit completely.

Commited some changes, pushed them, and need to undo the commit:

git revert HEAD

This will automatically create a new commit, reverting the changes from the previous

Share

Agile distributed team – using chat to run scrum meetings

The current development team that I’m working in is really small – just 2 software gurus, a product owner and me as a scum-master/system admninistrator/part-time developer/architect.

Our team is distributed to the extent that sometimes all four of us are located in different places during our meetings – but time zones difference is in most of the cases withing 1-2 hours. Up until recently some of us didn’t have a permanent office and had to participate in daily scrum meetings, and sprint planning/reviews from public open spaces. Continue reading “Agile distributed team – using chat to run scrum meetings”

Share

Choosing Mobile Development Platform

mobileplatformThere’s been a lot of heated discussions in the blogosphere in the past month about mobile platforms from independent developer perspective. Which platform to choose, if you want to develop cool applications, reach a lot of users and maximize your revenues?

I previously wrote on this subject a year ago, when Android was announced, and three years ago, when I was really disappointed by a pretty much dead S60 applications market.

This time it started with a great presentation by Teemu Kurppa (a mastermind behind mobile Jaiku) at MobileDevCamp Helsinki – “Platform = Stage. How to choose a mobile development platform?“. It is a must see for every mobile developer.

Continue reading “Choosing Mobile Development Platform”

Share

Introducing Moozement

moozementLast year I got involved in the development of a new social network – Moozement. There are plenty of social networks out there, there are even white label social networks. So why create another one?

Jyri Engeström wrote some time ago about the case for object-centered sociality: “‘social networking’ makes little sense if we leave out the objects that mediate the ties between people”. I could not agree with him more. The glue of each community is something that unites them – common interest, social object. When you join new social network, you typically start by building your social graph – re-establishing links to the real people you know, checking if they have already registered, inviting those whom you would like to see in the new environment. But there must be something beyond the initial phase of building the social graph. And this is the problem that haunts giants like Facebook and MySpace. You cannot possibly have common interest with everyone, and you don’t want to share the same things with everyone.

Continue reading “Introducing Moozement”

Share

Nokia Beta Labs – Tommi Vilkamo

Nokia - Connecting PeopleConratulations Tommi, congratulations Nokia! It is really a lucky break for Nokia that someone with such a great track record of blogging and openly talking to community of Nokia users will head Beta Labs.

As Stephen Johnston said “Plenty of improvement ideas are in the pipeline, and the key one for me will be to build up a sense of community of Nokia early adopters and use them as lead innovators to help us know what we should be working on next.”

From my point of view the most important change would be that of the spirit of development in Nokia, so that beta culture really grows roots, and more interesting and innovative projects have a chance to emerge from our own developers and see the light of day.

William L.McKnight, 3M chairman of the board, formulated management principles already in 1948 where he encouraged 3M management to “delegate responsibility and encourage men and women to exercise their initiative.”

This is one principle I see us in Nokia adopting this very moment.

Share