Ivan Kuznetsov » ruby

Increasing Ruby interpreter performance by adjusting garbage collector settings

Ivan Kuznetsov — Sun, 10 Oct 2010 23:10:12 +0000

According to Evan Weaver from Twitter it is possible for a typical production Rails app on Ruby 1.8 to recover 20% to 40% of user CPU by simply adjusting Ruby garbage collector settings. In August I set out on a quest to verify that statement on HeiaHeia servers. Results have really exceeded my expectations. Time to execute application tests locally decreased by 46%. On production servers CPU utilisation decreased by almost 40%.

But let’s start from the beginning. I should say right away that we at HeiaHeia are using Ruby Enterprise Edition, so I didn’t have to apply patches to Ruby source code that Evan is talking about in his post. Before starting to analyse GC current usage it will be useful to read a brilliant overview of Ruby garbage collection by Joe Damato. It’ll help to understand what’s going to happen next. It is also useful to read REE documentation on garbage collector performance tuning.

And as before any optimisation it is good to get the reference metrics, so that you know you actually improved something by changing settings, and didn’t make it worse. In this case I measured:

number of garbage collector calls when loading HeiaHeia-feed
local tests execution time (unit + functional)
application server CPU load and average response time

To measure number of garbage collector calls per one page I used Scrap – a nice tool by Chris Heald. Chris also describes the tuning process in great detail, so I’m not going to repeat it – just go and read his blog.

To measure local test execution time I just ran rake test 5 times and took the average of all runs.

To measure application server CPU load and average response time I used NewRelic tool (free version should be enough to do the measurement).

When I first loaded feed page with Scrap enabled I saw 36 GC cycles, and Ruby spend 1.12s in GC cycles (these figures are from the development server, so response time is big). After playing a bit with the settings and monitoring GC cycles number and unused heap after each allocation with Scrap, I ended up with the same settings as Twitter uses in production:

export RUBY_HEAP_MIN_SLOTS=500000
export RUBY_HEAP_SLOTS_INCREMENT=250000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=50000000

These settings reduced GC cycles number down to 7 (from 36), and Ruby spent now only 0.62s in GC instead of 1.12s when loading feed page (again, load times are bigger on our development server).

After introducing same settings on my local machine project tests took only 148s – down from 274 seconds before optimisation – a whopping 46% improvement.

We have multiple identical application servers, so I introduced the new settings only on one of the application servers, to compare the results on a live system (during low traffic hours). Here’s the picture from NewRelic:

Server        Instances        Apdex          Resp.time    Throughput    CPU          Memory
hh-app1     3 Instances         0.86      	407 ms       30 rpm      14 %	      349 MB
hh-app2     3 Instances         0.91   	        311 ms	     30 rpm       8 %	      413 MB

hh-app2 had optimised garbage collector. With the same throughput CPU load was only 8% vs 14% with non-optimised GC. However that improvement came at a cost of increased memory consumption (413M vs 349M). However response time and lower CPU load proved to be a lot more important than memory consumption, so I rolled out the new settings on all production servers.

Making nginx utilise GC settings when spawning Passenger instances is easy, and is well described by Chris. Here are instructions that work nice with server and Nginx setup on Ubuntu as I described in earlier posts.

Create /usr/local/bin/ruby-with-env file (as a root) that will set GC settings in the environment and then launch Ruby:

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1500000
export RUBY_HEAP_SLOTS_INCREMENT=500000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=50000000
exec "/usr/local/bin/ruby" "$@"

Make this file executable by all:

sudo chmod a+x /usr/local/bin/ruby-with-env

Now tell Passenger to use that file instead of launching Ruby directly – edit /opt/nginx/conf/nginx.conf and replace

passenger_ruby /usr/local/bin/ruby;

with

passenger_ruby /usr/local/bin/ruby-with-env;

Now restart Nginx – and you’ve got yourself a faster Ruby!

Running Rails applications using Nginx with Passenger on Ubuntu Server

Ivan Kuznetsov — Fri, 14 May 2010 10:40:50 +0000

If you’re planning to run Rails applications on Nginx using Phusion Passenger, and do it on Ubuntu Linux, here’s what needs to be done.

Even though there’s Ubuntu nginx package available (which works perfectly when you’re running PHP apps using FCGI), if you want to take into use Phusion Passenger, you’ll need to recompile Nginx from sources.

Instructions below were verified on Ubuntu 10.04 (Lucid Lynx) Server Edition.

If you already have Nginx ubuntu package installed, uninstall it (NOTE! purge will delete all configuration files – so if you changed them – make a backup for future reference):

sudo aptitude purge nginx

Assuming you already have Rails stack installed, install Passenger gem:

sudo gem install passenger

At the time of this writing the latest version of Passenger is 2.2.11

Let’s check dependencies that Ubuntu nginx package has and install them before compilation:

aptitude show nginx | grep Depends

You’ll see something like:

Depends: libc6 (>= 2.4), libpcre3 (>= 7.7), libssl0.9.8 (>= 0.9.8k-1), zlib1g

Install build dependencies and start nginx module installation (it will offer you to recompile nginx)

sudo apt-get install libc6 libpcre3 libssl0.9.8 zlib1g
sudo /usr/local/bin/passenger-install-nginx-module

Choose option 1 (Yes: download, compile and install Nginx for me) unless you need special configuration parameters or need features not enabled by default (like SSL).

Further instructions assume that you also chose default installation directory /opt/nginx.

If you’ve purchased Passenger Enterprise Edition, don’t forget to register it:

sudo /usr/local/bin/passenger-make-enterprisey

Now add nginx init script (I just copied this from Ubuntu default nginx package):

sudo vim /etc/init.d/nginx

with the following content:

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/nginx/sbin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

. /lib/lsb/init-functions

test_nginx_config() {
  if nginx -t $DAEMON_OPTS
  then
    return 0
  else
    return $?
  fi
}

case "$1" in
  start)
        echo -n "Starting $DESC: "
        test_nginx_config
        start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
                --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
                --exec $DAEMON || true
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
                /var/run/$NAME.pid --exec $DAEMON || true
        sleep 1
        test_nginx_config
        start-stop-daemon --start --quiet --pidfile \
                /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;
  reload)
        echo -n "Reloading $DESC configuration: "
        test_nginx_config
        start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/$NAME.pid \
            --exec $DAEMON || true
        echo "$NAME."
        ;;
  configtest)
        echo -n "Testing $DESC configuration: "
        if test_nginx_config
        then
          echo "$NAME."
        else
          exit $?
        fi
        ;;
  status)
        status_of_proc -p /var/run/$NAME.pid "$DAEMON" nginx && exit 0 || exit $?
        ;;
  *)
        echo "Usage: $NAME {start|stop|restart|reload|force-reload|status|configtest}" >&2
        exit 1
        ;;
esac

exit 0

Now make this script executable add it to default run levels:

sudo chmod +x /etc/init.d/nginx
sudo /usr/sbin/update-rc.d -f nginx defaults

Edit nginx.conf to look like this:

user www-data;
worker_processes  4;

error_log  /opt/nginx/logs/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  8192;
    use epoll;
}

http {
    passenger_root /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11;
    passenger_ruby /usr/local/bin/ruby;

    include       /opt/nginx/conf/mime.types;

    # set a default type for the rare situation that
    # nothing matches from the mimie-type include
    default_type application/octet-stream;

    # This log format is compatible with any tool like awstats
    # that can parse standard apache logs.
    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"' ;

    access_log  /opt/nginx/logs/access.log;

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  0;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    include /opt/nginx/conf/sites-enabled/*;
}

Now create virtual hosts structure (so that it looks like the structure created by Ubuntu original nginx package):

sudo mkdir /opt/nginx/conf/sites-available
sudo mkdir /opt/nginx/conf/sites-enabled

Now it is time to configure your first rails site. Create virtual host configuration in sites-available.

sudo vim /opt/nginx/conf/sites-available/mysite.com

Content can be something like:

server {
        listen   80;
        server_name  www.mysite.com;

        access_log  /home/user/logs/www.mysite.com/access.log;
        root   /home/user/www.mysite.com/public;

        # serve static content directly
        location ~* \.(ico|jpg|gif|png|css|js|swf|html)$ {
          if (-f $request_filename) {
            expires max;
            break;
          }
        }

        passenger_enabled on;

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
          deny  all;
        }
}

Make a symlink in sites-enabled directory and restart nginx.

sudo ln -s /opt/nginx/conf/sites-available/mysite.com /opt/nginx/conf/sites-enabled/mysite.com
sudo /etc/init.d/nginx restart

Now open the browser and check that your site is working.

Setting up Ruby, Rails, Git and Redmine on Dreamhost

Ivan Kuznetsov — Thu, 30 Jul 2009 14:58:27 +0000

The task is to have:
- Redmine installation on redmine.mydomain.com
- Several Git repositories on git.mydomain.com with different access rights to each one

This proved to be a non-trivial task. There is a number of tutorials on the net, but none of them described the full solution. So after getting it all to work, I decided to share all the tips and tricks. Feel free to comment, if you will find problems with the following set of instructions.
SSH to redmine.mydomain.com as a user that will be running Redmine (in the following examples it will be ‘redmine_user’).
First, you need to compile openssl – it will be required for curl, git and redmine.

mkdir ~/tmp
cd ~/tmp
wget http://www.openssl.org/source/openssl-0.9.8k.tar.gz
tar xzvf openssl-0.9.8k.tar.gz
cd openssl-0.9.8k
./config shared zlib --prefix=$HOME/.packages
make
make install

Let’s tell the world that we keep binaries and libraries also in the local directory. Edit ~/.bashrc (it is used by all non-login shells):

export TZ='Europe/Helsinki'
export PATH="$HOME/.packages/bin:$PATH"
export LD_LIBRARY_PATH="$HOME/.packages/lib"
export GEM_HOME="$HOME/.gems"
export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8"
export PATH="$HOME/.packages/bin:$HOME/.gems/bin:$PATH"
export RUBYLIB="$HOME/.packages/lib:$RUBYLIB"
export LD_LIBRARY_PATH="$HOME/.packages/lib"

# this ensures our gem install processes don't get killed by the DreamHost police
alias gem="nice -n19 ~/.packages/bin/gem"

(You can skip TZ – it is just usefule to have correct time set for your environment. Use tzselect to find out correct TZ string for your region)

And edit ~/.bash_profile (it is used by login shells):

umask 002
PS1='[\h:$PWD]$ '
alias ll="ls -l"
EDITOR="/usr/bin/vim"
. .bashrc

Now let’s apply the changes in active shell:

cd ~ : . .bash_profile

Then you need to compile curl, to be able to compile git with curl and execute clone commands on your server.

cd ~/tmp
wget http://curl.haxx.se/download/curl-7.19.5.tar.gz
tar xzvf curl-7.19.5.tar.gz
cd curl-7.19.5
./configure --prefix=$HOME/.packages --with-ssl=$HOME/.packages
make
make install

Now get and compile Git. If you are not using Dreamhost PS, you might want to compile it with NO_MMPAP=1, to reduce
probability of git process getting killed by Dreamhost police robots due to extensive memory

cd ~/tmp
wget http://www.kernel.org/pub/software/scm/git/git-1.6.4.tar.gz
tar xzvf git-1.6.4.tar.gz
cd git-1.6.4
./configure --prefix=$HOME/.packages --with-curl=$HOME/.packages
make
make install

Let’s start with ruby and rails related stuff. First readline library is needed, for script/console to work.

cd ~/tmp
wget http://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz
tar xzvf readline-5.2.tar.gz
cd readline-5.2
./configure --prefix=$HOME/.packages
make
make install

Install ruby:

cd ~/tmp
wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
tar zxvf ruby-1.8.7-p72.tar.gz
cd ruby-1.8.7-p72
./configure --prefix=$HOME/.packages --with-openssl-dir=$HOME/.packages --with-readline-dir=$HOME/.packages
make
make install

Install rubygems:

cd ~/tmp
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar zxvf rubygems-1.3.5.tgz
cd rubygems-1.3.5
ruby setup.rb config --prefix=$HOME/.packages
ruby setup.rb setup
ruby setup.rb install

Now you can install all required gems and freeze them if necessary.

Installing RedMine

Refer to http://wiki.dreamhost.com/Redmine for details.

But there’s one trick – if you want RedMine to use just compiled version of Git – edit in lib/redmine/scm/adapters/git_adapter.rb:

# Git executable name
GIT_BIN = "/home/username/.packages/bin/git"

If this is not done, default Dreamhost git will be used (which is too old at the moment 1.4.4.4) and Git repository browsing will not work in RedMine
(see http://groups.google.com/group/phusion-passenger/browse_thread/thread/5080d7c7cfbcf20e).

Setting up Git repository

Refer to http://wiki.dreamhost.com/Git for details.

Couple of tricks here. There are bugs in WebDAV functionality in Ubuntu – so if you’re using it – launch Nautilus, use “File->Connect to server…” menu from there (not from the system menu), don’t enter user name in the dialog – leave it empty and enter it when you’re requested username and password in the next dialog.

When setting up WebDAV access rights, give access to user “redmine”.

Setting Git repository copy for Redmine

Create ~/.netrc for your redmine user on Dreamhost

Insert the following line in that file

machine git.mydomain.com login redmine password [redmine_password]

where redmine_password is the password you gave to user redmine in the previous step.
.netrc will ensure that password is not asked when git is accessing the repository.
You don’t need to bother about creating .netrc if you have a public repository (or at least available for cloning without password).

Create a local copy of the repository

For RedMine to be able to display Git repository, it needs to have a local clone of the repository.

mkdir ~/git_project_clones
cd git_project_clones
git clone http://git.mydomain.com/repository_name

Now you should have a local copy that can be used from RedMine.
All you have left to do is to set up regular pulls from the master repository to this local copy. Use command

crontab -e

Add the following line to pull latest change into local copy every 5 minutes:

*/5 * * * * cd /home/redmine_user/git_project_clones/repository_name && /home/redmine_user/.packages/bin/git pull

Make sure that you specify full path to git – otherwise it will execute Dreamhost default git 1.4.4.4 and command will fail with “refusing to create funny ref ‘remotes/origin/*’ locally” error.

Save and exit cron editor.

(You might also want to check Redmine own wiki: http://www.redmine.org/wiki/redmine/HowTo_keep_in_sync_your_git_repository_for_redmine)

Set repository in the settings of your project in RedMine: /home/redmine_user/git_project_clones/repository_name/.git

Now you need to manually update Git repository changesets in RedMine:

cd ~/mydomain.com
script/runner "Repository.fetch_changesets" -e production

And set a hook to your repository to do this every time repository is updated – edit ~/git_project_clones/repository_name/.git/hooks/post-update file and add the following command ther:

cd /home/redmine_user/mydomain.com && script/runner "Repository.fetch_changesets" -e production

When writing this blogpost I found a lot of useful information on Dreamhost own wiki as well on these blog posts:
http://www.wavethenavel.com/2008/09/08/bootstrapping-a-dreamhost-account-for-rails-and-git/
http://juliobiason.net/2008/05/19/git-repositories-on-dreamhost-via-ssh/
http://www.simonecarletti.com/blog/2009/07/configuring-git-repository-with-redmine/