Examples of crashes we observed:

/vendor/rails/actionpack/lib/action_controller/routing/route.rb:205):2:
   [BUG] Segmentation fault
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/yaml.rb:133:
   [BUG] Segmentation fault
/vendor/rails/activesupport/lib/active_support/vendor/i18n-0.3.7/i18n/
   backend/base.rb:257: [BUG] Segmentation fault
/vendor/rails/actionpack/lib/action_view/template.rb:226: [BUG] Segmentation fault
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/gems/1.8/gems/pauldix-sax-machine-0.0.14/
   lib/sax-machine/sax_document.rb:30: [BUG] Segmentation fault
/vendor/rails/activesupport/lib/active_support/memoizable.rb:32: [BUG] Segmentation fault

After banging my head against the wall for a week I found a solution (even two) and what might seem to be a likely reason for the segfaults. Two “suspects” – lack of available memory and incorrect version of libxml were ruled out. What seems to be the actual reason is the total size of the localisation files in config/locales loaded upon startup:

$ du -shb config/locales
1665858    config/locales

$ cd config/locales
$ find . -type f | wc -l
805

So ~1.6Mb in 805 files give occasional segfaults. Adding 200Kb of localisation files more started giving 100% segfaults on script/console startup.

Now I’ve found two workarounds for this problem.

1. Recompile REE with –no-tcmalloc flag

./ruby-enterprise-1.8.7-2011.03/installer --no-tcmalloc

Note that on 64-bit platforms tcmalloc is disabled by default.

2. Enable large pages feature in tcmalloc

This is described in tcmalloc documentation as: “Internally, tcmalloc divides its memory into “pages.”  The default page size is chosen to minimize memory use by reducing fragmentation. The cost is that keeping track of these pages can cost tcmalloc time. We’ve added a new, experimental flag to tcmalloc that enables a larger page size.  In general, this will increase the memory needs of applications using tcmalloc.  However, in many cases it will speed up the applications as well, particularly if they allocate and free a lot of memory.  We’ve seen average speedups of 3-5% on Google applications.”

There’s a warning – “this feature is still very experimental”, but it works to solve the problem with too many localisation files.

To compile REE with tcmalloc with large pages enables I just edited ruby-enterprise-1.8.7-2011.03/source/distro/google-perftools-1.7/src/common.h – replaced

#if defined(TCMALLOC_LARGE_PAGES)
static const size_t kPageShift  = 15;
static const size_t kNumClasses = 95;
static const size_t kMaxThreadCacheSize = 4 << 20;
#else
static const size_t kPageShift  = 12;
static const size_t kNumClasses = 61;
static const size_t kMaxThreadCacheSize = 2 << 20;
#endif

with

static const size_t kPageShift  = 15;
static const size_t kNumClasses = 95;
static const size_t kMaxThreadCacheSize = 4 << 20;

On production servers I opted for no tcmalloc for now – but I hope there’ll be a better way to deal with this issue soon.

Rails are providing developers with really excellent abstractions, but it’s always good to know what’s under the hood and how it all works.

There are a few things that might potentially cause bugs in your code, or waste your time (speaking from my own experience). So here goes:

1. Beware of globally keyed fragments

Let’s take example from Rails tutorial:

<% cache do %>
  All available products:
  <% Product.all.each do |p| %>
    <%= link_to p.name, product_url(p) %>
  <% end %>
<% end %>

Now if you need to deal with a multi-language site you might want to make cache fragment language dependent. What might seem a convenient solution:

<%- cache([user.locale.to_s]) do -%>

will turn into a source of very interesting problems. While calling the cache method without parameters will automatically create a controller/action specific cache key, calling it with a key will make this fragment a globally keyed fragment. Cache key in the first case is going to look like “views/localhost:3000/controller-name”, and in the other case “views/en” – this is not as unique identifier any more.

While automatic cache key naming provided by rails is very convenient, it is very easy to run into a problem with duplicate cache key names used in different places.

2. Another pitfall of automatic cache key naming is that you shall never assume that when creating a cache with global key you can later find it using e.g. telnet interface to memcache. Example – add

<%- cache('unique_cache_key') do -%>
<%- end -%>

in your view and then try to read directly from memcache:

$ telnet localhost 11211
GET unique_cache_key
END

At the same time

GET views/unique_cache_key

will work. It’s easy to make this mistake trying to check or delete cache keys directly from memcache when using Rails cache methods.

3. delete_matched is not supported by memcached (see rails/activesupport/lib/active_support/cache/mem_cache_store.rb)

In practice that means that if you’re using memcached as Rails cache engine and trying to delete or expire fragment cache using standard Rails methods and regexp – you’ll fail.

expire_fragment(/base\/xyz.*/)

will fail miserably. Ideal solution is not to use explicit cache expiration, but rather create cache keys in such a way that doesn’t require expiration. Alternatively it’s possible to use extensions implementing delete_matched for memcached (haven’t tried it myself though).

Tip: one very useful tool for checking memcached is peep by Evan Weaver – allows you to peek into the cache and see what’s really cached and how it is used.

I’d like to say big thanks to the organisers of the conference (especially CJ @cjkihlbom) – everything went really smooth, even though there’s been 150 people attending this year compared to 90 last year.
Some points that I’d really like to highlight are:

• a lot of time to meet people and discuss: 30 minutes talks followed by 30 minutes breaks, no q&a – those who had questions had an opportunity to talk to the speakers during the breaks
• venue was great (of course, the boat – there was enough space for everyone to move around, but at the same time it was compact enough not to get lost also everyone had an opportunity to have lunch and dinner together
• “job board” a huge white board where anyone can post information about open positions in their companies – it got filled withing firts few hours – job market is really hot
• lightning talks that any participant can give – 5 minute talks in the end of the day – it was really great
• real coffee espresso, latte, cappuccino, americano – you name it – professional baristas were at your service
• 5K Nordic Ruby run organised on the second day’s morning

Day 0

The conference started with a party on Göthenborg ship. It was a great way to get to know the speakers and participants in an informal setting.

Day 1

“GitHub Flavored Ruby” was by Tom Preston-Werner (@mojombo) from GitHub.

Tom had a hard task of waking up the auidence after yesterday’s pre-conference party, and he succeeded in this task. Tom explained what is a README driven development that he and some of his colleagues in GitHub are using – a golden mean between waterfall and cowboy coding.
Start a new project with writing a README file, where basic design principles and conventions will be described, and don’t forget to update it as the architecture changes.

Another interesting topic was TomDoc – code level documentation – a better alternative than RDoc/Yard, designed for hackers/humans, not for machines to read.

Modularisation was yet another subject – Tom explained how at GitHub they try to split the code into independent modules to reduce complexity of the system.
Grit is a good example of that – a module for Ruby-Git interaction was written even before the rails code for GitHub.

Other examples of modles created in GitHub (and open-sourced) are: resque, jekyll, proxymachine and chimney, albino, gollum, bart, camo, failbot, giraffe, nodeload (in node.js), ernie (in erlang), stratocaster (for event feed), ghromosome (which is apparently not used anymore, but worth mentioning just because of the fun name).

And naturally for versioning of the components Tom proposes to use Semantic versioning.

A few notes from the after-talk discussion with Tom and others:
- Github uses git powered deploys – instead of copying the new release from git and then changing symlink to current release, they use single directory and do direct git updates – rollbacks are also easy to do using git fucntionality
- it’s possible to deploy Github from Campfire by giving deploy command to a robot
- to avoid downtime – build_cache is executed on a local machine before deploy and pushed to the servers during deploy
- GitHub is still running on Rails 2.3, Tom’s estimate for migration to Rails 3 is ~2 weeks of “stop commits and migrate” (the estimate seems to be pretty consistent – I’ve been talking to several people about it)
- experience from a fairly large web-app developers: MySQL to Postgres migration took 3 weeks

“API Design Matters” was by Anthony Eden (@aeden) from DNSimple

Central points from Anthony’s presentation:
- APIs outlive their implementation (“Think about the children” when designing APIs
- Design APIs so that they are easy to extend
- 5 Cs: consistent, clear, convenient, concise and complete
- The best API is no API
- If you’re a software developer – you are and API designer
- And use semantic versioning that Tom was talking about

Reading recommendations:
- Jasmine Blanchette “The little manual of API design”
- Joshua Bloch “How to design a good API and why it matters”

“Bridging the gap – Using JavaScript in Rails to write DRY rich client applications” by Thorben Schröder (@walski), KOPFMaschine

An interesting talk about using Java Script from Ruby code and the other way around.
Implementing validations on the client side reduces amount of round-trips and speeds up user interaction, but since smart people can hack JavaScript and bypass client-side validations, also server side validations are necessary. To avoid writing validations twice use The Ruby Racer,  CommonJS and CommonJS models to DRY your code and write validations only once and use them from both Ruby and JS code. Pretty smart, huh?

“Limited Red Society” by Joseph Wilk (@josephwilk)

Slides: http://www.slideshare.net/josephwilk/the-limited-red-society-8367200

Kanban is limiting number of tasks the team has in progress at any given moment. Joseph described in his talk how to apply this methodology to testing.

It’s important to have metrics for your testing and it’s important to visualize them. How much time do you spend in “red” when your tests are failing, which modules in your code are changed most often, are they the same that usually fail tests, are there any oscillating test cases that pass then fail then pass again?

One thing to be noted – some metrics have limited lifespan – measure what you’re interested in and then throw them away.

Reading recommendations:
“Refactoring in Ruby”
“Refactoring. Ruby Edition” Kent Beck

“Must It Always Be About Sex?” by Joshua Wehner (@jaw6)

Links from the talk: http://pinboard.in/u:jaw6/t:nr2011/

Minorities and women are under-represented in our community. Black man is more likely to be hit by lightning than become a computer science professor.
75% of women engineers leave profession because they are not socially accepted in the working culture. In the beginning of computer era programming was mostly a job for women.
Do you think that your application is mobile friendly if you have iPhone optimised version? Think again – the most popular phone with a web browser is Nokia 1100 – more than 250 million phones sold to date,
and that kind of devices are the only way how most people in Africa or India access the Internet.

What do we need to do to change current situation?

Just today I came across an article on the same subject: Two 11-year-old entrepreneurs learned the hard way what it’s like to be a minority in tech during Startup Weekend

“Infinite data – finite solutions” by Randall Thomas (@daksis) from Engine Yard

Quick dive into Bayesian statistics. That is the technology that helps to filter out spam from your inbox: “Keeps spam at Bayes”

Bayes + DAG (Directed Acyclic Graph) = BBN (Bayesian Belief Network)

Probably the most scientific talk of the conference and two great demos: reconstruction of Bach’s first invention using BBN (amazing stuff) and analysing whiskey properties

Book to read: “Robust Nonparametric Statistical Methods”
Site to check: Eliezer Yudkowsky
Tool to check: Weka – analysis tool – live example how to analyze properites of whiskey.

A great collection of links on Bayesian Networks:  https://gist.github.com/1031597

“Taking Back Education” by Joe O’Brien (@objo)

Education. We are in the top 5 to 2% of the developers by just being here – attending the conference.
Why education system sucks? There’s a clear need for apprenticeship institute.

Harward Business Review research indicated that team IQ depends on diversity, not on IQ of individual members.
Adding a women to the team cnsistently improves team performance (good reference for Diversity problems talk by Joshua)

In the end of the first day there was a series of “lightning talks”.

* Anthony Eden (@aeden) proposed to use DNS protocol for resolving dependencies. Definitely idea worth studying.
* Twitter will be native on all iOS devices – think about it, and what do you want to do with it.
* Stephen Sykes – obfuscated code for fun and ruby quiz – really, really fun – see the slides here: http://www.stephensykes.com/obfu_lightening.pdf
* Emily Bach (backconsulting.com) shared her experiences in teaching TDD
* Theo from Burt talked about his experiences and lessons learned from scaling:
- build two of everything from the start – will help with horizontal scaling later
- fail fast (Göthenburg ship made it to China and back to Sweden and sank a hundred meters from the home harbour, Vaasa is a lot better example – it sank right after it was deployed
- queues
* Tom from GitHub talked about Hubot – a Campfire robot they wrote in node.js and keep developing just for fun – absolutely amazing stuff, every team should do that
* David from Streamio (one of the sponsors of Nordic Ruby) did a “shameless self-promotion” of his company
* Alex who has been doing a great job filming the conference talked about making a movie – what it takes
* Antony Sastre – talked about photography – really inspiring, Antony was a web developer, then left the field and was doing photography for living for two years, now came back to web development
* Nikolai (@nikolayb) Is this a bad idea? Deploying always to production on a site with millions of page views per day, one branch always – test code on a live service? Somehow it works…
* Paul Wilson (@paulanthonywils) talked about Agile Delivery Network – doing better government IT, and here’s a bunch of links from his talk: https://gist.github.com/1035392
* Paul Campbell (@paulk) talked about whenever.js and made it public on GitHub in the end of his talk http://whatever-js.heroku.com/#1, https://github.com/paulca/whenever.js

Day one ended with a BBQ dinner at the River Cafe.

Day 2

Paul Campbell who ended Day 1 with his lightning talk also started Day 2 with a autobiographical talk “In search of me fein” (myself in Irish).

7 days in my life, zombie movies and finding inspiration
Technologist, developers, live in a bubble – we have no problems finding jobs, we can travel, we can choose where we live and work. It’s not like that outsie of our industry.

“Actors on stage” by Elise Huard (@elise_huard)

Slides: http://www.slideshare.net/ehuard/ruby-hollywood-nordic

We have multiple cores in our processors now – how do we benefit from it?
MRI is the only Ruby where parallel threads are executed time-scliced in one real thread.

Cellucloid – a concurrent object framework for Ruby.

Languages that implement actors: Erlang, Scala (Actors are in standard library), Closure (Actors called Agents)

Mats hates threads
Should there still be a better cuncurrency primitives in MRI? shall we create a branch of Ruby to implement it?

“Beyond Ruby” by Jakob Mattson from Burt (@jakobmattsson)

Slides: http://jakobmattsson.tumblr.com/post/6664704401/beyond-ruby

Bold statement: Ruby is not a dynamic language. Not everyting in Ruby is an object, despite what many books claim.
E.g. dot (.) or “or” (||) operators cannot be redefined. Or operator doesn’t evaluate second argument if the first one is true.
Ruby is not as moldable as it could be.
It would be cool to implement calling by name – passing expression along with the context to the callee and letting it decide whether to evaluate it or not.
Io language is probably 80% there. A new language needs to be invented. And coding it would not be that difficult – because if it is
truly dynamic, then most of it can be implemented in itself.

“Fewer constraints” by Ryan Smith (@ryandotsmith) from Heroku

Slides: http://dl.dropbox.com/u/1579953/concurrency_deck_with_notes.pdf

Continuing the theme that Elise started this morning. How to improve performance of your application by parallelizing.
Amdahl’s law – adding processors gives only marginal speed-up. “Even though we have 10 workers
available to us, we are only seeing a speedup factor of 5.2, with 20 workers it get only to 6.9″.

Real life example – improving queue performance. Using Fuzzy FIFO instead of traditional queues FIFO implementation.
Example with using spinlock in PostgreSQL to implement it. See queue_classic on github.

“Mountain Dew and My Trail of Tears” by Aaron Patterson (@tenderlove) from AT&T Interactive

Aaron discussed in depth how to work with legacy code.

- use “ruby -w” to see the warnings in your code, warnings can be used by developers to inform about upcoming deprecations
- Liskov substitution principle (LSP)
- Single responsibility principple
- Method extraction
- Object#extend
- Code seams
- sleep sort: http://dis.4chan.org/read/prog/1295544154

Recommended reading: Working effectibely with legacy code

“Legacy” by Chad Fowler (@chadfowler)

Chad started with playing a recording of Beethoven – this is legacy. Miro is legacy. Gaudi is legacy too, and the only reason to visit Barcelona.

Legacy is good.

Nobody will remember your work when you die. There are still systems in use written 25 years ago – why don’t we write that kind of code anymore?
Average lifespan of a business software is 5 years (figure is probably made up)

Recommended reading:

Michael Feathers “Your Code.. it’s Alive..”
Richard P. Gabriel “Design Beyond Human Abilities”

What kind of software lasts – either small (cell-size) or systems consisting of cell-sized software (like UNIX).

Force heterogenity – languages, OSes, frameworks – will change over time. If it’s hard to force heterogenity – do it all the time.

“Must.Try.Harder” by Keavy McMinn (@keavy)

An absolutely fascinating and inspirational talk from a woman who decided to train for an IronMan from pretty much zero.
She did it. A rare inspirational example of how to go outside your comfort zone and achieve a goal that looked impossible.

Other conference notes:

Fredrik Rubensson: http://www.highlevelbits.com/2011/06/nordic-ruby.html
Andreas Ronge / Jayway:  http://blog.jayway.com/2011/06/19/notes-and-thoughts-from-a-great-conference/

1. Rails.vim by Tim Pope

To install just copy autoload/rails.vim, plugin/rails.vim, and doc/rails.txt to corresponding directories in ~/.vim

Vim scripts section has a full description of the functionality. Some highlights:

• Easy navigation between models, controllers and views with :Rmodel, :Rview, :Rcontroller commands
• Syntax highlighting
• CTRL-X CTRL-U for autocompletion
• :Rtree for project tree (see item 2)

2. NERD Tree by Marty Grenfell

Another must-have. Provides you with an easy way to navigate your project tree. Rails.vim nicely integrates with this one.

3. Colour schemas

If you want to save your eyes – use dark background when developing. Personally I prefer one of the standard vim themes – Desert , but Ocean Deep is also very good.

Copy theme file to ~/.vim/colors and then use

:colorscheme oceandeep

command to apply it.

In case you didn’t know – Adobe doesn’t have stable Flash player version for 64-bit Linux. Adobe Labs offer preview release codenamed “Square” for 64-bit platforms. It can’t be installed via standard Ubuntu repositories, so get ready to get your hands dirty in the terminal.

To install 64-bit Flash player plugin do the following:

1. Download the latest version of the plugin at http://labs.adobe.com/downloads/flashplayer10_square.html (currently it is v.10.2 preview 3 from November 30th, 2010)
2. Go to your downloads directory and extract the plugin binary

tar xvzf flashplayer10_2_p3_64bit_linux_111710.tar.gz

3. Create a directory for browser plugins in your users home directory

mkdir -p ~/.mozilla/plugins

4. Move extracted in step 2 Flash player plugin binary to its new location

mv libflashplayer.so ~/.mozilla/plugins

5. Close all browser windows and restart the browser.

6. In Firefox or Chrome go to about:plugins to verify that there’s Shockwave Flash plugin available

But let’s start from the beginning. I should say right away that we at HeiaHeia are using Ruby Enterprise Edition, so I didn’t have to apply patches to Ruby source code that Evan is talking about in his post. Before starting to analyse GC current usage it will be useful to read a brilliant overview of Ruby garbage collection by Joe Damato. It’ll help to understand what’s going to happen next. It is also useful to read REE documentation on garbage collector performance tuning.

And as before any optimisation it is good to get the reference metrics, so that you know you actually improved something by changing settings, and didn’t make it worse. In this case I measured:

To measure number of garbage collector calls per one page I used Scrap – a nice tool by Chris Heald. Chris also describes the tuning process in great detail, so I’m not going to repeat it – just go and read his blog.

To measure local test execution time I just ran rake test 5 times and took the average of all runs.

To measure application server CPU load and average response time I used NewRelic tool (free version should be enough to do the measurement).

When I first loaded feed page with Scrap enabled I saw 36 GC cycles, and Ruby spend 1.12s in GC cycles (these figures are from the development server, so response time is big). After playing a bit with the settings and monitoring GC cycles number and unused heap after each allocation with Scrap, I ended up with the same settings as Twitter uses in production:

export RUBY_HEAP_MIN_SLOTS=500000
export RUBY_HEAP_SLOTS_INCREMENT=250000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=50000000

These settings reduced GC cycles number down to 7 (from 36), and Ruby spent now only 0.62s in GC instead of 1.12s when loading feed page (again, load times are bigger on our development server).

After introducing same settings on my local machine project tests took only 148s – down from 274 seconds before optimisation – a whopping 46% improvement.

We have multiple identical application servers, so I introduced the new settings only on one of the application servers, to compare the results on a live system (during low traffic hours). Here’s the picture from NewRelic:

Server        Instances        Apdex          Resp.time    Throughput    CPU          Memory
hh-app1     3 Instances         0.86      	407 ms       30 rpm      14 %	      349 MB
hh-app2     3 Instances         0.91   	        311 ms	     30 rpm       8 %	      413 MB

hh-app2 had optimised garbage collector. With the same throughput CPU load  was only 8% vs 14% with non-optimised GC. However that improvement came at a cost of increased memory consumption (413M vs 349M). However response time and lower CPU load proved to be a lot more important than memory consumption, so I rolled out the new settings on all production servers.

Making nginx utilise GC settings when spawning Passenger instances is easy, and is well described by Chris. Here are instructions that work nice with server and Nginx setup on Ubuntu as I described in earlier posts.

Create /usr/local/bin/ruby-with-env file (as a root) that will set GC settings in the environment and then launch Ruby:

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1500000
export RUBY_HEAP_SLOTS_INCREMENT=500000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=50000000
exec "/usr/local/bin/ruby" "$@"

sudo chmod a+x /usr/local/bin/ruby-with-env

passenger_ruby /usr/local/bin/ruby;

passenger_ruby /usr/local/bin/ruby-with-env;

GD Support         enabled
GD Version         bundled (2.0.34 compatible)
GIF Read Support   enabled
GIF Create Support enabled
PNG Support        enabled
WBMP Support       enabled
XBM Support        enabled

Making sure that JPEG libraries are installed

sudo aptitude install libjpeg libjpeg-dev

and reconfiguring PHP with –with-jpeg-dir flag (the rest of the compilation process remains the same as here)

./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib
--enable-mbstring --with-openssl --with-mysql --with-mysql-sock
--with-gd --without-sqlite --disable-pdo --with-jpeg-dir=/usr/lib

and then restarting nginx

sudo /etc/init.d/nginx restart

helped to solve the problem.

Even though there’s Ubuntu nginx package available (which works perfectly when you’re running PHP apps using FCGI), if you want to take into use Phusion Passenger, you’ll need to recompile Nginx from sources.

Instructions below were verified on Ubuntu 10.04 (Lucid Lynx) Server Edition.

If you already have Nginx ubuntu package installed, uninstall it (NOTE! purge will delete all configuration files – so if you changed them – make a backup for future reference):

sudo aptitude purge nginx

Assuming you already have Rails stack installed, install Passenger gem:

sudo gem install passenger

At the time of this writing the latest version of Passenger is 2.2.11

Let’s check dependencies that Ubuntu nginx package has and install them before compilation:

aptitude show nginx | grep Depends

Depends: libc6 (>= 2.4), libpcre3 (>= 7.7), libssl0.9.8 (>= 0.9.8k-1), zlib1g

sudo apt-get install libc6 libpcre3 libssl0.9.8 zlib1g
sudo /usr/local/bin/passenger-install-nginx-module

sudo /usr/local/bin/passenger-make-enterprisey

sudo vim /etc/init.d/nginx

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/nginx/sbin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

. /lib/lsb/init-functions

test_nginx_config() {
  if nginx -t $DAEMON_OPTS
  then
    return 0
  else
    return $?
  fi
}

case "$1" in
  start)
        echo -n "Starting $DESC: "
        test_nginx_config
        start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
                --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
                --exec $DAEMON || true
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
                /var/run/$NAME.pid --exec $DAEMON || true
        sleep 1
        test_nginx_config
        start-stop-daemon --start --quiet --pidfile \
                /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;
  reload)
        echo -n "Reloading $DESC configuration: "
        test_nginx_config
        start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/$NAME.pid \
            --exec $DAEMON || true
        echo "$NAME."
        ;;
  configtest)
        echo -n "Testing $DESC configuration: "
        if test_nginx_config
        then
          echo "$NAME."
        else
          exit $?
        fi
        ;;
  status)
        status_of_proc -p /var/run/$NAME.pid "$DAEMON" nginx && exit 0 || exit $?
        ;;
  *)
        echo "Usage: $NAME {start|stop|restart|reload|force-reload|status|configtest}" >&2
        exit 1
        ;;
esac

exit 0

sudo chmod +x /etc/init.d/nginx
sudo /usr/sbin/update-rc.d -f nginx defaults

user www-data;
worker_processes  4;

error_log  /opt/nginx/logs/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  8192;
    use epoll;
}

http {
    passenger_root /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.11;
    passenger_ruby /usr/local/bin/ruby;

    include       /opt/nginx/conf/mime.types;

    # set a default type for the rare situation that
    # nothing matches from the mimie-type include
    default_type application/octet-stream;

    # This log format is compatible with any tool like awstats
    # that can parse standard apache logs.
    log_format main '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"' ;

    access_log  /opt/nginx/logs/access.log;

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  0;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_proxied any;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    include /opt/nginx/conf/sites-enabled/*;
}

sudo mkdir /opt/nginx/conf/sites-available
sudo mkdir /opt/nginx/conf/sites-enabled

sudo vim /opt/nginx/conf/sites-available/mysite.com

server {
        listen   80;
        server_name  www.mysite.com;

        access_log  /home/user/logs/www.mysite.com/access.log;
        root   /home/user/www.mysite.com/public;

        # serve static content directly
        location ~* \.(ico|jpg|gif|png|css|js|swf|html)$ {
          if (-f $request_filename) {
            expires max;
            break;
          }
        }

        passenger_enabled on;

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
          deny  all;
        }
}

sudo ln -s /opt/nginx/conf/sites-available/mysite.com /opt/nginx/conf/sites-enabled/mysite.com
sudo /etc/init.d/nginx restart

I have faced this problem and found a relatively simple and robust solution. Here’s how I did it on Ubuntu 9.10 (Karmic Koala) and 10.04 (Lucid Lynx) server edition.

Solution was to replace default FastCGI implementation with PHP-FPM (FastCGI Process Manager). PHP-FPM is not supported in PHP out of the box – so if you use PHP 5.2.*, you’ll need to apply a patch and recompile PHP, and if you’re using PHP 5.3.* (at least in 5.3.2 PHP-FPM is not yet in the core) – you’ll need to check out PHP-FPM from PHP SVN.

Let’s start with uninstalling default Ubuntu php packages:

sudo apt-get remove php5*

Now we need to install dependencies. Note, that Ubuntu comes with a new autoconf tool version, which is not compatible with PHP, that’s why for successful compilation you need to temporarily install autoconf2.13 package.

sudo apt-get install libcurl4-openssl-dev libmcrypt-dev libxml2-dev libpng-dev 
autoconf2.13 libevent-dev libltdl-dev

Download latest stable PHP 5.2.13, Suhosin patch, PHP-FPM patch

cd ~/tmp
wget http://pl2.php.net/get/php-5.2.13.tar.gz/from/pl.php.net/mirror
wget http://download.suhosin.org/suhosin-patch-5.2.13-0.9.7.patch.gz
wget http://php-fpm.org/downloads/php-5.2.13-fpm-0.5.13.diff.gz
tar xvzf php-5.2.13.tar.gz
gunzip suhosin-patch-5.2.13-0.9.7.patch.gz
gunzip php-5.2.13-fpm-0.5.13.diff.gz
cd php-5.2.13
patch -p 1 -i ../php-5.2.13-fpm-0.5.13.diff
patch -p 1 -i ../suhosin-patch-5.2.13-0.9.7.patch
./buildconf --force
./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl
--with-mysql --with-mysql-sock --with-gd --without-sqlite --disable-pdo
make
make test
sudo make install

Alternatively you can download latest stable PHP 5.3.2, Suhosin patch, apply PHP-FPM patch. Note, that not all PHP based projects and plugins work correctly with new PHP 5.3 – it is not backwards compatible with PHP 5.2. I had troubles at least with some Joomla plugins and ZenCart.

cd ~/tmp
http://fi.php.net/get/php-5.3.2.tar.gz/from/this/mirror
wget http://download.suhosin.org/suhosin-patch-5.3.2-0.9.9.1.patch.gz
tar xvzf php-5.3.2.tar.gz
gunzip suhosin-patch-5.3.2-0.9.9.1.patch.gz
cd php-5.3.2
patch -p 1 -i ../suhosin-patch-5.3.2-0.9.9.1.patch
svn co http://svn.php.net/repository/php/php-src/trunk/sapi/fpm sapi/fpm
./buildconf --force
./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl
--with-mysql --with-mysql-sock --with-gd --without-sqlite --disable-pdo --disable-reflection
make
make test
sudo make install

Uninstall autoconf2.13 after compilation.

sudo apt-get remove autoconf2.13

Change user and group of php-fpm processes to user and group of your choice (e.g. www-data and www-data) – lines 63 and 66

sudo vim /usr/local/etc/php-fpm.conf

Edit PHP settings

sudo vim /etc/php5/cgi/php.ini (in Ubuntu 9.xx)

sudo vim /etc/php5/apache2/php.ini (in Ubuntu 10.04)

Set:

max_execution_time = 30
memory_limit = 128M
error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR
display_errors = Off
log_errors = On
error_log = /var/log/php.log
register_globals = Off

Now if you haven’t done so yet, install Nginx. Ubuntu 10.04 comes with the latest stable Nginx 0.7.65, so just do:

sudo apt-get install nginx

Now you can congifure your sites, e.g. for WordPress Nginx configuration can look like this:

server {
        listen   80;
        server_name  blog.mysite.com;

        access_log  /home/user/logs/blog.mysite.com/access.log;

        location / {
          root   /home/user/blog.mysite.com;
          index  index.php index.html index.htm;

          # this serves static files that exist without running other rewrite tests
          if (-f $request_filename) {
              expires 30d;
              break;
          }

          # this sends all non-existing file or directory requests to index.php
          if (!-e $request_filename) {
              rewrite ^(.+)$ /index.php?q=$1 last;
          }

        }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        location ~ \.php$ {
          include /etc/nginx/fastcgi_params;
          fastcgi_pass  127.0.0.1:9000;
          fastcgi_index index.php;
          fastcgi_param  SCRIPT_FILENAME  /home/user/blog.mysite.com/$fastcgi_script_name;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
          deny  all;
        }
}

Of course there’s always an option to use github. And if you’re working on an open source project, or want to concentrate on coding and not system administration, github is a lot better option than setting up and managing your own git server (I’ve been so impressed by @defunkt‘s presentation on #frozenrails, that started recommending github to everyone But if you already pay for a virtual machine somewhere (like Linode), then setting up your own git server might be a viable option, especially that it is sooo easy.

The following instructions have been verified on Ubuntu 10.04 Lucid Lynx, but should work at least on Ubuntu 9.04 and 9.10 just as well.

Let’s start with installing gitosis itself. Issue the following command on the server:

sudo apt-get install gitosis

This will create a new user ‘gitosis’ and prepare a structure for repositories in /srv/gitosis. Now let’s initialize a gitosis-admin repo – it is used for managing repositories and access rights.

sudo -H -u gitosis gitosis-init < ~/tmp/my_public.key

You need to have a public key for accessing it. If you don’t have one yet, you can use

ssh-keygen -t rsa

command on your local machine to generate public/private key pair. Copy public key to the server before initializing gitosis-admin repository.

Now with gitosis-admin repo initialized on the server – let’s clone it to the local computer.

git clone gitosis@myserver.com:gitosis-admin.git

If you see an error like:

Permission denied (publickey).
fatal: The remote end hung up unexpectedly

That is most likely because you restricted access to you server via ssh to only certain users, and gitosis is not one of them. Edit /etc/ssh/sshd_config, find AllowUsers line and add gitosis to the list.

Now that you have successfully cloned gitosis-admin repo to your local computer, you can add  new users and new repositories.

To add new repository, edit gitosis.conf and add lines like:

[group myrepo]
writable = myrepo
members = user@computer

After that commit and push the changes to gitosis-admin project:

git commit -a -m "Added myrepo repository"

Now you can clone this new repository to your local machine (note .git added to the name of the repository):

git clone gitosis@mygitserver.com:myrepo.git

To allow new user to access your repository, get this user’s public key, copy it to gitosis-admin/keydir as newuser@computer.pub, then edit gitosis.conf and add newuser@computer (without .pub) to the list of members:

[group myrepo]
writable = myrepo
members = user@computer newuser@computer

Now add new files and commit and push changes to git server (make sure you really add all files and don’t forget to push – these are quite common mistakes when adding new users).

git add .
git commit -m "Added user newuser"
git push

With a freshly cloned empty repository you’ll need to add you first files, do a commit and push origin master:

vim README.txt
git commit -a -m "Added readme"
git push origin master

Now you can git pull and git push as much as you like.