So how to do it using R and how to visualize it. Inspired by examples described in “Retention, Cohorts, and Visualizations” I came up with the following solution.

First, get the data in a suitable format, like this:

cohort  signed_up  active_m0  active_m1  active_m2
2011-10 12345      10432      8765       6754
2011-11 12345      10432      8765       6754
2011-12 12345      10432      8765       6754

Cohort here is in “YYYY-MM” format, signed_up is the number of users who have created accounts in the given month, active_m0 – number of users who have been active in the same month as they registered, active_m1 – number of users who have been active in the following month, and so forth. For newest cohorts you’ll be getting zeroes in some of active_mN columns, since there’s no data on them yet. This is taken into account in processing scripts.

require(plyr)

# Load SystematicInvestor's plot.table (https://github.com/systematicinvestor/SIT)
con = gzcon(url('http://www.systematicportfolio.com/sit.gz', 'rb'))
source(con)
close(con)

# Read the data
cohorts
# Let's convert absolute values to percentages (% of the registered users remaining active)
cohort_p as.numeric(df$active_m0/df$signed_up), as.numeric(df$active_m1/df$signed_up), as.numeric(df$active_m2/df$signed_up),
as.numeric(df$active_m3/df$signed_up), as.numeric(df$active_m4/df$signed_up), as.numeric(df$active_m5/df$signed_up),
as.numeric(df$active_m6/df$signed_up), as.numeric(df$active_m7/df$signed_up), as.numeric(df$active_m8/df$signed_up) ))

# Create a matrix
temp = as.matrix(cohort_p[,3:(length(cohort_p[1,])-1)])
colnames(temp) = paste('Month', 0:(length(temp[1,])-1), sep=' ')
rownames(temp) = as.vector(cohort_p$V1)

# Drop 0 values and format data
temp[] = plota.format(100 * as.numeric(temp), 0, '', '%')
temp[temp == " 0%"] # Plot cohort analysis table
plot.table(temp, smain='Cohort(users)', highlight = TRUE, colorbar = TRUE)

This code produces nice visualizations of the cohort analysis as a table:

I used articles “Visualizing Tables with plot.table“ and “Response to Flowingdata Challenge: Graphing obesity trends” as an inspiration for this R code.

If you want to get nice colours as in the example above, you’ll need to adjust rainbow interval for plot.table. I managed to do it by editing functions code directly from R environment:

plot.table.helper.color <- edit(plot.table.helper.color)

function
(
 temp # matrix to plot
){
 # convert temp to numerical matrix
 temp = matrix(as.double(gsub('[%,$]', '', temp)), nrow(temp), ncol(temp))

highlight = as.vector(temp)
 cols = rep(NA, len(highlight))
 ncols = len(highlight[!is.na(highlight)])
 cols[1:ncols] = rainbow(ncols, start = 0, end = 0.3)

o = sort.list(highlight, na.last = TRUE, decreasing = FALSE)
 o1 = sort.list(o, na.last = TRUE, decreasing = FALSE)
 highlight = matrix(cols[o1], nrow = nrow(temp))
 highlight[is.na(temp)] = NA
 return(highlight)
}

Adjust interval in line 11 to 0.5, 0.6 to get shades of blue.
plot.table.helper.colorbar <- edit(plot.table.helper.colorbar)

function
(
 plot.matrix # matrix to plot
)
{
 nr = nrow(plot.matrix) + 1
 nc = ncol(plot.matrix) + 1

c = nc
 r1 = 1
 r2 = nr

rect((2*(c - 1) + .5), -(r1 - .5), (2*c + .5), -(r2 + .5), col='white', border='white')
 rect((2*(c - 1) + .5), -(r1 - .5), (2*(c - 1) + .5), -(r2 + .5), col='black', border='black')

y1= c( -(r2) : -(r1) )

graphics::image(x = c( (2*(c - 1) + 1.5) : (2*c + 0.5) ),
 y = y1,
 z = t(matrix( y1 , ncol = 1)),
 col = t(matrix( rainbow(len( y1 ), start = 0.5, end = 0.6) , ncol = 1)),
 add = T)
}

Adjust interval in line 21 to 0.5, 0.6 to get shades of blue.

Now if you want to draw the cycle-like graph:

# make matrix shorter for the graph (limit to 0-6 months)
temp = as.matrix(cohort_p[,3:(length(cohort_p[1,])-1)])
temp temp[temp == "0"]
library(RColorBrewer)
colnames(temp) = paste('Month', 0:(length(temp[1,])-1), 'retention', sep=' ')
palplot(temp[,1],pch=19,xaxt="n",col=pal[1],type="o",ylim=c(0,as.numeric(max(temp[,-2],na.rm=T))),xlab="Cohort by Month",ylab="Retention",main="Retention by Cohort")

for(i in 2:length(colnames(temp))) {
 points(temp[,i],pch=19,xaxt="n",col=pal[i])
 lines(temp[,i],pch=19,xaxt="n",col=pal[i])
}

axis(1,at=1:length(cohort_p$cohort),labels=as.vector(cohort_p$cohort),cex.axis=0.75)
legend("bottomleft",legend=colnames(temp),col=pal,lty=1,pch=19,bty="n")
abline(h=(seq(0,1,0.1)), col="lightgray", lty="dotted")

This code produces nice visualizations of the cohort analysis as multicolour cycle graph:

After a short historical introduction to the philosophy of the  Amazon platform authors proceed to basics of the EC2 instance management (using Ubuntu as example) and describe how several real life web applications benefited from migration to Amazon infrastructure. High level architecture descriptions help to understand how all pieces of the Amazon platform come together – ELB, EC2, RDS, S3, SQS, SNS, etc.  Examples are provided in PHP, Ruby and Java.

Don’t expect any secret knowledge to be revealed in this book. A lot of intricate details and answers to the questions that you will face when planning a migration to AWS or designing architecture for a new web application are left out of this book. At the same time the book gives a fairly good overview of how to make your application to be scaleble and highly available on AWS and can serve as a good starting point in your AWS journey.

Some of the recommendations and descriptions given in the book are outright sloppy. A couple of examples: the book recommends “You can choose an availability zone, but that doesn’t matter very much because network latency between zones is comparable to that within a zone“. But tests indicate that Cross Availability Zones latency can be 6 times higher than inner zone latency. For a network intensive application, better keep your instances crowded in the same zone.

Another example where you might want to go run your own tests before making a decision or look for another opinion “A disk is always slower than memory. If you run your own MySQL using local disks, that’s slow as well. But using disk-based operations in RDS is just horrible. Minimizing disk usage means implementing proper indexes, something you always want to do.” A very strong yet vague statement on RDS performance, where I’d really love to see a comparison in performance of MySQL installation on EC2+EBS vs. RDS and a list of what kind of MySQL fine-tuning will and will not work on RDS.

Another annoying bit about the book is that authors keep promoting their commercial Android client for AWS monitoring throughout all chapters. As I see it – if there are ads, the book should be free.

Bottom line: I see only two reasons why you might want to buy and read this book – learning about the history of AWS and learning how five selected web services designed their architecture when migrating to AWS.

I got the data from our database, filtering results by country (Finnish users only) in a tab separated form with the first line as the header. Three columns contained date, count of sick days logged on that date and count of Finnish users in the service on that date.

date count(*) user_cnt
2011-01-01 123 12345
2011-01-02 456 67890
...

Below is R source code for plotting the heat map. I made some small changes to the original code:

• data normalization (line 9): this is specific to the data used in this example
• days of the week have to be 1..7, not 0..6 as returned by $wday (line 19): dat$weekday = as.numeric(format(as.POSIXlt(dat$date),”%u”))
• date format (line 31): week of year calculation required date conversion to POSIX dat$week <- as.numeric(format(as.POSIXlt(dat$date),”%W”))
• custom header for the legend (line 39): adding + labs(fill=”per user per day”) allows you to customize legend header

require(zoo)
require(ggplot2)
require(plyr)

dat<-read.csv("~/data/sick_days_per_day.txt",header=TRUE,sep="\t")</pre>
colnames(dat) <- c("date", "count", "user_cnt")

# normalize data by number of users on each date
dat$norm_count <- dat$count / dat$user_cnt

# facet by year ~ month, and each subgraph will show week-of-month versus weekday the year is simple
dat$year<-as.numeric(as.POSIXlt(dat$date)$year+1900)
dat$month<-as.numeric(as.POSIXlt(dat$date)$mon+1)

# turn months into ordered facors to control the appearance/ordering in the presentation
dat$monthf<-factor(dat$month,levels=as.character(1:12),labels=c("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"),ordered=TRUE)

# the day of week is again easily found
dat$weekday = as.numeric(format(as.POSIXlt(dat$date),"%u"))

# again turn into factors to control appearance/abbreviation and ordering
# I use the reverse function rev here to order the week top down in the graph
# you can cut it out to reverse week order
dat$weekdayf<-factor(dat$weekday,levels=rev(1:7),labels=rev(c("Mon","Tue","Wed","Thu","Fri","Sat","Sun")),ordered=TRUE)

# the monthweek part is a bit trickier - first a factor which cuts the data into month chunks
dat$yearmonth<-as.yearmon(dat$date)
dat$yearmonthf<-factor(dat$yearmonth)

# then find the "week of year" for each day
dat$week <- as.numeric(format(as.POSIXlt(dat$date),"%W"))

# and now for each monthblock we normalize the week to start at 1
dat<-ddply(dat,.(yearmonthf),transform,monthweek=1+week-min(week))

# Now for the plot
P<- ggplot(dat, aes(monthweek, weekdayf, fill = dat$norm_count)) +
 geom_tile(colour = "white") + facet_grid(year~monthf) + scale_fill_gradient(low="green", high="red") +
 opts(title = "Time-Series Calendar Heatmap - HeiaHeia.com sick days logged") + xlab("Week of Month") + ylab("") + labs(fill="per user per day")
P

Here are the results. Green indicates the healthiest days with lowest values of sick days logged per user, red indicates the worst days with highest values of sick days logged per user. It’s quite clear that there are seasonal peaks around February, and 2011 was a lot worse than 2012 (one should note that January-February of 2011 were exceptionally cold in Finland). It matches quite well with the coverage in the national press: Flu season reaching peak (Feb’2012), Employers grapple with sick leaves brought by flu wave (Feb’2012).

It’s interesting that there are less sick days logged on the weekends than on the work days, and traditional holiday month of July is the healthiest month of all.

(click to see full-sized image)

To get a more formal validation of the data logged by HeiaHeia users, I used Google Correlate lab tool to check that heat map results make sense. I uploaded sick days per user weekly time series and plotted a correlation with Google search queries for “kuumeen hoito” (treatment of fever in Finnish).

(click to see full-sized image)

Pearson Correlation Coefficient r between HeiaHeia sick days time series and Google search activity σ (both normalized so that mean is 0 and standard deviation is 1) is 0.8257 – this is a pretty good match.

The setup works on Ubuntu 10.04 Server LTS (scheduled end of life April 2015). Other components of the setup are Nginx as the web server, Phusion Passenger as application server.

I’m using this setup most often on Linode VPS, however none of the instructions are Linode specific.

Create non-root user with sudo rights

(Login as root via ssh)

Add a new user (this user will be an administrator of the server – will have the ability to log in via ssh and use sudo, it is different from user account used for the web application):

adduser username

Add this user to sudoers list:

visudo

in the editor add

username ALL=(ALL) ALL

Securing sshd

Edit /etc/ssh/sshd_config:

Change the following lines (note, you’re changing default port for ssh connections – you’ll need to take that into account when connecting to the server later):

Port 6668
PermitRootLogin no
X11Forwarding no
Protocol 2
PasswordAuthentication no
UsePAM no

Add the following lines (here username is the name of the administrator user that was created earlier):

UseDNS no 
AllowUsers username

Restart ssh daemon:

/etc/init.d/ssh restart

Add authentication keys for new administrator user

Note, that ‘user’ here is the name of the administrator user that was created earlier.

mkdir /home/user/.ssh 
vim /home/user/.ssh/authorized_keys

When editing authorized_keys file, insert a public of that user (e.g. your personal public key, if you want to make yourself an administrator).

chown -R user:user /home/user/.ssh 
chmod 600 /home/user/.ssh/authorized_keys

Now you can log out as root and log in as a sudo user you just created. NOTE! You won’t be able to log in as rot via ssh anymore.

Set up iptables firewall

Save existing rules

sudo sh -c 'iptables-save > /etc/iptables.up.rules'

Create test rules

sudo vim /etc/iptables.test.rules

Here’s an example of the firewall setup:

*filter

#  Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

#  Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allows all outbound traffic
#  You can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Allows HTTP and HTTPS connections from anywhere (the normal ports for websites)
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

# (alternative) Uncomment to allow HTTP connections only from frontend server
# -A INPUT -p tcp -m state --state NEW -s frontend.server.ip --dport 80 -j ACCEPT

# Allow SSH connections
# THE -dport NUMBER IS THE SAME ONE YOU SET UP IN THE SSHD_CONFIG FILE
#
-A INPUT -p tcp -m state --state NEW --dport 6668 -j ACCEPT

# Uncomment to allow MySQL connections from defined servers
#-A INPUT -p tcp -m state --state NEW -s app.server.1.ip --dport 3306 -j ACCEPT

# Uncomment to allow memcached connections from app servers
#-A INPUT -p tcp -m state --state NEW -s app.server.1.ip --dport 11211 -j ACCEPT

# Uncomment to allow ping
#-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

Apply firewall rules:

sudo iptables-restore < /etc/iptables.test.rules

Check if everything is correct

sudo iptables -L

If everything is fine, save the rules

sudo sh -c 'iptables-save > /etc/iptables.up.rules'

Make sure firewall rules are applied as soon as network interface comes up:

sudo vim /etc/network/interfaces

Add in the editor

pre-up iptables-restore < /etc/iptables.up.rules
after
iface lo inet loopback

Fix locales

sudo locale-gen en_GB.UTF-8 
sudo /usr/sbin/update-locale LANG=en_GB.UTF-8

Set timezone and hostname

sudo dpkg-reconfigure tzdata

Select your timezone

sudo sh -c 'echo "name" > /etc/hostname' 
sudo hostname -F /etc/hostname

Edit /etc/hosts (insert your VPS IP address and your server hostname here)

IP.ad.dr.ess hostname

Update the system

Uncomment universe repositories

sudo vim /etc/apt/sources.list

And run safe and full upgrades

sudo aptitude update 
sudo aptitude safe-upgrade 
sudo aptitude full-upgrade

Install build essentials

sudo aptitude install build-essential 
sudo apt-get install zlibc zlib1g-dev libcurl4-openssl-dev libreadline5-dev

Install setlock for crontab

sudo apt-get install daemontools

Set up Ruby Enterprise Edition 2012.02

Install ruby-ee (ruby 1.8.7 (2012-02-08 MBARI 8/0×6770 on patchlevel 358) [x86_64-linux], MBARI 0×6770, Ruby Enterprise Edition 2012.02):

mkdir ~/tmp 
cd ~/tmp 
wget http://rubyenterpriseedition.googlecode.com/files/ruby-enterprise-1.8.7-2012.02.tar.gz
tar xvzf ruby-enterprise-1.8.7-2012.02.tar.gz

Now enable tc_malloc large pages feature (32bit systems only): http://www.ivankuznetsov.com/2011/07/ree-segfaults-when-rails-application-has-too-many-localisation-files.html

sudo ./ruby-enterprise-1.8.7-2012.02/installer

Create soft links to ruby tools

sudo ln -s /opt/ruby-enterprise-1.8.7-2012.02/bin/ruby /usr/local/bin/ruby
sudo ln -s /opt/ruby-enterprise-1.8.7-2012.02/bin/gem /usr/local/bin/gem
sudo ln -s /opt/ruby-enterprise-1.8.7-2012.02/bin/irb /usr/local/bin/irb
sudo ln -s /opt/ruby-enterprise-1.8.7-2012.02/bin/rake /usr/local/bin/rake
sudo ln -s /opt/ruby-enterprise-1.8.7-2012.02/bin/bundle /usr/local/bin/bundle

Create parametrised ruby launcher

sudo vim /usr/local/bin/ruby-with-env

With the following content:

#!/bin/bash 
export RUBY_HEAP_MIN_SLOTS=1500000 
export RUBY_HEAP_SLOTS_INCREMENT=500000 
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 
export RUBY_GC_MALLOC_LIMIT=50000000 
exec "/usr/local/bin/ruby" "$@"

and make this file executable:

sudo chmod +x /usr/local/bin/ruby-with-env

Set up necessary components

Install readline wrap

sudo apt-get install rlwrap

Install git

sudo apt-get install git-core

Install native dependencies for gems

sudo apt-get install libcurl4-gnutls-dev libxslt-dev libssl-dev

Install MySQL

sudo apt-get install mysql-server libmysqlclient-dev

Install Passenger 3.0.11 and let it compile nginx

cd /tmp wget http://nginx.org/download/nginx-1.0.14.tar.gz
tar xvzf nginx-1.0.14.tar.gz 
sudo /opt/ruby-enterprise-1.8.7-2012.02/bin/passenger-install-nginx-module

- choose installation option 2
- provide source path /tmp/nginx-1.0.14
- provide additional compilation options –with-http_realip_module –with-http_gzip_static_module –without-mail_pop3_module –without-mail_smtp_module –without-mail_imap_module

Create nginx init script:

sudo vim /etc/init.d/nginx

Enter the following content:

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/opt/nginx/sbin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
    . /etc/default/nginx
fi

set -e

. /lib/lsb/init-functions

test_nginx_config() {
  if nginx -t
  then
    return 0
  else
    return $?
  fi
}

case "$1" in
  start)
    echo -n "Starting $DESC: "
        test_nginx_config
    start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON || true
    echo "$NAME."
    ;;
  restart|force-reload)
    echo -n "Restarting $DESC: "
    start-stop-daemon --stop --quiet --pidfile \
        /var/run/$NAME.pid --exec $DAEMON || true
    sleep 1
        test_nginx_config
    start-stop-daemon --start --quiet --pidfile \
        /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;
  reload)
        echo -n "Reloading $DESC configuration: "
        test_nginx_config
        start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/$NAME.pid \
            --exec $DAEMON || true
        echo "$NAME."
        ;;
  configtest)
        echo -n "Testing $DESC configuration: "
        if test_nginx_config
        then
          echo "$NAME."
        else
          exit $?
        fi
        ;;
  status)
    status_of_proc -p /var/run/$NAME.pid "$DAEMON" nginx && exit 0 || exit $?
    ;;
  *)
    echo "Usage: $NAME {start|stop|restart|reload|force-reload|status|configtest}" >&2
    exit 1
    ;;
esac

exit 0

Make it executable:

sudo chmod +x /etc/init.d/nginx

Add nginx to autostartup list:

sudo /usr/sbin/update-rc.d -f nginx defaults

Edit main nginx config

sudo vim /opt/nginx/conf/nginx.conf

Enter the following content:

user  www-data;
worker_processes  4;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  8192;
    use epoll;
}

http {
    passenger_root /opt/ruby-enterprise-1.8.7-2012.02/lib/ruby/gems/1.8/gems/passenger-3.0.11;
    passenger_ruby /usr/local/bin/ruby-with-env;

    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log;

    sendfile       on;
    tcp_nopush     on;
    tcp_nodelay        on;
    keepalive_timeout  65;

    # Passenger never sleeps!
    passenger_pool_idle_time 0;

    # Use more instances, because memory is enough
    passenger_max_pool_size 15;

    # Start application instantly
    passenger_pre_start http://127.0.0.1/;

    client_max_body_size 4m;

    include /opt/nginx/conf/sites-enabled/*;
}

Create site configuration and log directories:

sudo mkdir /opt/nginx/conf/sites-enabled 
sudo mkdir /opt/nginx/conf/sites-available 
sudo mkdir /var/log/nginx

Set up virtual hosts

Create a separate user for each virtual host, create a home directory, set password, create folder for logs and web application, set permissions

sudo useradd newuser -d /home/newuser 
sudo mkdir /home/newuser 
sudo passwd newuser password
sudo mkdir /home/username/hostname 
sudo mkdir /home/username/logs 
sudo mkdir /home/username/logs/hostname 
sudo chown -R username:www-data /home/username/ 
sudo chmod 750 /home/username/

Configure hosts:

sudo vim /opt/nginx/conf/sites-available/mysite.com

server {
        listen   80;
        server_name  .mysite.com;

        access_log  /home/user/logs/mysite.com/access.log;
        error_log  /home/user/logs/mysite.com/error.log;
        root   /home/user/mysite.com/current/public;

        # uncomment if traffic is coming from a frontend server/loadbalancer
        #set_real_ip_from   IP.ad.dre.ss;
        #real_ip_header     X-Real-IP;

        passenger_enabled on;
        passenger_min_instances 5;
}

Enable site:

sudo ln -s /opt/nginx/conf/sites-available/mysite.com /opt/nginx/conf/sites-enabled/mysite.com

Set up log rotation:

sudo ln -s /home/user/mysite.com/current/config/logrotate.conf /etc/logrotate.d/mysite

Install fastcgi (for PHP setups)

Install dependencies

sudo apt-get install libmcrypt-dev libxml2-dev libpng-dev autoconf2.13 libevent-dev libltdl-dev

Download latest stable PHP 5.2.13, Suhosin patch, PHP-FPM patch

cd ~/tmp wget http://pl2.php.net/get/php-5.2.13.tar.gz/from/pl.php.net/mirror
wget http://download.suhosin.org/suhosin-patch-5.2.13-0.9.7.patch.gz
wget http://php-fpm.org/downloads/php-5.2.13-fpm-0.5.13.diff.gz
tar xvzf php-5.2.13.tar.gz gunzip suhosin-patch-5.2.13-0.9.7.patch.gz 
gunzip php-5.2.13-fpm-0.5.13.diff.gz 
cd php-5.2.13 
patch -p 1 -i ../php-5.2.13-fpm-0.5.13.diff 
patch -p 1 -i ../suhosin-patch-5.2.13-0.9.7.patch 
./buildconf --force 
./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl --with-mysql --with-mysql-sock --with-gd --without-sqlite --disable-pdo 
make 
make test 
sudo make install

Alternatively download latest stable PHP 5.3.2, Suhosin patch, apply PHP-FPM patch

cd ~/tmp http://fi.php.net/get/php-5.3.2.tar.gz/from/this/mirror
wget http://download.suhosin.org/suhosin-patch-5.3.2-0.9.9.1.patch.gz
tar xvzf php-5.3.2.tar.gz gunzip suhosin-patch-5.3.2-0.9.9.1.patch.gz 
cd php-5.3.2 
patch -p 1 -i ../suhosin-patch-5.3.2-0.9.9.1.patch 
svn co http://svn.php.net/repository/php/php-src/trunk/sapi/fpm sapi/fpm 
./buildconf --force 
./configure --enable-fastcgi --enable-fpm --with-mcrypt --with-zlib --enable-mbstring --with-openssl --with-mysql --with-mysql-sock --with-gd --without-sqlite --disable-pdo --disable-reflection 
make 
make test 
sudo make install

Uninstall autoconf2.13 after compilation, since it is an old version and only required for PHP compilation

sudo apt-get uninstall autoconf2.13

Change user and group of php-fpm processes to your user dedicated to this application and www-data – lines 63 and 66 respectively

sudo vim /usr/local/etc/php-fpm.conf

Edit PHP settings

sudo vim /etc/php5/cgi/php.ini

Set:

max_execution_time = 30 
memory_limit = 64M 
error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR 
display_errors = Off 
log_errors = On 
error_log = /var/log/php.log 
register_globals = Off

sudo  /etc/init.d/nginx restart

Entrepreneurs, such as Illya Segalovich (co-founder of Yandex), Loic LeMeur (founder of Seesmic and LeWeb), Peter Arvai (co-founder of Prezi) and many others (see full list on the book’s website: www.europeanfoundersatwork.com) tell about how they started, built, pivoted and drove their businesses to success.

The book gives a unique first-hand perspective on how to grow a successful business from Europe, what is the importance of US market, what are the challenges European start-ups are facing and what are the competitive advantages of being in Europe.

It is an inspiring book, and it is very relevant to European entrepreneurs. While stories of US start-ups quite often start with “we got $N mln in funding and started growing from there”, in Europe it’s more about bootstrapping and building a profit-generating machine. I would definitely recommend it to anyone who is thinking of starting a technology company in Europe or is already running one.

 A couple of quotes from the book that I particularly liked:

From interview with Peter Arvai, co-founder of Prezi (company is in the business of SaaS presentation software):

“In my case, and I think I alluded to this if I’ve not said it directly, I’ve been very concerned with not building a company where there’s too much of a feeling that there’s an A and a B team. And naturally you’d think that the A team is in San Francisco. That is the cool place where everyone wants to be. And the B team is in Budapest, this former communist, gray country—and that’s where you outsource people. This is how most people tend to look at the world, and it’s a view that bugs the hell out of me.

And so what can you do as an entrepreneur is to try to work on those preconceptions. So one thing that we do, for example, is a fellowship program. The fellowship program allows Prezi employees to work in the other office and Prezi sponsors the travel and the living costs for the person if this is within, of course, the three-month visa restrictions that are allowed. So this means that people can experience the different places with all of their strengths and weaknesses.

Another thing is, so far we’ve had a semi-annual power week. I don’t know if we will be able to keep this up, but during this week we collect the entire company in one place and so far it’s always been Budapest because that’s always where we’ve had more people. This was probably a very important moment in Prezi history as well.”

Having the head office of HeiaHeia in Helsinki, Finland and R&D in Krasnodar, Russia it takes certain amount of effort not to split into A and B teams, so it’s very interesting to see how other companies with similar setup are addressing the issue.

And a couple of valuable notes on how to go to the US being a European company:

From interview with Jos White, founder of MessageLabs (UK SaaS startup, made $700M exit to Symantec in 2008), now a partner in Notion Capital, VC specializing in B2B SaaS businesses:

“You usually cannot charge as much for your service in the US, so you have to make it up on scale. Initially, US can be a loss leader until you really build up the scale. So we did have more users in the US than anywhere else, but we still had Europe as our largest market in terms of revenue and profit.

[...]

…initially we entered the US with a fairly small budget and we just thought we would have a sales team out there and see if we can do it that way. And it just doesn’t work. I think that
in the US you really have to make a full investment into the market. You need to have a fairly autonomous management team. You need to be prepared to look at spending money for some time before delivering any profitability.”

From interview with Peldi Guilizzoni, founder of Balsamiq:

“I wanted to keep an American company because half of our customers are American. And they like to call an American phone number if they have a problem, and they like to mail a check, in dollars, in the mail, to an American address. It’s just the way software business is done in the US.”

Spending 3 days with people who are working at the bleeding edge of data science was an unforgettable experience. I got my doze of inspiration and got a lot of new ideas how to apply data science in HeiaHeia. It’s difficult to underestimate the importance data analysis will have in the nearest years. Companies that do not get the importance of understanding data and making their decisions based on data analysis instead of gut feeling of board members/operative management will simply fade away.

Unfortunately HeiaHeia was the only company from Finland attending the conference. But I’m really happy to see that recently there are more and more signals that companies in Finland are starting to realize the importance of data, and there are new Finnish start-ups dealing with data analysis. I believe that Finland has an excellent opportunity to have not only a cluster of game development companies, but also big data companies and start-ups. So far it seems that the Valley, London and Australia are leading in this field.

By the way, Trulia (co-founded by Sami Inkinen) had an excellent demo in the halls of the conference venue – check it out in their blog.

Below are my notes from the conference – I added presentation links and videos that I have found, but otherwise those are quite unstructured. There were multiple tracks and it was very difficult to choose between them. Highlights of the conference are talks by Avinash Kaushik, Jeremy Howard, Matt Biddulph, Ben Goldacre, and Alasdair Allan and the Oxford-style debate on the proposition “In data science, domain expertise is more important than machine learning skill.” (see videos below).

== Day 1 ==

Michael Rys / Microsoft – SQL and NoSQL – two sides of the same coin

http://www.slideshare.net/MichaelRys/sql-and-nosql-are-two-sides-of-the-same-coin-strata-2012

Web 2.0 – money are made by “motetizing the social”
- improving individual experience
- re-selling agreate data

MySpace 500 database servers (use stored procedures)
FB is running on 1800 MySQL databases (+Cassandra, HBase, etc)

Shard/partition data. Global transactions do not scale.
Read cache might help.

Social networking – user status update. Update user’s own db, then service dispatcher asynchronously updates servers for user’s friends.

Eventual consistency – changes are taking time to propagate.
Propagation needs to be able to deal with failures – retries.

Scaling requires:
- elasticity
- multiversion schema support
- flexible schema support

Most of the customers end up developing own solutions on top of the existing db servers.

NoSQL:
- low cost
- HA, scale out, performance
- data first: flexible schema

MS Azure with federation: split/drop of shards without downtime (no merge supported yet)
Running aggregate queries on federated data using map-reduce approach in SQL:
select sum(scores)/sum(count) from (select scores, count from federated_table …)

NoSQL issues:
- no secondary indices
- eventual consistency is not always and optimal choice

Architecture:
- multiple shards, each one has read-only replicas for OLAP processing

Claudia Perlich – chief scientists Media 6 degrees, From Knowing ‘What’ To Understanding ‘Why’

Does showing online ad to a person increases probability of that person going and buying the product.
Collecting full set of events – from the patterns on browsing the web to identify the target and
bid to show the ad to that target and then monitoring if that person will visit the site.

Clicks are worth nothing – conversion is everything. But raw conversion raws are very deceiving.
Does it makes sense to show an ad to the people who will buy the product anyway?
The idea is to track how advertising changes the rate of conversion.

E.g. user visited the web site within 5 days after seeing the ad.

Understand the causal process. A/B testing is the way to do it – but it costs a lot of money.
Non invasive causal estimation.
TMLE – targeted maximum likelihood estimate

Have null tests: showing unrelated ads should have no effect on conversion!

A lot of people don’t want to be told that what they are doing is subobtimal – that hinders the process.
Marketer, ad agency, creative designers – all follow their own best practices.

Tools:
1. always pull the data yourself – it takes more time to figure out how the data was pulled than to pull it yourself
2. hadoop + hive, perl/python to clean up the data, R

Time-wise: 0,5-1 year to build the model that is correct and believable (iterative process)

Monica Rogati / senior data scientist @ LinkedIn – The Model and the Train Wreck: A Training Data How-to

Peter Norvig: “The unreasonable effectiveness of data talk” – more data beats clever algorithms.
But what beats more data is better data.

Cold start problem: what to do if we don’t know enough about users yet?
Random recommendations are not good – too easy to offend people.

Jacob Perkins / Weotta – Corpus Bootstrapping with NLTK

translating existing corpus with Yahoo Babelfish API: English -> Spanish
then using Naive Bayes to classify spanish movie reviews

bootstrapping phrase selection

Yelp study indicated that naive bayes classifier produces more accurate classification compared to mechanical turk

Ben Gimpert / Altos research – The Importance of Importance: An Introduction to Feature Selection

collect real estate listings

importance of selecting proper features for data mining/predictions
Gartner’s hype cycle 2011: www.gartner.com/it/page.jps?id=1763814 – data mining is on the rise

Data dimensionality doubles every year, Moore’s law predicts computational power to double only every 2 years.

CART model, OLS, Ridge – time goes down, accuracy goes up

Feature selection:
- manual (apply domain expertise)
- automatic – look for efficiency of all subsets (but this is only a thought experiment)
- forward stepwise feature selection
- information gain feature selection (if taking out a feature increases entropy, it must be pretty important)
- Ridge regression regularization
- least angle regression (LARS)
- principles components analysis (PCA) – does badly agains non-linear data shapes (it’s also difficult to explain it)
- ensamble of decision trees (CART or random forests)

Matt Biddulph (@mattb) / Dopplr co-founder, Hackdiary.com – Social Network Analysis Isn’t Just For People

- triadic closure for people you might know implementation
- community detection in SNA 1/2m SUM(Aij – ki * kj / 2m) delta (ci, cj)
- belgian phonecall network – split between french and flemish speaking communities
- Nokia map app – all route calculation requests are logged, it can be used for identifying social links between cities
- gephi.org – photoshop of graphs
- betweenness centrality algorithm to size and colour network of topics tagged in del.ico.us
- links between most active pages on wikipedia as a social network
- map of affinity between journalists and topics – using Guaridian API
- map of the developers using GitHub API
- links between programming languages and music (last.fm API), JS, Node.js – alternative and rock, Ruby – singer-songwriter

Robert Lefkowitz / 1010data – Array Theory vs. Set Theory in Managing Data

- array structure are a superset of set structures, so array-based databases are more powerful than set based databases
- permutation vector (order in R) gives you a list of indices which can be used to retrieve original data elements in order
- example: in a database with “date, place, low, high” structure how do you get the “range” (high-low) for each date
- it’s possible to make all operations defined in the set theory (in relational databases) with permutation vectors
and also grading, proximity joins, fuzzy joins, transation isolation, sharding
- 1010data.com has an implementation of the database based on array theory, another one is SciDB

Robert Lancaster / Orbitz – Survival Analysis for Cache Time-to-Live Optimization

- survival analysis (heart-transplant pationes – time to death, leukemia in remission – time to relapse)
- survival function (Weibull, exponential, log-logistic)
- Kaplan-Meier estimates
- Orbitz applied this to cache of rates and availability pulled from the partners
- change deployed on Feb 2, 2012 – sharp decrease in traffic, impact on look-to-book is positive

Debate – who should be the first hire in a data science team – a domain expert or machine learning expert?

O’Reilly has provided a recording of this debate for free – click here to watch it.

Mike Driscoll (@medriscoll), moderator of this debate, summarized it nicely in his blog: “Thus who you decide to hire as your first data scientist — a domain expert or a machine learner — might be as simple as this: could you currently prepare your data for a Kaggle competition?  If so, then hire a machine learner.  If not, hire a data scientist who has the domain expertise and the data hacking skills to get you there.”

== Day 2 ==

Edd Dumbill / O’Reilly Media, Alistair Croll / Bitcurrent

Data science is growing. This year latecomers like MS are joining in. Data visualization is growing in importance.
20 years ago computing became available to all businesses – we built an exo-skeletone for the business.
Now data is becoming nervous system of the businesses.

Doug Cutting / Cloudera – The Apache Hadoop Ecosystem

- we’re now at peta-scale kMGTPEZY
- big data is using commodity hw, but scales better
- big data: distributes, raw data (vs. schema), open source (vs. proprietory)
- Hadoop became de-facto industry standard, it is the kernel
- there are a lot of projects around the kernel (like nobody uses Linux kernel alone)
- most of the tools are open source at apache

Dave Campbell / Microsoft – Do We Have The Tools We Need To Navigate The New World Of Data?

- microsoft announced hadoop support last september
- signal -> data -> information -> knowledge -> insights & actions
- Haiti earthquake – how ML was used to build a statistical translation engine for Haitian Creole – English translations
- search & acquire -> explore & analyze -> explain & share
- new agenda for the tooling: data visualisation shall be as easy as creating a powerpoint

Abhishek Mehta / Tresata – Decoding the Great American ZIP myth

- levitt (creator of suburbia – same houses), ford – homogeniety
- we are data rich but information poor society
- tools for solving data problems are finally here – financial system and all o fthe industries that are in catarsis can be rebuilt

Mike Olson / Cloudera CEO – Guns, Drugs and Oil: Attacking Big Problems with Big Data

- Drugs: genome analysis using hadoop
- Guns: Santa Cruz – predictive policing
- Oil: subsurface structure mapping
- importance is in applying technology to solving big social problems

Flavio Villanustre / LexisNexis Risk Solutions and HPCC Systems – Machine Learning and Big Data: Sustainable Value or Hype?

- company with 15 years of big data experience
- released HPCC as open source last year
- ML is establishing itself as an important discipline, companies that don’t leverage the data they have will loose to competitors

Steve Schoettler / co-founder of Zynga, Junyo – Learning Analytics: What Could You Do With Five Orders of Magnitude More Data About Learning?

- Junyo – learning analytics
- US government takes two years to collect and process school scores nationwide (now 2009 is the last data point available)
- there’s no analysis – there’s no way to improve
- most important factor affecting student achievement is feedback
- it’s possible to double student achievement – need to tailor instructions to students’ needs
- technology needs to be used for that, because that’s a lot of work

Avinash Kaushik / Market Motive – A Big Data Imperative: Driving Big Action

- information is powerful. But it is how we use it that will define us.
- bring people in the organisation closer to the data, let them run reports
- known knowns | known unknowns | unknown unknowns – Donald Rumsfeld
- tops (top-10, bottom-10) do not give you anything – it’s gauss, the information is in the middle
- GA is data puking, highlighting differences between predictions and real data is important
- Occam Razor blog

Ben Goldacre / Bad Science – The Information Architecture of Medicine is Broken

- it’s possible to see if any small negative studies are missing
- Tamiflu story: no data is given by Roche
- Reboxetine: 76% of all the trials were witheld
- research fraud
- it’s possible to find worst offenders by matching data from various sources
- alltrials.org

Eddie Satterly / Expedia, Sanjay Mehta / Splunk – Turning Big Data Into Competitive Advantage

- Big Data vs. Barak Obama search popularity – data wins
- this in itself is a big data problem
- splunk: collects data in real time from various sources reliably, indexes it and makes available for search, analysis, visualisation and pattern recognition
- Expedia: 4,000 people in R&D – 8,000 total
- Expedia took Splunk into use in 3 months for all machines that are logging and everything else – call logs on voip, tickets, changes, scripts
- Data ~6Tb/day
- Expedia deploy cycle – twice a week
- Expedia was not prepared to give out all BI data to SaaS solution – one of the reasons for Splunk

Peter Skomoroch / LinkedIn – Street Fighting Data Science

- act as a street fighter: analyze, improvise, anticipate, adapt
- how to approach a new problem as a new grad: predict sales and recommend price for new shoes model
- go and see academic papers, what other people have done, create a model how to drop the price over time
- result: pricing model decreases sales by 30% – what has gone wrong
- ran complex black box model
- didn’t analyze the data first
- didn’t anticipate elastisicty error
- talk at Velocity 2011 by John Rauser from Amazon: “Look at your data”: http://www.youtube.com/watch?v=coNDCIMH8bk
- look at your data!
- looking at the top end of anything doesn’t really tell you much!
- book: street fighting mathematics
- red teaming: plan for worst case scenarios, to avoid situations like described in teh beginning of the talk
- blog: http://dataist.wordpress.com/

Sam Shah / LinkedIn – Collaborative Filtering using MapReduce

Collaborative filtering techinques:
- latente factor models
- neighbourhood based

algorithmic challenges:
- performance
- desparsification

Themes:
- reliance
- user experience
- productization

Implementation: co-occurence graph
1. Map-reduce implementation: time-viewer-viewee
2. Index: alice, bob, 1 (co-occurence)

generating pairs might be expensive – limit by time
first version does at 60% of optimal solution

improvements:
- check your outputs (Obama problem)
- check your inputs (repeated profile views probably should be discounted)
- drill-down: if people you might know shows a lot of “john smiths” (because of failed searches) then CTR will be very low

presentation matters:
- adding avatars to people you might know: 50% to CTR

putting it to production:

- pairs->scoring->top-n->push production
- azkaban – for managing pig jobs
- oozie -
- voldemort – working with batch-computed data (Serving large scale batch computed data with project voldemort. In Fast 2012) https://www.usenix.org/conference/fast12/serving-large-scale-batch-computed-data-project-voldemort

Philip Kromer / Infochimps – Disambiguation: Embrace wrong answers & find truth

- human entered geolocations
- instead of correcting those, they can be used to identify strong links

Xavier Amatriain / Netflix – Netflix recommendations: beyond the 5 stars

(update 2012-04-07: now also available as blog post http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html)

- Netflix prize: 10% improvement in predicted rating
- Netflix went from dvd by mail company to a global streaming company that operates not only on the web, but on many other devices
- make user aware the results provided are personalized
- diversity in the top-10 recommendations (for you, for wife, for kids)
- social support – not only recommendation based on personal preferences, but how many friends have liked it
- note: in the US there’s a legal problem with showing friends’ likes/favourites
- ranking = scoring + sorting + filtering
- models used by netflix: lo/lin regression, elastic nets, scd, boltzman machines, gradient boosted decision trees, …
- recommending new movies: even a few more ratings beat more data
- Norvig’s statement about more data beats better alorithms is valid for language processing, but Netflix indicates that more training examples are not helping if there are just a few relevant features

== Day 3 ==

Jonathan Gosier / metaLayer Inc. – Democratization of Data Platforms

- art used to live in the domain of specialists – paintings were comisisoned by institutions
- not until recently math and science were also in the domain of specialist
- data science now is in a similar position – it is for experts only
- there need to be tools available for general public
- examples: subprime mortgage market – how to understand who owns which loan, automatic trading systems – who understands how that works
- ushahidi – open source distress signal collection
- there should be an equivalent of a calculator for unstructured data
- “…take the best and spread it around to everybody…” – Steve Jobs
- metaLayer offer drag and drop insights

Luke Lonergan / Greenplum, a division of EMC – 5 Big Questions about Big Data

Coco Krumme / MIT Media Lab – The Trouble with Taste

- data analysis of the wine-tasting evening

Pete Warden / Jetpac – Embrace the Chaos

- building a business on somebody else’s data is very dangerous
- Google had to deal with a chaos of billions of web pages
- take support emails or some other seeminly chaotic data and try to extract value from it

Usman Haque / Pachube.com – Open Data and the Internet of Things

- general infrastructure for collecting data from various sensors
- Internet of things bill of rights
- people own the data they create

Gary Lang / MarkLogic – Big Data’s Next Step: Applications

Hal Varian / Google – Using Google Data for Short-term Economic Forecasting

- Google time-series for searches: vodka searches peak every Saturday and Dec 31st is a spike, hangover searches peak every Sunday and Jan 1st is a spike
- initial claims vs unemployement level using governmant data
- peaks in the end of every recession
- challenges: simple corellation don’t work (like with unemployment)
- human judgement doesn’t scale
- baseball and auto sales both peak in summer, but just because of seasonality, not causality
- Google Research blog
- fat regression
- incremental predictability
- during resession coupon-related queries are growing
- prediction for UM consumer sentiment

Theo Schlossnagle / OmniTI – Is this normal? Finding anomalies in real-time data

- author of “Scalable Internet Architectures”
- hard real-time systems: outputs are considered incorrect if the latency of their delivery is above a specified amount
- soft real-time systems: similar, but less useful instead of incorrect
- Big data systems:
- traditional: oracel, postgres, mysql
- the shiny: hadoop, hive, hbase, pig, cassandra
- the real time: sql stream, s4, flumebase, truviso, esper, storm
- 300k datum/sec
- real-time queries done with Esper
- Holt-Winters – tri-variate seasunal regression
- look at historic data
- use that to predict the immediate future with some quantifiable confidence
- implemented the Snowth for storage of data
- implemented a C/lua distributed system to analyze 4 weeks of data
- to ensure the system real-time – need to ensure that queries return in less than 2ms
- how to transform batch processing, offline analytics to online real-time analytics without crazy brainpower

Jeremy Howard / Kaggle – From Predictive Modelling to Optimization: The Next Frontier

- predicting something doesn’t mean that you’ve made an impact on human lifes
- 1. figure out the objective
- 2. figure out the levers to pull – what can we change
- 3. figure out what data we have
- 4. write a PhD on PageRank
- working in insurance – objective is to make shit loads of money made on each individual client
- to get necessary data – insurance companies had to start charging random prices
- to convince them to do that, a simulation was run showing what happens if that is done vs. not done
- how to apply this to marketing?
- objective: maximise lifetime value of a customer
- levers: recommendations, offers, discounts, care calls, etc.
- what data can we collect?
- Amazon recommends pratchet books to those who buys a lot of them. that’s not helpful
- to improve it – start offering random stuff to collect the data that can later be mined

Alyona Medelyan / Pingar, Anna Divoli / Pingar – Mining Unstructured Data: Practical Applications

- IDC: 17h/wk searching information, 14h/wk writing emails
- 17h/wk = $37,000/year
- pingar provides metadata extraction tools

Alasdair Allan / University of Exeter – Migratory data: the distributed data you carry with you

- calendar, phonebook, camera phone pictures, bookmarks, text messages – all that is carried with you
- photos are geotagged – and there’s a lot of other metadata collected
- iOS spotlight index contains a cache of your smses and contacts even deleted ones
- web cache, sms draft cache – are all metadata
- facetime calls cache on iOS
- google search cache is not wiped when you clear you browser cache on iOS
- gps cache on iOS is never deleted for a year in iOS4, in iOS5 caching time is decreased
- history of all maps searches on iOS are cached
- keyboard cache is stored as well – text messages can be reconstructed from it
- cache of last visible screen when swithing between applications
- mitmproxy – an SSL-capable man-in-the-middle proxy
- siri proxy – proxy to listen in to siri traffic
- angry birds and cut the rope are offering “look at my local community” – that uploads your phonbook to the internet
- data exhaust: broadcasting your location, rfid cards like oyster, tesco purchases tracking
- who owns that data? consumer or companies?
- bit.ly/FindingYourFriends – http://www.cs.rochester.edu/~sadilek/publications/Sadilek-Kautz-Bigham_Finding-Your-Friends-and-Following-Them-to-Where-You-Are_WSDM-12.pdf
- after you stop sharing your location, it’s possible to pinpoint you to a 100m with a 80% accuracy by locations of your friends
- how to prevent CC fraud – compare postcode of the latest credit card transaction and iPhone location
- mapping receipts coming to email with online transaction

Robbie Allen / Automated Insights, Inc. – From Big Data to Big Insights

- big data by itself is not helpful
- visualizations is one possible answer, but one problem here is that they still require interpretation
- automatic writing by software – it can process more data and can be continuosly improved
- examples: automatic stock market reports, insider trading reports
- parked domains: generating relevant content for those pages from a list of keywords

Cheryl Phillips / The Seattle Times – Exploring the Stories Behind the Data

- philip meyer
- seattle times visualisations of data brought to the users – illegal forest logging, casualties of war
- The Nutgraph: important and visual information that attracts attention and makes people explore data further
- never put more than one number into a paragraph of text
- don’t overwhelm the reader with detail that they don’t need

It was a quick read, and I should say I’m a bit disappointed. If you want to save time and money – go to the book’s web page and you’ll get all main ideas that are described in the full version.

Yes, the world is changing very fast. You don’t want to become a Detroit of the modern age. You should not expect a life-time employment in one company. Go learn new stuff, go meet people to find interesting opportunities. I was expecting a more insightful book with less direct LinkedIn service promotion, but in the end got a help-yourself, very Silicon Valley-centric guide on building a network using LinkedIn.

Unfortunately I cannot recommend this book, unless it’s news for you that you need to invest in continuous self-education and network building to succeed.

I’m conflicted about recommending this book. On one hand – it’s 4 hours of your time that can be spent on better. On the other hand – it made me reconsider my own information diet and I see how I can now do more in a day because of that. So next time when you want to open IM, GMail, Twitter or Facebook, consider reading this book instead.

Update: after  months trying to follow the recommendations of this book – I have noticed that I have significantly improved my productivity. Topic of responsible information consumption also came up many times in conversations I had with many smart people in the past few months. There’s definitely a tectonic shift going on when it comes to information consumption habits.

Examples of crashes we observed:

/vendor/rails/actionpack/lib/action_controller/routing/route.rb:205):2:
   [BUG] Segmentation fault
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/yaml.rb:133: 
   [BUG] Segmentation fault
/vendor/rails/activesupport/lib/active_support/vendor/i18n-0.3.7/i18n/
   backend/base.rb:257: [BUG] Segmentation fault
/vendor/rails/actionpack/lib/action_view/template.rb:226: [BUG] Segmentation fault
/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/gems/1.8/gems/pauldix-sax-machine-0.0.14/
   lib/sax-machine/sax_document.rb:30: [BUG] Segmentation fault
/vendor/rails/activesupport/lib/active_support/memoizable.rb:32: [BUG] Segmentation fault

After banging my head against the wall for a week I found a solution (even two) and what might seem to be a likely reason for the segfaults. Two “suspects” – lack of available memory and incorrect version of libxml were ruled out. What seems to be the actual reason is the total size of the localisation files in config/locales loaded upon startup:

$ du -shb config/locales
1665858    config/locales

$ cd config/locales
$ find . -type f | wc -l
805

So ~1.6Mb in 805 files give occasional segfaults. Adding 200Kb of localisation files more started giving 100% segfaults on script/console startup.

Now I’ve found two workarounds for this problem.

1. Recompile REE with –no-tcmalloc flag

./ruby-enterprise-1.8.7-2011.03/installer --no-tcmalloc

Note that on 64-bit platforms tcmalloc is disabled by default.

2. Enable large pages feature in tcmalloc

This is described in tcmalloc documentation as: “Internally, tcmalloc divides its memory into “pages.”  The default page size is chosen to minimize memory use by reducing fragmentation. The cost is that keeping track of these pages can cost tcmalloc time. We’ve added a new, experimental flag to tcmalloc that enables a larger page size.  In general, this will increase the memory needs of applications using tcmalloc.  However, in many cases it will speed up the applications as well, particularly if they allocate and free a lot of memory.  We’ve seen average speedups of 3-5% on Google applications.”

There’s a warning – “this feature is still very experimental”, but it works to solve the problem with too many localisation files.

To compile REE with tcmalloc with large pages enables I just edited ruby-enterprise-1.8.7-2011.03/source/distro/google-perftools-1.7/src/common.h – replaced

#if defined(TCMALLOC_LARGE_PAGES)
static const size_t kPageShift  = 15;
static const size_t kNumClasses = 95;
static const size_t kMaxThreadCacheSize = 4 << 20;
#else
static const size_t kPageShift  = 12;
static const size_t kNumClasses = 61;
static const size_t kMaxThreadCacheSize = 2 << 20;
#endif

with

static const size_t kPageShift  = 15;
static const size_t kNumClasses = 95;
static const size_t kMaxThreadCacheSize = 4 << 20;

On production servers I opted for no tcmalloc for now – but I hope there’ll be a better way to deal with this issue soon.

Rails are providing developers with really excellent abstractions, but it’s always good to know what’s under the hood and how it all works.

There are a few things that might potentially cause bugs in your code, or waste your time (speaking from my own experience). So here goes:

1. Beware of globally keyed fragments

Let’s take example from Rails tutorial:

<% cache do %>
  All available products:
  <% Product.all.each do |p| %>
    <%= link_to p.name, product_url(p) %>
  <% end %>
<% end %>

Now if you need to deal with a multi-language site you might want to make cache fragment language dependent. What might seem a convenient solution:

<%- cache([user.locale.to_s]) do -%>

will turn into a source of very interesting problems. While calling the cache method without parameters will automatically create a controller/action specific cache key, calling it with a key will make this fragment a globally keyed fragment. Cache key in the first case is going to look like “views/localhost:3000/controller-name”, and in the other case “views/en” – this is not as unique identifier any more.

While automatic cache key naming provided by rails is very convenient, it is very easy to run into a problem with duplicate cache key names used in different places.

2. Another pitfall of automatic cache key naming is that you shall never assume that when creating a cache with global key you can later find it using e.g. telnet interface to memcache. Example – add

<%- cache('unique_cache_key') do -%>
<%- end -%>

in your view and then try to read directly from memcache:

$ telnet localhost 11211
GET unique_cache_key
END

At the same time

GET views/unique_cache_key

will work. It’s easy to make this mistake trying to check or delete cache keys directly from memcache when using Rails cache methods.

3. delete_matched is not supported by memcached (see rails/activesupport/lib/active_support/cache/mem_cache_store.rb)

In practice that means that if you’re using memcached as Rails cache engine and trying to delete or expire fragment cache using standard Rails methods and regexp – you’ll fail.

expire_fragment(/base\/xyz.*/)

will fail miserably. Ideal solution is not to use explicit cache expiration, but rather create cache keys in such a way that doesn’t require expiration. Alternatively it’s possible to use extensions implementing delete_matched for memcached (haven’t tried it myself though).

Tip: one very useful tool for checking memcached is peep by Evan Weaver – allows you to peek into the cache and see what’s really cached and how it is used.