This proved to be a non-trivial task. There is a number of tutorials on the net, but none of them described the full solution. So after getting it all to work, I decided to share all the tips and tricks. Feel free to comment, if you will find problems with the following set of instructions.
SSH to redmine.mydomain.com as a user that will be running Redmine (in the following examples it will be ‘redmine_user’).
First, you need to compile openssl – it will be required for curl, git and redmine.

mkdir ~/tmp
cd ~/tmp
wget http://www.openssl.org/source/openssl-0.9.8k.tar.gz
tar xzvf openssl-0.9.8k.tar.gz
cd openssl-0.9.8k
./config shared zlib --prefix=$HOME/.packages
make
make install

Let’s tell the world that we keep binaries and libraries also in the local directory. Edit ~/.bashrc (it is used by all non-login shells):

export TZ='Europe/Helsinki'
export PATH="$HOME/.packages/bin:$PATH"
export LD_LIBRARY_PATH="$HOME/.packages/lib"
export GEM_HOME="$HOME/.gems"
export GEM_PATH="$GEM_HOME:/usr/lib/ruby/gems/1.8"
export PATH="$HOME/.packages/bin:$HOME/.gems/bin:$PATH"
export RUBYLIB="$HOME/.packages/lib:$RUBYLIB"
export LD_LIBRARY_PATH="$HOME/.packages/lib"

# this ensures our gem install processes don't get killed by the DreamHost police
alias gem="nice -n19 ~/.packages/bin/gem"

(You can skip TZ – it is just usefule to have correct time set for your environment. Use tzselect to find out correct TZ string for your region)

And edit ~/.bash_profile (it is used by login shells):

umask 002
PS1='[\h:$PWD]$ '
alias ll="ls -l"
EDITOR="/usr/bin/vim"
. .bashrc

Now let’s apply the changes in active shell:

cd ~ : . .bash_profile

Then you need to compile curl, to be able to compile git with curl and execute clone commands on your server.

cd ~/tmp
wget http://curl.haxx.se/download/curl-7.19.5.tar.gz
tar xzvf curl-7.19.5.tar.gz
cd curl-7.19.5
./configure --prefix=$HOME/.packages --with-ssl=$HOME/.packages
make
make install

Now get and compile Git. If you are not using Dreamhost PS, you might want to compile it with NO_MMPAP=1, to reduce
probability of git process getting killed by Dreamhost police robots due to extensive memory

cd ~/tmp
wget http://www.kernel.org/pub/software/scm/git/git-1.6.4.tar.gz
tar xzvf git-1.6.4.tar.gz
cd git-1.6.4
./configure --prefix=$HOME/.packages --with-curl=$HOME/.packages
make
make install

Let’s start with ruby and rails related stuff. First readline library is needed, for script/console to work.

cd ~/tmp
wget http://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz
tar xzvf readline-5.2.tar.gz
cd readline-5.2
./configure --prefix=$HOME/.packages
make
make install

Install ruby:

cd ~/tmp
wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
tar zxvf ruby-1.8.7-p72.tar.gz
cd ruby-1.8.7-p72
./configure --prefix=$HOME/.packages --with-openssl-dir=$HOME/.packages --with-readline-dir=$HOME/.packages
make
make install

Install rubygems:

cd ~/tmp
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar zxvf rubygems-1.3.5.tgz
cd rubygems-1.3.5
ruby setup.rb config --prefix=$HOME/.packages
ruby setup.rb setup
ruby setup.rb install

Now you can install all required gems and freeze them if necessary.

Installing RedMine

Refer to http://wiki.dreamhost.com/Redmine for details.

But there’s one trick – if you want RedMine to use just compiled version of Git – edit in lib/redmine/scm/adapters/git_adapter.rb:

# Git executable name
GIT_BIN = "/home/username/.packages/bin/git"

If this is not done, default Dreamhost git will be used (which is too old at the moment 1.4.4.4) and Git repository browsing will not work in RedMine
(see http://groups.google.com/group/phusion-passenger/browse_thread/thread/5080d7c7cfbcf20e).

Setting up Git repository

Refer to http://wiki.dreamhost.com/Git for details.

Couple of tricks here. There are bugs in WebDAV functionality in Ubuntu – so if you’re using it – launch Nautilus, use “File->Connect to server…” menu from there (not from the system menu), don’t enter user name in the dialog – leave it empty and enter it when you’re requested username and password in the next dialog.

When setting up WebDAV access rights, give access to user “redmine”.

Setting Git repository copy for Redmine

Create ~/.netrc for your redmine user on Dreamhost

Insert the following line in that file

machine git.mydomain.com login redmine password [redmine_password]

where redmine_password is the password you gave to user redmine in the previous step.
.netrc will ensure that password is not asked when git is accessing the repository.
You don’t need to bother about creating .netrc if you have a public repository (or at least available for cloning without password).

Create a local copy of the repository

For RedMine to be able to display Git repository, it needs to have a local clone of the repository.

mkdir ~/git_project_clones
cd git_project_clones
git clone http://git.mydomain.com/repository_name

Now you should have a local copy that can be used from RedMine.
All you have left to do is to set up regular pulls from the master repository to this local copy. Use command

crontab -e

Add the following line to pull latest change into local copy every 5 minutes:

*/5 * * * * cd /home/redmine_user/git_project_clones/repository_name && /home/redmine_user/.packages/bin/git pull

Make sure that you specify full path to git – otherwise it will execute Dreamhost default git 1.4.4.4 and command will fail with “refusing to create funny ref ‘remotes/origin/*’ locally” error.

Save and exit cron editor.

(You might also want to check Redmine own wiki: http://www.redmine.org/wiki/redmine/HowTo_keep_in_sync_your_git_repository_for_redmine)

Set repository in the settings of your project in RedMine: /home/redmine_user/git_project_clones/repository_name/.git

Now you need to manually update Git repository changesets in RedMine:

cd ~/mydomain.com
script/runner "Repository.fetch_changesets" -e production

And set a hook to your repository to do this every time repository is updated – edit ~/git_project_clones/repository_name/.git/hooks/post-update file and add the following command ther:

cd /home/redmine_user/mydomain.com && script/runner "Repository.fetch_changesets" -e production

When writing this blogpost I found a lot of useful information on Dreamhost own wiki as well on these blog posts:
http://www.wavethenavel.com/2008/09/08/bootstrapping-a-dreamhost-account-for-rails-and-git/
http://juliobiason.net/2008/05/19/git-repositories-on-dreamhost-via-ssh/
http://www.simonecarletti.com/blog/2009/07/configuring-git-repository-with-redmine/