I decided I really needed to work on a server backup system, so here are my notes on the system I have now.

First things first, the files I want to backup are owned by all different users, so the only user who can run the backup process is root. Therefore I can’t just run rsync from my local machine and grab the files from the server, the backup process has to be run on the server and backup to the backup machine. Now in my case this was a bit of a trick because the backup machine was behind a firewall, so the server had no direct line of communication to it. So I wrote a script to turn on a reverse ssh port forward.

recv-serv-backup.sh

#!/bin/sh

ssh -R 8000:127.0.0.1:22 -N user@kvasir.mindstab.net

When run on the backup machine behind a firewall, it connects to the server (kvasir) and listens on port 8000. When ssh on kvasir connects to port 8000 it redirects that traffic to local port 22, the ssh port of the backup machine. This is how the firewall is gotten around. Reverse port mapping is a cool trick to master.

Next as root on kvasir I generated a public ssh key and put it on the backup machine so root could automatically log on repeatedly to the back up machine (think lots of rsync calls) once the key was loaded once. Then I hooked up the key to keychain. That is all better outlined at: Gentoo Linux Documentation: OpenSSH key management, Part 1 and Gentoo Linux Documentation: OpenSSH key management, Part 2.

The all I needed to do was poke the rsync syntax to use the nonstandard ssh port for backup. The best method I found was

rsync -e "ssh -p 8000" -av

rsync for those of you who don’t know is a great little backup tools. It’s like a smart (in that it only copies files that have been modified since the last backup) network aware (since it can use ssh) copy tool. Simple but really useful.

So with that I wrote a backup script on the server

server-backup.sh

#!/bin/sh

HOSTNAME=taru
keychain /root/.ssh/id_rsa
. /root/.keychain/$HOSTNAME-sh

LOG=/root/$HOSTNAME-backup.log

# backup server
SERVER=bion.mindstab.net
USER=dan

date > $LOG

backup () {
        echo "$1..."
        DST=`echo "$1" | gawk '{split($0,a,"/");  result = "/"; for (i=2 ; i < length(a)  ; i++)   result = result "/" a[i];  print result;  };'`
        echo "rsync -e \"ssh -p 8000\" -acv $1 $USER@$SERVER:~/backup/$HOSTNAME$DST" >> $LOG
        rsync -e "ssh " -acv $1 $USER@$SERVER:~/backup/$HOSTNAME$DST  >> $LOG
}

backup "/root"
backup "/etc"
backup "/git"
...

There is one caveat, rsync won’t create subdirectories on the other side specified in the path so you need to create the basic directory structure.

rsync -e "ssh -p 8000" -av /git user@127.0.0.1:~/kvasir/git

is fine because it will create /git just fine, but

rsync -e "ssh -p 8000" -av /home/user user@127.0.0.1:~/kvasir/home/user

will fail if ~/kvasir/home doesn’t exist. So you’ll need to create the basic directory structure or enhance the backup function to strip out extra directories in the target path.

Finally, I didn’t want anyone and everyone to potentially be able to gain access to private data on the backup machine, so the target directory needed to be encrypted. There are a lot of options, but I opted for the easy encFS route and just installed “cryptkeeper” and had it setup the directory.

Now all I have to do is mount the encrypted backup directory, run the script to turn on the reverse ssh tunnel, and run the backup script, and I have an encrypted backup solution for my server that gets around firewalls.

Not bad.

References