Set up an incremental backup with duplicity, rsync, and backupninja on Debian
Par Mathieu le vendredi 20 février 2015, 22:08 - Hacks - Lien permanent
This is a not-so-concise how-to about setting up an incremental backup, using Backupninja with Duplicity backend on Debian.
Abstract
If you know what a backup is, you should know there are several types of backups :
- A full backup is when you just copy all your files, hoping that the hard drive on the backup server will not explode after 3 backups.
- An incremental backup consist in a base full backup, and the next backups are just "diffs" sent to the backup server, to keep track of modified files.
Obviously, a full backup is easier to read and to restore because it's just plain files, whereas an incremental backup has a specific file format to represent diffs. But considering the gain in speed, bandwidth, and disk space, your choice for a long-term backup solution should be the incremental backup.
The tools
Duplicity is an opensource software similar to rdiff-backup. It creates incremental backups. Duplicity can also encrypt your backups, so they can be safely sent to any remote disk provider. A classic setup for Duplicity would be using rsync as a backend to send files faster to the remote backup server, but you can also use a local drive, a remote FTP server, or an Amazon E3 cloud server. As the title says, I will be using rsync for that setup.
But what about databases ? Databases can't be saved by simply copying files, it could lead to corrupted and unusable data in your backups, so you would use a backup script to fetch your databases before sending it to Duplicity.
Good news : backupninja is the global solution you need. Backupninja is a sort of "backup-master" : it can fetch different type of data (files, databases...) from different sources and sent it to different destinations (plain backup, duplicity, etc). you just have to write a specific config file for each source !
We will use backupninja to fetch our databases, we will add these SQL archives to our files backup, send this to Duplicity backend, and finally send it to our backup server with rsync. And with just 3 config files (one by SQL type, one for Duplicity and rsync).
Let's go !
Requirements
- A Debian machine to backup, let's call it production
- A backup server, with SSH access (SSH is required for rsync, but an FTP server can do the job), let's call it backup
Configure the backup user on the remote machine
In order to send the backed up files to backup, we will need a way to authenticate from production without specifying a password.
First, be root on production : sudo -s
Then, create ssh key pairs : ssh-keygen -t rsa
The public key will set up on /root/.ssh/id_rsa.pub and the private key in /root/.ssh/id_rsa
Now, let's consider backup, check if your SSH server on backup acccepts the public key authentications. In /etc/ssh/sshd_config, you should have the lines :
RSAAuthentication yes PubkeyAuthentication yes AuthorizedKeysFile %h/.ssh/authorized_keys
If not, simply add it (and restart ssh service).
Now, add the prod_server user on backup : useradd prod_server
You may want to change its home, I assume you know how to do it.
Now, back to production, simply copy the SSH public key to the backup file named /home/prod_server/.ssh/authorized_keys
The trashy way : scp /root/.ssh/id_rsa.pub root@backup:/home/prod_server/.ssh/authorized_keys
The classic way : copy/paste the content of production:/root/.ssh/id_rsa.pub at the end of the file backup:/home/prod_server/.ssh/authorized_keys
If you did it right, you should now be able to ssh from production to backup without password.
Install backupninja (and friends)
On production, use the Debian magic line : sudo apt-get install backupninja duplicity rsync
Configure databases dump
Okay, let's start configuring backupninja !
For MySQL, copy the sample file from /usr/share/doc/backupninja/examples/example.mysql to /etc/backup.d/10-alldb.mysql and change it for your own needs (the file is pretty trivial and well commented).
databases = all backupdir = /var/backups/mysql hotcopy = no sqldump = yes compress = yes configfile = /etc/mysql/debian.cnf
For PostgreSQL, all the same ! Copy /usr/share/doc/backupninja/examples/example.mysql to /etc/backup.d/10-alldb.pgsql and change the desired values.
backupdir = /var/backups/postgres databases = all compress = yes format = plain
Why am I using that weird 10- at the beginning of the file ? For the same reason you have to prefix your files with number in Nginx configuration : for precedence. I need the database dumps to be done just before the files backup, otherwise, backupninja will send to duplicity the dumps of yesterday, and next create new dumps without including it in the Duplicity files for the current backup ! So, my database backup configuration files have to be like 10- and my Duplicity configuration file should be like 20-, for instance.
Note that the SQL dumps will be generated in /var/backups/ .
Configure Duplicity files backup
Again, copy the sample file : cp /usr/share/doc/backupninja/examples/example.dup /etc/backup.d/20-files.dup
And change its values :
# Disable testconnect because we use desturl testconnect = no # You may change this if you are a partition-maniac tmpdir = /tmp [gpg] # Using symetric encryption for archive files # Note that an encryption method is mandaory, either with symetric or private keys # Don't forget to note that password somewhere ! password = whateveryouwant [source] # Specify the paths to your files include = /var/spool/cron/crontabs include = /var/log include = /var/mail include = /var/www include = /etc include = /root include = /home include = /usr/local # And to your database dumps ! include = /var/backups/mysql include = /var/backups/postgresql # There are some files we don't need # Don't forget to add the tmpdir in the exclude list, if it was included in the previous paths ! exclude = /home/*/.gnupg exclude = /var/cache/backupninja/duplicity exclude = /tmp [dest] # Adjust these to your own taste incremental = yes increments = 15 keep = 30 keepincroffulls = all # Specify the backup server crendentials ## desturl = file:///usr/local/backup ## desturl = rsync://user@other.host//var/backup/bla ## desturl = s3+http:// ## desturl = ftp://myftpuser@ftp.example.org/remote/ftp/path desturl = rsync://prod_server@backup//home/prod_server/data # Only if you choose FTP # ftp_password = whateveryouwant
Note that this setup uses symetric encryption for duplicity archives, if you want an asymetric encryption to enforce the safety of your backups, you should read the docs and adapt this sample configuration.
Configure backupninja update frequency
You may have noticed that backupninja set up a cronjob in /etc/cron.d/backupninja but if you open that file, you will notice that this cronjob runs every hour. Why ? Because this cronjob is only used to start the backup manager, to check if a refresh is needed, not to actually run the backup every hour. You can set the backup frequency in the file /etc/backupninja.conf :
when = everyday at 01:00
Adjust it to your needs, and then save it. Note that you can combo multiple lines, if you want multiple start dates (read more).
Run you first backup, and check it
It's time for the first run ! If you don't want to wait for the cronjob, execute your first run with the command backupninja -d -n
If you see no errors, Bravo ! you have configured backupninja !
Now, let's check our backup : connect to backup and list files under /home/prod_server/data/
You should see the files created by duplicity, you cannot use it without some commands :
- Show the collection status : duplicity collection-status file:///home/prod_server/data
- List files in archive : duplicity list-current-files file:///home/prod_server/data
- Restore the latest backup in a specific directory duplicity restore file:///home/prod_server/data/ /home/prod_server/test-restore/ (if you see errors but files are there after the command, it's because duplicity tries to chown files, while you were not root when starting the command).
Note that you must have Duplicity installed on backup on in order to use these commands on backup.
Also note that instead of using file:// on backup, you can run the command on production and use the ssh desturl.
Conclusion
You should now be able to implement a complete backup solution, using backupninja.
I hope this post was useful to you, you can write a comment below if you experienced problems with this how-to.
Commentaires
will duplicity create incremental backups of mysql dumps too?
As the MySQL backups are zipped, I am afraid that they will not be as incremental as they could be.