Another fun mail to start some discussion. ;)
The topic this time is backups.
Currently, we do backups two ways:
1. We have a backup server in phx2 with a tape drive running bacula. It
backs up machines and spools the backups to tape. It backs up the
following:
ask01
bastion01
bastion02
collab04
db01
db04
db05
db-fas01
fas01
hosted04
hosted-lists01
lockbox01
log02
nfs01
people03
pkgs01
proxy01
proxy02
releng03
releng04
relepel01
It does a subset of data on those, but for the most part all of them.
2. We have a backup server at ibiblio. It backs up much more
selectively. It keeps lvm snapshots of 3 days worth of these backups
(so there's the current one, and 2 older ones available):
db01
db04
db05
db-fas01
(only database backups from those)
pkgs01
(git and lookaside)
lockbox01
(git repos and infra rpms)
We are finally getting some more storage in phx2, and I thought this
might be a good time to look at backups and revamping them.
I'm going to get a backups volume added to backup03 in phx2.
I'd like to install rdiff-backup there and have it run backups to disk
there, and then change the tape backups to just backup the rdiff backup
data to tape. I'd like to also rethink what we backup, and restrict
rdiff-backup to /etc and /home and /actual-data. If we are restoring
an instance we would do a new install, ansible or puppet and then
restore data, so it doesn't make much sense to me to backup system
binaries, etc. With rdiff-backup we could keep a pretty long history if
needed.
Some of our instances don't actually have any data on them, thats all
in database. Logs should hopefully go to log02, so if we back that up
we should have all those.
I'm not sure what we could do better on external backups, but open to
ideas there. Should we add or remove anything there?
Thoughts? Comments?
kevin