[server-admins] Server backups

[Jonathan Marsden]

Troy A. Griffitts wrote:

> Our current backup strategy uses this script:
> 
> /root/mkbackup.sh
> 
> I usually run it once a month or so to get complete backups of the most
> important stuff.  I'll put it in cron now to be sure it runs at least
> once a month.
> 
> Let me know if you think anything should be added.

I think I'm a *lot* pickier about backups than you are :)  Apologies in
advance for the length of my list of backup issues!

(1) I can't see inside /opt/firebird/data -- is there something that is
regularly exporting the firebird database content to a flat file (like
mysqldump for MySQL and pg_dump for PostgreSQL)?  If not, then a simple
tar is probably not a reliable way to back up a live database.  Has this
approach been tested by recovering the firebird database(s) on a
different machine, using only this backup?  How does it ensure that the
backed-up tarball contains the database is a consistent state, not part
way through a transaction?  I'm not all that familiar with Firebird
(more familiar with MySQL, PostgreSQL, and older versions of Oracle),
but I suspect there is an issue here that needs addressing.

(2) The set of directories backed up currently omits everything about
the system itself.  At minimum, I would suggest that everything under
/etc should be considered "most important stuff" on any Unix-like
machine, and so added to /root/backupdirs.  If you are using /root for
custom scripts then perhaps add /root also.

Including info on disk partitioning ( fdisk -lu >somefile.txt ), and
also a list of all installed RPM packages ( rpm -qa --last
>somefile2.txt ) would be wise too, in case you ever need to recover the
machine.  The size of /etc (plus a couple of text files for the
partition list and package list) is usually small relative to the total
backup size, and *very* useful indeed when doing disaster recovery under
pressure.

(3) The current set of directories backed up omits the wiki.  Perhaps we
could use my script to create a wiki backup as part of this process.
The size is small, currently the 3 .gz files it creates total under
32MBytes.  (FYI, my personal daily backups of the wiki are currently in
~jmarsden/mediawiki/ and rsynced from there to my desktop machine here).
 Keeping a full month of daily full backups of the mediawiki this way
will use under 1GB total.  Keeping one backup around 'forever' for every
month will increase that space requirement at just 32MB per month, or
384MB/year.  With over 700GB free on /backup, that's tiny :)

(4) IMO, server backups should generally be much more complete than
this, and should be stored on separate media, preferably on a remote
machine which has independent power and network access from the original
machine.  There are RPM packages such as rsnapshot which are great for
this kind of "network backup over SSH" approach, and can keep multiple
copies on the remote machine using very little space, by using hard
links to unchanging files.  Is it worth considering using something like
that, and backing up much more of the CrossWire server filespace than
just /space/svn/ and /opt/firebird/data/ ?  You could probably do
something very similar using your /dev/sdb1 backup partition, too, with
an rsync-based backup run from cron.

(5) IMO (again), backups should generally be made more frequently than
monthly, such as daily.  You can use rsync or differential backups etc,
you don't need to copy stuff that has not changed every time, but you
should (IMO) back up considerably more frequently.

For rapidly changing databases or similar sets of data, hourly backups
to local disk (overwritten every 24 hours) can be very useful, in
addition to the overall (remote or tape) daily backups.  Having such
hourly backups has, on occasion, saved a client of mine a *lot* of
grief, and one such incident happened this week, so this is fresh in my
mind :)  For CrossWire, daily is probably "good enough".

(6) You seem to have left /dev/sdb1 mounted on /backup.  I'm not sure
whether that is deliberate or not; the mkbackup.sh script umounts it, or
seems to (it doesn't check whether the umount succeeds).  Leaving
/backup mounted currently allows anyone on the machine to obtain copies
of all the backups there, since /backup is 0755 and the file permissions
are 0664.  That is probably undesirable, so modifying /root/mkbackup.sh
to use a more restrictive umask when creating the backup archive files
would be good, from a security perspective.

Jonathan