Went to do a test restore of one of my databases and I noticed the dump files over the last few months were all 0kb. Glad I caught it this way and not because I needed to restore. Put it on your calendar, schedule a test restore of your critical stuff a couple times a year. I know y’all are busy but it is worth the time and effort. A backup you can’t actually restore isn’t a backup at all.
I started out in a big iron shop in the 90s. I was in charge of backups. We had an outage after a power loss, and the generator not kicking in. Our local backup set didn’t restore. The basement set, in a fire/waterproof safe didn’t restore. The off-site set restored.
The hours between the first fail and success were pure terror.
Also, if applicable, have a different person perform the restore every time and have them do it just by following the documentation. This way multiple persons have actual experience with the process if the shit ever hits the fan and this also makes sure the documentation is accurate and up-to-date.
3-2-1…
Three copies of your data, in two physical locations, equals one backup. And it’s not real until you do a test restore yes indeed!
My understanding of 3-2-1 is different;
3 copies of your data, on two different storage mediums, with one offsite.
E.g. SSD live copy, hot HDD backup, cold HDD backup offsite.
Is this wrong?
Afaik that’s the common read on 3-2-1 though im wondering these days if the “separate mediums” still is that relevant to me it means storing data on different types of disks however in the end all is zfs for me so not really a different medium (since same file system)? Anyway I still have to set up my off-site backups anyway to adhere to 3-2-1 :D
Ain’t got time for that
Instead, I let my faith carry me though
We do a restore test once a quarter, but we are SOC2 so we must do it. Doesn’t have to restore an entire VM, just a random used file.
A few months after setting up the backups for my server to two remote locations and patting myself in the back for it, I woke up in the middle of the night realizing I had no idea how to restore.
I worked at a datacenter that wanted to change backup vendors, as we dug into the details we found out that the agent based backups needed an agent running on the machine to restore to and they didn’t have a linux agent. Despite this obvious problem mgmt chose this vendor. It didn’t take long before sysadmins were rebuilding linux boxes from scratch in the wee hours. I left shortly after.
I worked at a small shop and every week I took the backup and booted up the backup hardware we had with a dumb hub and would connect my laptop and make sure everything work. I would then exchange the backup drives in the bosses office who was to take them home on the weekend and bring back the older backups from the week before on monday. For the life of me I could not seem to get them to understand the important of this which got worse when my boss left (as he got it as was part of the leadership circle). The owner had this thing to with having external drives and expecting each employee to back up to it and did not get why it was important they backup to the network drive where backups were tested.
Rules of Tech Support
Rule T9C - If you can’t restore from it, you don’t have a backup.
Rule T9D - If you haven’t tested your backup recently, you don’t have a backup.
How do you test one
It depends on the type of backup:
For a filesystem backup, restore one or more files to a secondary location. E.g. pick a few files out of the backup and try to restore them to a temporary folder. Then hash the original and restored files to verify integrity.
For a full machine backup (e.g. VM backup), restore a copy of the machine to a test location. Spin up the test machine to verify that it can boot.
For a database backup, restore a copy of the database to a test location (e.g. change the database name as part of the restore process), compare a few tables against the real database to verify integrity.Pretty much, it’s going to be some version of “Restore X to a test location and verify integrity”. You want to both prove that the backup can be restored and that the restored copy is actually intact.
Thanks for explaining this! Very helpful and easy to understand. Do you have preferred programs for the two actions? I currently just rsync my servers.
I use restic to back up files to Wasabi.
It’s all scripted but the steps are:
- Back up
- Prune based on rules
- Perform a repository consistency check and do further checks (takes longer) on 1% of the repo
- Choose 1 random file from the current backup set and get the hash of the current file
- Restore that file to a temporary location and hash it.
- Compare hashes
- Send push alert to me with success/failure and a summary
