If the worst happens, we can get you back

The problem

You've been told there are backups. They run every night. Nobody has ever restored one, and it's not clear anyone knows how.

There's probably also a server that was set up years ago by someone who has since left. It has some settings on it that nobody fully understands, and everyone is quietly hoping it never dies.

None of this is a problem until the day it is. A hosting provider outage, a compromised machine, a mistake. Then the question is how long the business can run without its systems, and nobody knows the answer, because nobody has ever tested it.

What we do

Rebuildable servers

Every server we run is built from a script. Not set up by hand and then written up in a document that slowly drifts out of date, but created by code that we can run again tomorrow and get the same result.

If a server is lost, we run the script and get it back. The script is the configuration, so nobody has to remember anything and nobody's departure takes knowledge with them. It also means the live servers, the staging server and every developer's machine are built the same way, which removes a whole class of "it works everywhere except live" problems.

Fire drills

A backup you've never restored is a hope, not a backup. So at least once a quarter we run a fire drill: take a real backup, bring the system up from it somewhere fresh, and check it works.

Doing this regularly means the process is known, the steps are written down, and we know how long it takes. If we ever have to do it for real, we've done it recently. Nobody wants to be learning that on the day.

The drill also catches the quiet failures: the backup that's been silently empty for a month, the restore step that depended on a tool that's since been removed. Those only show up when you actually try.

What it means for you

You know how long recovery would take, because it's been measured. You're not dependent on any one person's memory of how things were set up. And "are we backed up?" has an answer you can trust, because someone checked last month.