4

There are several compliance and certification criteria that mention "multi-failure" disaster recovery. What exactly is the definition of this? A google search did not turn up a clear answer for me.

  • 1
    I suppose it depends on the criteria, but presumably if X goes down you have a backup. While X is down Y goes down, and the backup for Y is compatible with X's backup. Or if X's backup goes down you have a backup of the backup. Might help to put the question into context a bit. – Steve Feb 13 '14 at 21:25
  • I think it means that there is no single point of failure in your backup scheme. You could have multiple failures in your disaster (tape backup is down, and your offsite storage was hit by a tsunami) – Daisetsu Feb 13 '14 at 21:56
2

Multi-failure tolerant systems are exactly that; Systems that are designed such that they can handle multiple failures through multiple/layered redundancies. You won't find a more detailed definition of this in any RFP, compliance, audit, due-diligence document as it's a general concept and will look different for every system/service.

For those looking for a bit more:

If there is any question to your ability to handle multiple failures, you need to run some scenarios. Get a high level view of your system, try to identify any single points of failure, if you succeed, you have work to do. Assuming you didn't, identify your failure domains. At a high level, this would usually be a Data Center. Lower levels could be Racks, servers attached to a common NAS/SAN, VMs on a bare metal host, individual applications, etc. In your scenarios, randomly pick a piece of hardware, application, rack or other object in your system architecture and mark is as dead. Also mark anything that relies on that object dead. If it's a rack, everything in that rack is dead, if it's a database cluster, every thing that needs data from that source is dead. If its a data center, everything there is dead. You get the idea. Now, with that object dead, and it's failure domain dead, add another random object to the down list, along with it's dependencies.

If you find yourself wanting to avoid certain parings of objects in these scenarios, you probably have some work to do. If you find that for the vast majority of combinations of two or three random objects marked offline, your service is still online and clients are happy, you've done well. Most companies will be able to identify at least a couple areas where they need improvement.

If you have three data centers with complete copies of servers/apps/storage in each dc, you can easily check yes on this question. But a lot of companies don't operate with three production capable sites.

As a related note, while AWS, Rackspace, etc offer multiple availability zones, etc... At a high level, any single service provider is still a single failure domain. This is a common note that pops up on these types of scenarios. This may or may not be a risk your business can accept, but you should at least be aware of it and actively make that decision. Numerous cloud based companies use more than one provider or use a cloud provider and in-house resources. It's also really sad when you see a company offline because they had all of their resources in one availability zone when it went down.

Disaster Mitigation Planning is all about identifying failure domains and ensuring there is n+1 replicas of any subsystem, where 'n' is the number of failures you have at any given time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.