Disaster recovery and multi-region
We'll cover the following
Two numbers decide everything
| Term | Meaning | Example |
|---|---|---|
| RPO (recovery point objective) | How much data you can afford to lose, measured in time. | RPO 5 minutes: replication or backups are never more than 5 minutes behind. RPO 0 needs synchronous replication. |
| RTO (recovery time objective) | How long the service may stay down. | RTO 1 hour: restore and redirect traffic within an hour. |
Tighter numbers cost a lot more. Set them per service with the business (payments is not analytics), not one number for the whole company.
Strategies, cheapest to most expensive
| Strategy | Second region runs | RTO | RPO | Cost |
|---|---|---|---|---|
| Backup and restore | Nothing; backups are copied cross-region | Hours | Hours (last backup) | Lowest |
| Pilot light | Data replicated, core database up, app servers off (scaled up from infrastructure-as-code on failover) | Tens of minutes to hours | Seconds to minutes | Low |
| Warm standby | A scaled-down full copy that is already running | Minutes | Seconds | Medium |
| Active-active | Full stack in both regions, both serve traffic | Near zero | Near zero (async) or zero (sync) | Highest, hardest |
Active-active is a data problem
- Stateless services are easy to run in two regions. The hard part is where writes go.
- One home region per user or tenant: all of a user's writes go to their home region; failover moves ownership. Simplest, and the usual choice.
- Multi-leader writes in every region with conflict resolution (last write wins, CRDTs): always available, but you must live with conflicts and anomalies.
- Globally consistent database (Spanner, CockroachDB): consensus across regions, so no lost writes, but every write pays cross-region latency (tens to 100+ ms).
- Async replication means a failover can lose the last few seconds (RPO above zero) and risks two primaries if the old one comes back. Sync replication gives RPO 0 but slows every write and stalls writes when the other region is unreachable.
- Routing the traffic: DNS failover with health checks (clients cache DNS, so the TTL adds to your RTO) or a global load balancer / anycast.
What shows you mean it
- Untested failover is a hope. Run game days and chaos tests. Untested backups are not backups: do restore drills.
- Failback is its own problem: reconciling what was written to the failover region while the primary was gone.
- Dependencies need DR too: auth, DNS, secrets, CI, third-party APIs. A standby region that cannot log in is not a standby.
- Avoid correlated failure: separate cloud account, region and credentials for backups; limit blast radius with cells or regional isolation.
- Automate the failover, keep a human decision for cases that would lose data.
Check yourselfProduct asks for no data loss and a 30-second cross-region failover. What do you ask before agreeing?
What it is worth, and for which data. RPO 0 means synchronous cross-region commits (extra latency on every write, and writes stall if the other region is unreachable); a 30-second RTO means a hot standby with automated failover. Split the data: payments and orders may justify it, feeds and analytics can lose a few seconds. Then price the options.
Summary: say it in the interview
- "I start with RPO and RTO per service, pick the cheapest strategy that meets them, and prove it with a restore drill and a failover game day."