Operations, Security & Reliability
Failover, Disaster Recovery, and Reliability
Reliable failover is a rehearsed control loop that detects real impact, prevents two writers through fencing, promotes a sufficiently current copy, reroutes clients, measures loss, rebuilds redundancy, and learns within an explicit reliability budget.
- Detection must balance speed and false promotion.
- Fence before promoting.
- Promotion chooses a data-loss boundary.
- Client routing is part of recovery.
- Failover ends only after redundancy and protections return.
- DR includes regional dependencies and human operation.
- Reliability consumes a budget.
1 DETECT: declare impact, commander, timeline; freeze risky automation.
2 ASSESS: primary reachability, candidate receive/flush/replay positions, archive health.
3 FENCE: revoke route/lease/storage/credential or power off old writer; verify from independent path.
4 PROMOTE: choose most suitable eligible candidate; record expected RPO/data-loss window.
5 ROUTE: update proxy/DNS/service discovery; drain stale pools; verify writes/read-your-writes.
6 VERIFY: business invariants, unknown commits, audit/security, backup/archive, lag and objectives.
7 REBUILD: create replicas from new timeline; restore capacity and monitoring.
8 FAILBACK: only as a planned migration after evidence review.
9 LEARN: postmortem, owners, tested actions, game-day update.| Dependency | Failure question | Exercise evidence |
|---|---|---|
| Identity/secrets/KMS | Can responders and apps authenticate/decrypt? | clean-room issuance and rotation |
| Network/DNS/proxy | Can clients converge without stale writer paths? | TTL/pool drain and fence checks |
| Backup/archive/object store | Are bytes, catalog, keys independently reachable? | isolated restore to selected target |
| Artifacts/config/extensions | Can compatible database/app be rebuilt? | versioned manifests and bootstrap |
| Observability/audit | Can responders see and preserve evidence? | out-of-region dashboards and immutable sink |
| People/vendors | Who has authority and access off-hours? | game-day contacts, approvals, handoff |