Reliability & Resilience
Chaos Engineering
Chaos engineering deliberately injects failure into a production or production-like system to verify that the fault-tolerance mechanisms you built actually work — replacing "we assume this handles failure" with a tested fact.
- Popularized by Netflix's Chaos Monkey, which randomly terminates production instances during business hours, forcing resilience to be continuously exercised rather than assumed
- A chaos experiment starts with a hypothesis ("the system keeps serving normally if X fails"), a deliberately small blast radius, and an automatic abort if steady-state metrics degrade past a threshold
- Game days: scheduled, deliberate failure-injection exercises with the team watching and ready to intervene — distinct from always-on, automated chaos tooling
- Chaos engineering validates the other patterns in this domain — circuit breakers, bulkheads, retries — actually fire correctly under real conditions, not just in unit tests written against ideal assumptions
- Requires solid observability first (Monitoring And Metrics) — an experiment you can't observe teaches nothing, and might hide a real incident inside the noise
A test suite proves code does what it's supposed to under conditions the author thought to simulate. Chaos engineering inverts that: it injects real failure conditions — killed instances, added latency, dropped packets, a full region going dark — into a live system and observes whether the resilience mechanisms actually hold, rather than trusting that they will.
Running an experiment safely
| Tool type | Injects | Validates |
|---|---|---|
| Instance/pod termination | Kill a running instance | Redundancy, health checks, load balancer failover |
| Latency injection | Add artificial delay to a dependency call | Timeouts, circuit breakers |
| Error injection | Force a percentage of calls to fail | Retries, fallbacks, circuit breakers |
| Region/AZ failure | Simulate an entire region going dark | Multi-region failover (Multi Region Architecture) |