On this page
What Is Chaos Engineering?
Chaos engineering deliberately injects failures into a running system to find weaknesses before they cause outages. Learn the principles and how to start.
Quick answer
- Chaos engineering runs controlled experiments that inject failures into production to verify the system survives them.
- It follows the scientific method: state a hypothesis, inject a fault, and measure the blast radius.
- Start small and safe — kill a redundant instance, not the database — and grow from there.
What is chaos engineering?
Chaos engineering is the practice of testing a system’s resilience by deliberately introducing failures — killing instances, adding latency, exhausting disk — and observing how the system behaves. Popularized by Netflix’s Chaos Monkey, it flips testing on its head: instead of assuming the system works and looking for proof, you assume it will fail and find out how it fails.
How does it work?
A chaos experiment follows the scientific method. Define a steady state (what “healthy” looks like, measured in metrics). Form a hypothesis (“if one web server dies, users see no errors”). Inject a real fault in a controlled way. Observe the metrics to confirm or refute the hypothesis. Then fix what broke and rerun. The blast radius is kept deliberately small — a redundant replica, a staging-like slice — until confidence grows.
Why does it matter?
Complex systems fail in ways no design review predicts. Chaos engineering surfaces those weaknesses on your schedule, with the team watching, rather than at 3 a.m. It also builds the muscle memory of incident response and proves that redundancy actually works instead of merely existing.
Where this bites vibecoders
A vibecoder’s app often has untested assumptions baked in: “the cache is optional,” “one database is enough,” “the retry will save us.” Chaos engineering is mostly overkill for a solo project, but its core habit — asking “what happens if this dies?” and then testing it — is cheap and valuable. Kill a dependency in staging and watch what actually happens.
Where AI coding assistants get this wrong
- Generating chaos tooling that injects high-blast-radius faults into production on day one.
- Treating chaos as random destruction rather than hypothesis-driven experiments.
- Injecting failures with no observability in place, so the result is unmeasurable.
- Skipping the steady-state definition, making “did it survive?” unanswerable.
Checklist
- Define steady state in metrics before injecting anything.
- Start with the smallest blast radius that still tests the hypothesis.
- Run experiments during business hours with the team on call.
- Record results and fix the weaknesses you find.
- Grow the fault types and blast radius as confidence increases.
FAQ
Is chaos engineering the same as fault injection?
Fault injection is a technique; chaos engineering is the broader practice of hypothesis-driven resilience experiments. Fault injection (adding latency, killing processes) is the tool you use inside a chaos experiment.
Do I need to run experiments in production?
The most valuable experiments run in production because that’s where the real failure modes live, but you should start in staging and graduate only after you can measure blast radius confidently. A small, controlled production experiment is standard practice for mature teams.
How is this different from load testing?
Load testing answers “how much traffic can we handle?” Chaos engineering answers “what happens when something breaks?” They’re complementary: one stresses capacity, the other stresses resilience.