Last updated: 31 October 2026.
Metastable failures are outages that keep going after the thing that started them has gone. A traffic spike ends, a bad deploy is rolled back, a rack is repaired, yet the system stays overloaded and useful work stays near zero, because something inside the system is now feeding its own overload. The most common feeder is a retry. The fix that engineers reach for first, such as adding more retries or restarting more servers, is often the thing that keeps the system down. This article explains the mechanism, what the research on real incidents shows, and how to design against it.
Key takeaway: A metastable failure has two parts: a trigger that pushes the system into overload, and a sustaining effect, a reinforcing feedback loop that keeps it there after the trigger is removed. Removing the trigger does not help. You must break the loop, usually by shedding load.
The idea in one picture
Imagine a ball resting in a valley. A small push moves it a bit, and it rolls back. That is a stable system. Now imagine the landscape has a second, lower valley beyond a hill. A big enough push lifts the ball over the hill, and it settles in the second valley and stays. The push is gone, but the ball does not come back. The system now has a second stable state, and it is a bad one.
Figure 1. A healthy system has a vulnerable zone where a trigger can push it into a self-sustaining bad state.
The term “metastable” was brought into the distributed-systems literature by Nathan Bronson, Abutalib Aghayev, Aleksey Charapko and Timothy Zhu in “Metastable Failures in Distributed Systems” (HotOS 2021). They describe a metastable failure as a state of permanent overload with very low goodput, where goodput is the rate of useful work that finishes successfully, as opposed to throughput, which counts work that arrives or is attempted.
What the evidence from real incidents shows
The follow-up paper, “Metastable Failures in the Wild” (Lexiang Huang and colleagues, OSDI 2022), studied 22 metastable failures from 11 organisations, drawn from public incident reports. Its findings are the best empirical grounding for the concept:
- At least four of fifteen major outages at Amazon Web Services over the previous decade were caused by metastable failures, according to the authors.
- Retries were the most common sustaining effect, present in more than half of the incidents studied.
- Triggers come in two kinds: a load spike (a sudden burst of real demand) and a capacity decrease (a failed rack, a slow dependency, or inefficient code shipped to production).
- Sustaining effects also come in two kinds: workload amplification (feedback that raises load above what users actually ask for, as retries do) and capacity degradation amplification (feedback that lowers capacity, such as garbage-collection pressure that slows every request).
Named examples in the paper include a 2011 Amazon EC2 and RDS incident triggered by a network configuration change and sustained by retries; a 2014 Amazon SimpleDB incident where a power loss started it and storage servers timing out and retrying against an overloaded locking service kept it going; and a 2021 Microsoft Azure incident in which a software bug and a load spike were sustained by retries across several services.
The common thread is that in every case the trigger was short-lived or fixable, and the outage was long because of the system’s own response.
How a retry storm sustains an outage
Consider a service that can complete 1,000 requests per second, serving ordinary demand of 800 per second. These numbers are illustrative.
- Trigger. A brief spike pushes demand to 1,300 per second for thirty seconds. A queue builds.
- Latency rises. Requests wait in the queue. Soon the wait exceeds the client timeout, say two seconds.
- Wasted work. Clients give up, but the server still processes the queued requests whose answers nobody is waiting for. Goodput drops even though the server is busy.
- Retry amplification. Each timed-out client retries, up to three attempts. Effective offered load becomes up to four times the base rate of users, far above 1,000 per second.
- Sustained overload. The spike has ended and organic demand is back to 800, but the retry traffic plus the stale queue keeps the offered load above capacity. The system is metastable.
This is a textbook reinforcing loop: more overload creates more timeouts, which create more retries, which create more overload. The loop is described in general terms in reinforcing and balancing feedback loops. Mechanically it combines with a delay: the queue has memory, so the system stays overloaded long after the demand itself has normalised. For why delays make feedback systems behave counter-intuitively, see delays in systems thinking.
Other sustaining effects
Retries are the headline, but the OSDI paper’s taxonomy shows the same shape in other forms:
- Cache collapse. A cache warms slowly. After a restart or eviction storm, many requests miss and hit the database, which slows, which makes more requests time out and miss. The database runs at its limit and the cache never refills.
- Garbage-collection and memory pressure. A busy service allocates more, spends more time in collection, serves fewer requests, and queues build, which increases memory pressure further.
- Failover cascades. When a node fails, its load shifts to its neighbours, which pushes them over capacity, so they fail and shift their load onward. This is close to the cascading failures described in cascading failures in AI agent systems, in a different substrate.
- Expensive-path switching. Under stress, some systems fall back to slower code paths such as full reads or synchronous calls, which raises the per-request cost just when capacity is short.
Why they are so hard to catch before they happen
Three properties make metastable failures slippery.
They hide in efficient systems
A system run close to its capacity limit has more value per dollar and a smaller margin for absorbing a trigger. Efficiency optimisation moves the system toward the vulnerable zone. This trade-off is the core of resilience versus efficiency: headroom looks like waste until it is needed.
Normal testing does not reach them
Load tests usually ramp up steadily and then stop. A metastable failure needs a spike, followed by recovery, to expose whether the system returns to normal or stays stuck. Tests that do not include the “after” phase miss the whole problem.
The obvious remedies make them worse
More retries, a restart that sends all traffic back at once, or autoscaling that is too slow to help can all add load to a system already beyond its capacity. In a metastable state, the standard instinct to “try again” is the sustaining loop.
Design principles that prevent or end metastability
| Technique | How it breaks the loop |
|---|---|
| Retry budgets | Limit retries to a small fraction of recent traffic, so retry load cannot multiply base load. |
| Exponential backoff with jitter | Spaces retries out and breaks synchronisation so clients do not retry in waves. |
| Load shedding | Rejects excess work early and cheaply, so capacity serves requests that will complete. |
| Circuit breakers | Stop calling a failing dependency for a while, giving it room to recover. |
| Deadline propagation and cancellation | Drops requests whose caller has already given up, preventing wasted work on stale queue entries. |
| Adaptive concurrency limits | Cap in-flight work to what the service completes quickly, keeping queues short. |
| Newest-first queueing under overload | Serves the freshest requests, whose callers are still waiting, instead of stale ones. |
| Cache warm-up controls | Gradually admit traffic after restarts so the cache fills without crushing the backend. |
The unifying idea is to make the system’s response to overload cheaper than the overload itself. A well-behaved system under stress does less work per request, not more.
Recovery: what to do once you are inside one
If a system is already stuck, the usual playbook is to reduce offered load below capacity long enough for queues to drain and caches to refill:
- Confirm the pattern: high load, low goodput, rising retries or timeouts, and the original trigger already gone.
- Cut load aggressively. Shed traffic at the edge, disable non-essential features, or turn off client retries if you control the clients.
- Drain or discard stale queues, so the service stops spending capacity on answers nobody wants.
- Restore traffic in stages, and watch goodput as well as throughput.
- Only then investigate the trigger. Fixing the trigger first often changes nothing.
Beyond software
The same structure appears wherever a system has a second stable state that it can be pushed into. Hospitals under capacity pressure cancel procedures that return later as additional demand. Traffic networks jam beyond the point where throughput collapses. Supply chains hoard during shortages and so deepen them. In each case a temporary shock leaves a lasting condition because the response feeds the problem. The general lesson is the one that systems thinking keeps returning to: look for the loop that sustains the state, not only the event that started it.
Frequently asked questions
What is a metastable failure in simple terms?
It is an outage that persists after its cause is removed, because the system’s own reactions, mainly retries, keep it overloaded.
How is it different from a cascading failure?
A cascading failure spreads from component to component. A metastable failure can occur in a single service and is defined by a self-sustaining loop. They often occur together, since failover and overload cascades can be sustaining effects.
Does autoscaling solve it?
Not by itself. Autoscaling is slow compared with a spike, new instances start cold with empty caches, and added capacity may still be below the amplified load. It helps only if combined with load shedding and retry control.
How do I test for metastability?
Run spike tests that include a recovery phase: apply a short overload, remove it, and check whether goodput returns to baseline. Also test capacity-decrease triggers such as killing a fraction of instances or slowing a dependency.
Conclusion
Metastable failures show that reliability is a property of feedback structure. The trigger is rarely the real story; the sustaining loop is. Teams that budget retries, shed load early, bound queues and test recovery, not just peak load, remove the loop that turns a minor spike into a long outage.
Sources and further reading
- Huang, L. et al., “Metastable Failures in the Wild”, USENIX OSDI 2022.
- Bronson, N., Aghayev, A., Charapko, A. and Zhu, T., “Metastable Failures in Distributed Systems”, HotOS 2021.
Related concept: Metastable failure in our glossary.
Discussion