On 19 October 2025, at 11:48pm Pacific, a race condition in DynamoDB's DNS automation wiped the DNS records for the service in us-east-1.
Two processes ran concurrently. A stale plan check let an older plan overwrite a newer one, cleanup automation then removed what was left, and DynamoDB became unreachable in AWS's largest region.
The DNS problem was resolved by 2:40am, roughly three hours in.
Full recovery came at 2:20pm. Another twelve hours.
That gap is the part of incident response nobody optimises for, and I think it is the more interesting half of this story.
What happened in the twelve hours
Once DynamoDB went dark, things downstream started falling over in sequence.
EC2's DropletWorkflow Manager, which handles the leases on physical servers, lost its connection to DynamoDB. Leases timed out across the entire EC2 fleet in the region. When DynamoDB came back, DWFM tried to re-establish all of them at once and went into congestive collapse, which took another five hours or so to work through. Network load balancers had their own problems on top.
More than 140 AWS services were affected before it was over. Independent measurement put 20 to 30% of internet-facing services as disrupted. Snapchat, Fortnite, Roblox, Ring, McDonald's ordering, United's booking, HMRC. Downdetector logged over 6.5 million reports across more than a thousand services.
None of that was still happening because anybody was confused about the cause. The cause was known and fixed. The system was working its way back through a dependency chain it had fallen down.
Two different questions
I have started thinking about incidents as having two phases that ask genuinely different things.
The first phase asks what is wrong. This is diagnosis, it is what most tooling is built around, and it is what everyone means when they talk about reducing time to resolution.
The second phase asks what is still broken. Not why. What. Which services are back, which are recovering, which have not started, and what order the rest will come in.
Every tool I have used is good at the first and vague about the second. During the second phase people fall back to opening dashboards one at a time and asking in channels whether anybody has checked payments recently.
For AWS on that day, the second phase was four fifths of the outage.
Why "what is still broken" is harder than it sounds
It seems like it should be trivial. Look at the health of everything and see what is red.
Two things make it not trivial.
The first is that recovery is not simultaneous, and it is not random either. Services come back in dependency order, and if you do not know the order you cannot tell the difference between something that is still recovering normally and something that is stuck. Those two states look identical on a dashboard. One requires patience and the other requires intervention, and getting it wrong in either direction is expensive.
The second is that recovery generates its own failures. DWFM did not fail because DynamoDB was down. It failed because DynamoDB came back and everything tried to reconnect at once. That is a new incident, caused by the recovery from the old one, and it will not be explained by anything in your original diagnosis.
So the mental model of an incident as one cause producing one outage that ends when you fix the cause is just wrong for anything with depth to it. What actually happens is a wave going down through your dependencies and then another one coming back up, and the second wave is where the surprises are.
The question most teams could not answer that day
Here is what I find sobering about the customer side of this.
For thousands of companies, the entire incident was somebody noticing their product was broken, eventually working out that AWS was having a bad day, and then waiting. Their own diagnosis was correct and useless, because the cause was outside their control.
But the second question was very much theirs, and most could not answer it. Which of our services actually depend on us-east-1. Which parts of our product are degraded right now versus fully down. What do we tell customers. What comes back first when this clears.
Many of them, as the post-incident commentary noted, did not know they depended on that region at all. Global products relying on it for authentication or metadata without anyone having written that down.
The unpleasant version of this is that during a large upstream outage, your ability to respond well is decided entirely by work you did beforehand. There is nothing to diagnose. There is only what you already know about your own system.
What this asks of tooling
I would want three things, and I do not think any of them are exotic.
Health across everything at once, in one place, so that "what is still broken" is a single question rather than fifteen tabs. During a wide incident the picture changes every few minutes and it needs to be cheap to re-ask.
Dependency direction, so you can tell whether something is still down for its own reasons or because the thing beneath it has not come back yet. That distinction is the difference between escalating and waiting.
And a distinction between degraded and unreachable, because they mean different things to the people asking you for updates.
This is a reasonable description of what we built Vörr to answer, and I would rather be plain about the framing than pretend otherwise. It is not primarily a diagnosis tool in this scenario, because during an upstream outage there is nothing for it to diagnose. It is the thing that lets one person hold the whole picture while it is moving, which during a fifteen hour incident is worth more than another opinion about the cause.
What I would take away
Time your own incidents in two parts. How long until you knew, and how long until it was over. If the second number is much larger than the first, more diagnostic capability will not help you, and it is probably where your effort should go.
Write down what you depend on that you do not control, including the boring things like authentication and metadata lookups. That list is only useful if it exists before you need it.
And look at your last recovery for failures that the recovery itself caused. Thundering herds on reconnect, queues that filled while you were down, caches that all expired at once. In AWS's case that was five of the fifteen hours, and it is the part that gets left out of the summary.
The race condition is the interesting engineering story. The twelve hours afterwards is the one that actually cost everybody money.
Author note
I used to read postmortems for the root cause, which I now think is the least useful part of them. The cause is usually specific to that system and will never happen to me.
The recovery is where the transferable material is. How they worked out what was still broken, what order things came back in, what the fix broke on its way through. Nobody writes that part up as carefully, which is a shame, because it is the part I keep needing.
Mohan Gopi, Associate DevOps Engineer at Frigga Cloud Labs, working across AWS, GCP and Azure. LinkedIn.