At 11:20 UTC on 18 November 2025, Cloudflare's core network started failing. Within minutes X, ChatGPT, Spotify and Canva were returning errors, and a decent slice of the internet stopped working.
Inside Cloudflare, the first theory was a hyper-scale DDoS attack. Their CEO wrote in an internal chat that he suspected <cite index="43-1">the big botnet flexing</cite>, referring to the Aisuru botnet that had been running high-volume attacks in the preceding weeks.
It was not an attack. Their postmortem says so plainly: the cause was a change to database permissions, made at 11:05, fifteen minutes before anything broke.
What I find worth sitting with is not that they got it wrong. It is why the wrong answer was the reasonable one.
The symptoms were lying, and they were lying convincingly
Three things pointed at an attack, and each one was independently sensible.
The failure was global and simultaneous. Every proxy in the network started returning 5xx errors at roughly the same moment, which is not how software bugs usually behave and is very much how attacks behave.
The service kept recovering and failing again. This is the detail I keep coming back to. A ClickHouse permissions grant was being rolled out gradually across the cluster, and the bot management feature file regenerated every five minutes. Depending on which node answered, the file came out fine or came out broken. So the network would recover for a few minutes and then fall over again.
Intermittent recovery under load is close to a signature of a DDoS. It is not a signature of a config change, because config changes are usually binary. This one was not, purely by accident of the rollout.
And then, by genuine coincidence, Cloudflare's status page went down at the same time. It is hosted off their infrastructure entirely, so the failure was unrelated. But if you are already wondering whether you are under attack, your status page dying is not a neutral data point.
Three signals, all pointing the same way, all wrong.
The evidence that would have settled it was elsewhere
Here is the part that has changed how I think about investigation.
Cloudflare was not short of data. They had every metric, every log, every trace. The proxy crashes were fully visible. The 5xx rate was on a dashboard. None of that was missing.
The thing that would have resolved it in one step was a permissions change on a ClickHouse cluster fifteen minutes earlier.
That change did not look like an incident. It did not fire an alert. It was routine maintenance, and it appeared in a completely different system from the one that was on fire. Nothing in the observability stack connected a database grant to a bot management file to a proxy crash, because that chain crosses three domains that are usually monitored separately.
So the investigation had all the evidence about the symptom and none about the cause, and no amount of looking harder at the symptom was going to close that gap.
The full chain reads almost absurdly once you know it. A permissions grant made a metadata query return rows from a second schema. The duplicated rows doubled the feature file. The file went from around 60 features to over 200. The proxy had a hard limit at 200. It crashed.
Five steps, four systems, fifteen minutes.
Why this pattern is not a Cloudflare problem
It would be easy to read this as a story about one company having a bad day, and Cloudflare's own writeup is unusually good precisely because it does not do that.
The pattern generalises. The symptom is loud and lives in one place. The cause is quiet and lives somewhere else. The gap between them is a boundary between two systems that nobody joined up.
Think about how often that shape appears. An error rate spike caused by a config change. A memory problem caused by a dependency upgrade. A latency jump caused by a schema migration. Every one of them is a case where the interesting evidence is in a system that is not currently on fire, and therefore not the system anyone is looking at.
Which raises the question I would actually want a reader to sit with. If a routine change in one of your systems broke a completely different one fifteen minutes later, what in your setup would connect them?
For most teams the honest answer is a person. Somebody who happens to remember that maintenance was scheduled, or who thinks to ask what changed recently and knows where to look.
What "what changed" needs to mean
Most tooling can answer "what deployed". Deploy events are easy, they are discrete, they are already in the pipeline.
That is a much narrower question than "what changed", and the Cloudflare case is exactly the gap between them. No deploy happened. A permission was granted on a database. If your change history only contains deploys, that event is invisible to you, and it was the whole story.
The useful version has to span the code, the pipelines, the infrastructure and the data layer, and it has to be correlated by time against the thing that is currently failing rather than browsed system by system.
This is the part of Vörr I would point at here. When something is degraded, it looks at what changed across everything it is connected to in the window before the symptom appeared, rather than only at the deploy history for that one service. That does not make the diagnosis automatic. It means the fifteen-minutes-earlier event in a different system is in front of you, instead of depending on somebody remembering it.
I would not claim it would have caught this specific case. Cloudflare's chain crossed into a ClickHouse permissions model, and I have no idea whether that would have surfaced. The general shape, though, of a change in one place breaking something in another, is the shape this is built for.
Two things worth taking from it
The first is about intermittency. A failure that comes and goes reads as external pressure, and that instinct is usually right. Cloudflare's did not, because of a gradual rollout that nobody designed to look like an attack. When you next see flapping recovery, it is worth explicitly asking whether something is rolling out, before you settle on load.
The second is the one I have actually changed my behaviour on. When I cannot explain a symptom, I now go looking for changes in adjacent systems before I go looking harder at the failing one. The failing service is where the evidence is loudest. It is frequently not where the evidence is.
Cloudflare had five hours and thirty-eight minutes of outage and one of the better postmortems of the year. The generous reading is that their engineers reasoned correctly from the signals available. The uncomfortable one is that the signal that mattered was sitting in a maintenance log the whole time.
Author note
I have read this postmortem three times now, and what stays with me is not the bug. It is that the wrong answer was the well-reasoned one.
That is a slightly unsettling thing to accept about incident response. Being methodical does not protect you when the evidence in front of you is genuinely misleading. The only thing that helps is widening what counts as evidence, earlier than feels necessary.
Mohan Gopi, Associate DevOps Engineer at Frigga Cloud Labs, working across AWS, GCP and Azure. LinkedIn.