Your Incident Probably Did Not Have a Root Cause


Read enough postmortems and you notice they all end the same way. One box, one sentence, usually naming a deployment.

Root cause: bad deploy.

It is tidy, it closes the ticket, and it is almost always an incomplete account of what happened. Worse, it quietly throws away most of the useful material in the incident.

The Shakespeare outage is the clearest example

Google's own published postmortem in the SRE book describes a cascading failure that came from a combination of exceptionally high load and a resource leak that only triggered when searches failed on terms absent from the corpus.

A newly discovered sonnet contained a word that had never appeared in Shakespeare's works. That was the word everyone searched for.

Now try to name the root cause.

The resource leak had been there for ages and, under normal conditions, failed at a rate low enough that nobody noticed. The traffic surge on its own would have been survivable. The novel search term on its own was harmless. The outage required all three, and removing any one of them means it does not happen.

That is not a chain. It is a conjunction, and a form with one root cause field cannot represent it.

Trigger and explanation are different things

The framing I have found most useful is that an incident can have a trigger without having one sufficient explanation.

A deployment may start the failure. Capacity limits may make it widespread. Retries may amplify it. Weak isolation may widen the blast radius. Delayed alerting and an unsafe rollback may extend how long customers felt it.

Label only the deployment as the root cause and you have discarded five opportunities to improve, four of which are probably cheaper to fix than the deployment process.

The same analysis offers a comparison worth stealing. Instead of "bad deploy", something closer to: a deployment path allowed simultaneous restarts beyond available warm-cache capacity, combined with retry amplification, and alerting that did not measure customer write success.

That is longer. It is also actionable in three separate places by three different people, which "bad deploy" is not.

Safety engineering worked this out a long time ago. OSHA's guidance says investigations should not stop at a single triggering factor and notes there is often more than one root cause. NASA's mishap framework represents proximate, intermediate and root causes alongside contributing factors and failed barriers. Software does not need to adopt all of that vocabulary, but it does need one that resists a single-cause story.

Why we write them anyway

I do not think this is carelessness. Three forces push towards the single cause and all of them are structural.

The template asks for one. If your postmortem document has a field called Root Cause, somebody will put one thing in it, because that is what a singular field means.

A single cause is easier to communicate upward. "A bad deploy caused it, we have added a check" is a sentence a leadership update can carry. The honest version needs a paragraph and sounds less resolved.

And most importantly, the single cause is the one you have evidence for. Whatever system you happened to be looking at during the incident is the system that supplies your explanation.

That last one is the interesting failure, because it means the shape of your postmortem is determined by the shape of your tooling.

Your evidence decides your conclusion

Think about how the write-up actually gets assembled.

Somebody reconstructs a timeline the morning after, from a Slack thread, some dashboard screenshots, and whatever they remember. The deployment is in there because deployment events are easy to find and have timestamps. The retry amplification is not, because nobody screenshotted the retry rate. The alerting gap is not, because it is an absence rather than an event.

So the postmortem contains the contributing factors that were convenient to retrieve, and those get promoted to root cause by default.

This is where I think tooling actually matters, rather than the usual claim about faster resolution. Vorr AI holds the relationships across code, cloud, deployments and telemetry, which means reconstructing what happened is a set of questions rather than an archaeology exercise across six systems and someone's memory.

What changed in the window, across every connected system rather than only the pipeline you thought about. What else was degraded at the same time. What depends on the thing that failed, so the blast radius is a fact rather than a recollection. What the configured limits were, not just the utilisation, because "we hit the cap" and "we were near the cap" are different findings.

Every answer arrives with its source, which matters more for a postmortem than during the incident. During an incident you need to act. Afterwards you need to be able to show that the timeline is right, because somebody will disagree with it and they may be correct.

The point is not that a tool writes a better postmortem. It is that a fuller evidence set makes the single-cause story harder to tell, and the single-cause story is the failure mode.

Two practical changes

Delete the singular field. If policy requires one, allow multiple entries, or write a systemic sentence followed by the full contributing-factor list. This sounds like a documentation nitpick and it changes what people look for, because a form with three slots prompts you to go and find three things.

Ask what would have had to be different. For each factor, would removing it have prevented the incident, or only reduced its severity or duration? That single question separates the necessary conditions from the amplifiers, and it is the distinction that tells you where to spend engineering time. An amplifier that turned four minutes into ninety is often the better investment, because the trigger will be different next time and the amplifier will not be.

Worth noting the data on triggers while we are here: binary and configuration pushes account for 37% and 31% of change-caused incidents respectively. That distribution is why the industry invested in canaries and config validation, and it is a good argument for taking triggers seriously. It is not an argument for stopping there.

What this is really about

The reason single-cause postmortems keep producing incidents that feel familiar is that they fix the trigger and leave the conditions in place.

Google's own write-up on that outage flagged something similar, and I have seen the same note in other companies' postmortems: a remark that this was unfortunately very similar to something that happened before. When that sentence appears, it usually means a previous investigation stopped at the trigger, fixed it, and left everything that made the trigger dangerous exactly where it was.

The trigger changes every time. The conditions are what you actually own.


Author note

I have started reading postmortems backwards, beginning with the action items rather than the summary.

If every action item is about the trigger, the investigation stopped early. If some of them are about capacity, isolation, alerting or rollback, somebody kept asking after they had an answer that would have been accepted.

The second kind is rarer than it should be, and it is almost always the more useful document.

Mohan Gopi, Associate DevOps Engineer at Frigga Cloud, working on Vorr AI across AWS, GCP and Azure. LinkedIn.

Post a Comment

Previous Post Next Post