Half the Answer Is in the Code and Half Is in Production


Every investigation I have been part of needs two kinds of knowledge, and they never live in the same place.

The first is what production is doing. Error rate, latency, memory, what deployed and when, which pods are unhappy. That is Grafana, or Datadog, or whatever you have.

The second is what the code does. Which function handles this path, who calls it, what changed in it recently, whether the failing branch is even reachable from where the error appears. That is your repository.

Neither half is an answer. A spike in 500s tells you nothing about cause. A recently changed function tells you nothing about whether it is the one failing. You need both, and getting both means switching, repeatedly, under time pressure.

The switching is most of the work

There is a study of debugging episodes that I keep coming back to, because the finding is so much worse than I expected.

Researchers measuring how developers actually spend debugging time found that 74% of it went to activity other than looking at code: interacting with the development environment, navigating file systems, installing things, opening other tools. Not thinking about the bug. Operating the machinery around the bug.

That lands on top of what we already know about comprehension. Multiple studies put the share of developer time spent understanding existing code at somewhere between 58% and 70%. Reading, not writing.

And the recovery cost is real. Gloria Mark's long-running work at UC Irvine found it takes an average of 23 minutes and 15 seconds to fully return to a task after an interruption. More recent measurement puts recovery from an on-screen interruption at 10 to 15 minutes for bug fixes specifically.

Multiply that by the number of times an incident makes you cross from monitoring to code and back.

This is not ordinary context switching

The usual advice about context switching assumes you are moving between unrelated tasks. Batch your notifications, protect focus blocks, and so on. None of that applies here, because you are not switching tasks. You are switching representations of the same task, and the two representations do not share vocabulary.

The service is checkout-service in Grafana. The repository is checkout, or payments-api, or something a team renamed eighteen months ago. The failing version is a container tag. The code you are reading is a branch. The deploy that went out at 14:32 does not announce which commit it carried unless someone wired that up.

So every crossing costs you a small act of translation, and each one is an opportunity to translate wrongly and spend ten minutes reading the wrong file with complete confidence.

Holding both at once

This is the part of Vörr I use most, and it is the reason I stopped keeping eleven tabs open during an incident.

It reads code and operations through the same interface. I can ask where a function is defined and get the repository, the path and the line. I can trace a function through its callers and its callees, which is the question that actually matters when you are deciding whether a change could have caused what you are seeing. I can find every place a symbol is used across the whole organisation's code, not just the repository I happen to be in, which is how you discover that the thing you were about to change has four consumers nobody mentioned.

And I can ask what that same service is doing right now without leaving.

There is a heavier version for when the question is bigger than one function. Rather than opening six files in sequence, it assembles a context pack for a feature or a flow in one go, which is closer to how you actually need to understand something. A single file rarely contains a behaviour. A behaviour is spread across a handler, a service, a client and a config value, and reading them one at a time is how you lose the thread.

What ties it together is provenance. Every answer comes back with the repository, the path, and the line it came from. That matters more than it sounds, because an answer you cannot trace to a specific line is a claim rather than evidence, and during an incident the difference between those two things is the difference between fixing something and guessing at it convincingly.

What it will not tell you

The code says what happens. It does not say why anybody wanted it to.

If a retry limit is set to three because of an incident in 2023 that nobody wrote down, no amount of static analysis will surface that. The reasoning lives in a pull request comment, a Slack thread, or the head of someone who has since left. That gap is real and it is not a tooling problem.

Reading the code faster also does not make you right. It removes the mechanical cost of finding things, which frees attention for judgement. It does not supply the judgement.

What I would take from this regardless of tooling

Name things consistently across the boundary. If your service is checkout-service in monitoring, the repository should not be called something else. This sounds trivial and it costs real minutes in every incident, forever, because the translation happens at exactly the moment nobody has spare attention.

Put the commit in the deploy record. Whatever your CI system is, make the artefact carry the commit it was built from, and make that visible next to the deploy event. The single most common question during an incident is what shipped, and it should not require joining two systems by timestamp.

Then look at where your time actually goes during an investigation. Not the thinking, the operating. If most of it is spent finding things rather than understanding them, that is the 74% showing up in your own week, and it is more fixable than it looks.


Author note

I have started judging tools by how many tabs they let me close rather than by what they can do. It is a crude measure and it has been more predictive than any feature list.

The thing that shifted my thinking here was realising that the expensive part of an investigation is rarely the reasoning. It is the fetching. Finding the file, matching the service to the repository, working out which commit was live at 14:32. None of that is hard. All of it is slow, and it happens at the worst possible time.

Mohan Gopi, Associate DevOps Engineer at Frigga Cloud Labs, where I work on Vörr. LinkedIn if you want to argue about any of this. 

Post a Comment

Previous Post Next Post