Six Questions, One Window



The thing nobody warns you about with your first few incidents isn't the pressure. It's the tab count.

You start with an alert. By minute four you've got Grafana open, a terminal running kubectl, your CI system in another tab, GitHub in a fourth, and you're copying a timestamp between two of them to work out whether a deploy lines up with a spike.

None of that is the hard part of the job. It's all fetching. But it's where the minutes go.

What changed for me was realising I could just ask the questions in order, in one place, and get the answers without going anywhere. Here's what that actually looks like.

Question one: is it actually down?

Alert says checkout is erroring. First thing I want is whether that's real, and how bad.

I ask Vörr for the health of checkout. What comes back is up-or-down plus an error rate, and it fans out to every source that can answer that. If there's a dedicated health-check system and there's also Grafana deriving health from metrics, I get both, per source, rather than having to pick which one to trust.

Ten seconds. And I already know whether I'm dealing with a partial degradation or a service that's genuinely gone.

Question two: what does the infrastructure look like?

Health tells me something's wrong. It doesn't tell me why.

So I ask the same question again in infra mode. Now I get CPU, memory as a percentage of its configured limit, network, container restarts, and pods-ready. Each one comes back with a flag showing whether it's breached its threshold.

That percentage-of-limit detail is the one I've come to rely on. Raw memory in megabytes means nothing to me at three in the morning because I don't remember what this service is allowed to use. Eighty-seven per cent of limit means something instantly.

Restarts climbing plus memory near the ceiling is a picture I can read without thinking. That's an OOM story.

Question three: what changed?

Nothing degrades on its own schedule. Something changed.

Vörr pulls recent deploy events across every connected CI and deploy system at once. Not Jenkins in one tab and GitHub Actions in another and Argo somewhere else. All of them, on one timeline, next to the thing that's currently broken.

This is the question that used to cost me the most, because answering it by hand means opening each system, finding the right job, and matching timestamps. Now it's a line in the same conversation, and the deploy either lines up with the incident window or it doesn't.

Question four: what actually caused it?

Here's where I stopped guessing.

Rather than picking a suspect and going to look for evidence, I ask Vörr to score the causes. It rates every category against real telemetry, memory, CPU, disk, network, deployment, dependency, database and external, then hands back a ranked list with the top candidate and the competing explanations sitting underneath it.

Two things in that output earn their place.

It tells you whether memory is trending like a leak or spiking like a shock, which are different problems with different fixes. And it surfaces the deploy immediately preceding the symptom, so if a change triggered this, it's named rather than inferred.

The competing explanations matter as much as the top one. A category that scores near zero has been ruled out for me, and ruling things out by hand is the tedious work everybody skips when they're under pressure.

Question five: is anything else affected?

Before I touch anything I want to know what else is in the blast radius.

Vörr can tell me what checkout depends on and what depends on checkout. If the root cause is a dependency, I'm now looking at the wrong service and I'd rather know that before I restart something.

This is also the question that stops you fixing a symptom. Three services showing errors because one shared thing underneath them is unhealthy looks like three incidents until you can see the edges between them.

Question six: show me the code

By now I know it's the deploy. I want to see what shipped.

Vörr reads code as well as operations, so I can find the function, trace who calls it, and see what changed in it recently. Every answer comes back with the repository, the file path and the line number, which means I can hand it to someone else and they don't have to take my word for it.

That's six questions. One window. No timestamp copied between tabs.

Why the connected context is the whole point

Any one of those answers is available somewhere else. Grafana has the metrics. Jenkins has the deploys. GitHub has the code. Nothing in that list is secret.

The difference is that they're normally six systems with six vocabularies, and the work of joining them is done in your head, under pressure, by whoever's awake. That join is where the time goes and where the mistakes happen.

Vörr does the joining. It knows that the service in your monitoring and the repository in your source control and the job in your CI system are the same thing wearing three different names. Once that's true, an incident stops being a research project and becomes a conversation.

I'm about two years into this career and I've never had to learn the old way of doing it properly. I'm not sure I want to.


Author note

Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud Labs.

Something I've noticed watching more experienced engineers work: they're not faster at reasoning than me. They're faster at fetching. They know which dashboard, which job, which repo, without looking it up, and that knowledge took them years to build.

Tooling that removes the fetching doesn't make anyone smarter. It just means the gap between us is about judgement, which feels like the right thing for the gap to be about.

LinkedIn if you want to compare incident workflows.

Post a Comment

Previous Post Next Post