Frigga Blogs

Chaos engineering has a premise I find genuinely convincing. Tests only check the cases you thought of. Injecting real failure into a real system finds the ones you did not. It also has a problem sitting right at the front of it, and I have not seen i…

Every alert you have is built around something going wrong. A pod crashes, a request errors, a metric crosses a line. Something occurs, and the occurrence is what fires the alert. Now think about a nightly backup that just stops running. There'…

kubectl apply came back clean. Service created. Deployment rolled out. Pods running. And every request returns 503. The thing that took me a while to accept about Kubernetes is that a manifest being valid says almost nothing about whether it c…

Read enough postmortems and you notice they all end the same way. One box, one sentence, usually naming a deployment. Root cause: bad deploy. It is tidy, it closes the ticket, and it is almost always an incomplete account of what happened. Worse, it q…

Checkout starts failing. Before anyone can fix anything, somebody has to answer one question, and everything downstream depends on it. Is this our code, or is a provider having a bad day? If it's ours, you're rolling back or shipping a fix…

An alert fires. Checkout latency, p95, over threshold. What you have been handed is a pod name and a number. What you need is a function in a file in a repository, changed by a commit, shipped by a deployment, roughly forty minutes ago. Between th…

A database gets slow. Not down, just slow enough that queries start timing out. The service in front of it retries. So does the service in front of that one. And the client. Nobody coordinated this, they are all just doing the sensible thing, whic…

There's a category of outage that makes you doubt your monitoring. Error rate is through the roof. Requests are timing out. And every panel you open looks fine. CPU is low. Memory is low. The database is up, responding, barely working at all. …

Watch the first two minutes of any investigation and you will notice that almost none of it is investigation. An alert fires for a service. Before anybody can reason about what went wrong, somebody has to establish what the service even is. Which clus…