Frigga Blogs

Chaos engineering has a premise I find genuinely convincing. Tests only check the cases you thought of. Injecting real failure into a real system finds the ones you did not. It also has a problem sitting right at the front of it, and I have not seen i…

Every alert you have is built around something going wrong. A pod crashes, a request errors, a metric crosses a line. Something occurs, and the occurrence is what fires the alert. Now think about a nightly backup that just stops running. There'…

kubectl apply came back clean. Service created. Deployment rolled out. Pods running. And every request returns 503. The thing that took me a while to accept about Kubernetes is that a manifest being valid says almost nothing about whether it c…

Read enough postmortems and you notice they all end the same way. One box, one sentence, usually naming a deployment. Root cause: bad deploy. It is tidy, it closes the ticket, and it is almost always an incomplete account of what happened. Worse, it q…

Checkout starts failing. Before anyone can fix anything, somebody has to answer one question, and everything downstream depends on it. Is this our code, or is a provider having a bad day? If it's ours, you're rolling back or shipping a fix…

A database gets slow. Not down, just slow enough that queries start timing out. The service in front of it retries. So does the service in front of that one. And the client. Nobody coordinated this, they are all just doing the sensible thing, whic…

There's a category of outage that makes you doubt your monitoring. Error rate is through the roof. Requests are timing out. And every panel you open looks fine. CPU is low. Memory is low. The database is up, responding, barely working at all. …

The thing nobody warns you about with your first few incidents isn't the pressure. It's the tab count. You start with an alert. By minute four you've got Grafana open, a terminal running kubectl , your CI system in another tab, GitHub in…

A node memory alert goes off. The question that follows is always the same, and it's never as easy to answer as it should be. Which workload is doing this? The usual route is kubectl top pods , sorted by hand, one namespace at a time. If you…