Frigga Blogs

The thing nobody warns you about with your first few incidents isn't the pressure. It's the tab count. You start with an alert. By minute four you've got Grafana open, a terminal running kubectl , your CI system in another tab, GitHub in…

A node memory alert goes off. The question that follows is always the same, and it's never as easy to answer as it should be. Which workload is doing this? The usual route is kubectl top pods , sorted by hand, one namespace at a time. If you…

Watch how an investigation actually begins. An alert fires. Somebody in the channel says "I bet it's the database." Within ninety seconds four people are looking at database metrics, somebody is checking connection pool saturation, and a…

Open your Grafana instance and look at the list. Not the two or three you actually use. The whole list. Most of those panels have an origin story, and it is nearly always the same one. Something broke, nobody could see it while it was happening, and…

Somebody on your team is going to add a label this month. It will be a reasonable label. It will make a dashboard more useful, it will answer a question somebody asked in a retro, and it will pass review without comment. Three weeks later your Prome…

Production went down at 11:02pm on a Thursday. The alert fired correctly. It went to Slack, as configured. The first person to notice something was wrong was a customer, who sent a support ticket at 11:43pm. The engineer saw the Slack notification at …

Managing infrastructure across AWS, GCP, and Azure means you are dealing with systems that generate telemetry from a lot of different directions at once. Metrics from CloudWatch, logs from GCP's Cloud Logging, traces from whatever the application …