A node memory alert goes off. The question that follows is always the same, and it's never as easy to answer as it should be.
Which workload is doing this?
The usual route is kubectl top pods, sorted by hand, one namespace at a time. If you don't know which namespace to start with, you're doing that across all of them. Or you open a dashboard that shows memory per namespace, which tells you which team to go talk to but not which pod to look at.
Either way you're assembling a ranking manually from a tool that wasn't built to give you one.
The question I actually want to ask
I came into infrastructure from electronics, and this is one of the places where the old habits still poke at me.
If a board is pulling more current than it should, you put a probe on the rail and you look at the actual draw. You don't read a summary of the average draw across the whole board over the last five minutes and try to reason backwards to which component is responsible.
That's roughly what we do with clusters, though. We look at aggregate memory on a node and then guess our way down to the pod.
What I want is the ranking. Heaviest first. Right now.
One question, ranked answer
This is what I use Vörr for more than almost anything else. I ask for the top memory consumers and I get them back ordered, with actual numbers, pulled from container metrics.
No namespace-by-namespace sweep, no sorting in my head.
resource: memory
groupBy: pod
window: 1h
limit: 10And the shape of what comes back:
1. checkout-api-7d9f4b 2,140 MB
2. search-indexer-3f2a1c 1,880 MB
3. media-worker-9b7e22 1,455 MB
4. checkout-api-7d9f4b8 1,390 MB
5. notification-svc-2c4d 880 MBStraight away you can see something the node graph never showed you: two replicas of the same deployment are near the top, which is a different problem from one badly behaved pod. That's a sizing question, not a bug.
The zoom level is the useful part
The thing that took me longest to appreciate is that you can ask the same question at four different levels, and each one answers a different question.
Group by pod and you find the specific replica misbehaving. Good when one instance has drifted from its siblings.
Group by deployment and you find the service that's heavy across all its replicas. That's a limits and requests conversation.
Group by namespace and you find which team or environment is consuming the cluster. Useful for chargeback, and for the awkward discovery that staging is using more memory than production.
Group by node and you find out whether you have a scheduling problem rather than an application problem. If one node is carrying everything heavy, the workloads might all be fine and the scheduler just stacked them.
Same question, four zoom levels. I switch between them constantly now, and the answer changes meaning each time.
One thing worth getting straight
Slow and heavy are not the same thing, and I mixed these up early on.
A request that's blocked on I/O, waiting for a database or an external API, is slow and consumes almost no CPU. It'll sit near the bottom of a CPU ranking while being the reason your latency graph looks terrible.
So if the complaint is "this endpoint is slow", the resource ranking probably isn't your answer. If the complaint is "the node is under pressure" or "something got OOM-killed", it is exactly your answer.
Knowing which question you're actually asking saves a lot of time.
Windows change what you see
Default look-back is 15 minutes, which is what you want mid-incident. Right now, what's heavy.
An hour is better for seeing whether something is climbing rather than just currently high. And 24 hours will show you a slow leak that a 15-minute view makes look like a normal steady state.
I usually run the short window first to deal with whatever's on fire, then the long one afterwards to find out whether it's been building all week.
What I do with it after the incident
Two things, both of which take about a minute.
Run it grouped by deployment and compare against your configured memory limits. Anything consistently near its ceiling is going to get OOM-killed eventually, and you'd rather find that on a Wednesday afternoon.
Run it grouped by namespace weekly. The list doesn't change much, and when it does change that's usually the interesting part. Something new appearing in the top five is worth ten seconds of attention.
The ranking was always sitting in the metrics. The difference is being able to ask for it in one go instead of building it by hand while a node is running hot.
Author note
Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud Labs.
My degree is in electronics, not computer science, which I used to think was a gap. Mostly it isn't. A cluster under memory pressure and a supply rail under load behave more alike than you'd expect, and both get easier the moment you stop looking at averages and start looking at what's actually drawing.
Still learning the software half. If you've got a better workflow for catching memory creep before it becomes an OOM kill, tell me about it on LinkedIn.