A database gets slow. Not down, just slow enough that queries start timing out.
The service in front of it retries. So does the service in front of that one. And the client. Nobody coordinated this, they are all just doing the sensible thing, which is trying again after a failure.
The maths on that is worse than it feels. Three retries across three service layers is 27 times the original load, arriving at a dependency that was already struggling. The retries are what finishes it off.
Meanwhile your pager receives fifty alerts.
Every one of those alerts is correct
This is the part that makes it hard to fix by tuning.
The web tier really is returning errors. The cache layer really is timing out. The API gateway really is seeing elevated latency. Every threshold that fired was set sensibly and is reporting something true.
There is nothing wrong with any individual alert. The problem is that fifty true statements about symptoms have arrived without the one statement that matters, which is that a database got slow and everything downstream is a consequence.
You can raise thresholds until the noise stops, and what you will have built is a system that misses the next incident. The alerts were not wrong. They were unstructured.
The shape of the failure is the nasty bit
Retry cascades do not degrade in a way that gives you warning.
Recent work on this makes the point that systems hold, and then drop sharply past an inflection point. There is no gradual slope to catch. It is fine, and then the feedback loop closes and it is not.
Which is also why staging tells you nothing here. Load testing at a level below the inflection point demonstrates that the system is fine below the inflection point.
The same analysis found something worth knowing if you are running anything agentic: Datadog measured rate limit errors accounting for around 60% of all LLM call span errors in February 2026, roughly 8.4 million in a single month. Agents fan out into many tool calls and retry aggressively, which is a retry storm generator wearing a new hat.
The counterintuitive fix
Once you understand the amplification, the mitigations follow.
Exponential backoff with jitter, because synchronised retries are what create the waves. Circuit breakers, so a failing dependency stops receiving traffic entirely and gets a chance to recover. Retry budgets, capping total retries across all clients rather than per client.
There is a trap in the circuit breaker, though, and it is a good one to know. If an open breaker immediately returns 503, an uncoordinated upstream client will simply retry, and you have moved the herd one hop up rather than stopped it. Same with a single-probe half-open state, which lets one request through, sees it succeed, reopens the floodgates and collapses again.
But the line that reframed this for me is simpler than any of the patterns. During an outage, the best thing most clients can do is not retry.
That runs against every instinct. A failed request feels like something you should do something about. In a cascade, doing nothing is the contribution.
Now the pager problem
Fixing the cascade is an application architecture job. What arrives at your on-call rotation is a separate problem, and it needs a separate answer.
Fifty alerts is not a volume problem you solve by sending fewer. It is a structure problem. When a database goes down, you do not need fifty separate alerts for the web servers, cache layers and API gateways that depend on it. You need one alert pointing at the database.
Getting that requires knowing which things depend on which. Correlation by time alone gives you a cluster of things that happened together and no idea which one caused the others. Correlation by topology can say that these forty-nine are downstream of that one.
This is where Shankh sits. It takes the signals, works out which are consequences of which, suppresses the downstream noise, and pages once with the thing that actually needs attention. The ownership mapping matters as much as the suppression, because a correlated alert is only useful if it reaches the team that owns the service at the root rather than the team that owns the loudest symptom.
The reason that is not just a filtering rule is that it depends on knowing your topology. A suppression list maintained by hand is a set of guesses about what depends on what, frozen at the moment somebody wrote it.
Suppressed is not deleted
One implementation detail I would insist on, and it is worth checking in whatever you use.
Suppressed alerts should still appear in the incident timeline. Not paged, but recorded and visible.
Two reasons. During the incident, the shape of what got suppressed tells you the blast radius. Forty-nine downstream alerts is a different situation from four, and you want that visible without being woken forty-nine times.
Afterwards, the suppressed set is your evidence about what was actually affected. A system that drops them is destroying the record of the incident in order to keep the pager quiet, which is a bad trade you will not notice until the postmortem.
The guidance I have seen suggests aiming for a suppression rate above 80% during cascading failures, which sounds aggressive until you look at what fifty alerts from one cause actually costs you.
What I would check
Count the alerts from your last cascade, and count the causes. If the ratio is anywhere near ten to one, correlation is worth more to you than any amount of threshold tuning.
Look at whether your retries have jitter. Plain exponential backoff still synchronises. Randomising by roughly a quarter is what breaks the waves.
Find out what your circuit breakers return, and what the caller does with it. An immediate 503 into a retrying client is not protection, it is relocation.
Check that suppressed alerts are still recorded somewhere. If the answer is that they are dropped, you are trading your postmortem for your sleep.
Then ask what your correlation is based on. If it is time windows, it will group things correctly and still not tell you which one to fix. If it is topology, it can, and that only works if the topology is derived rather than declared.
The unglamorous truth about most alert fatigue is that it is not caused by bad alerts. It is caused by good alerts arriving without any relationship between them, and asking a human at 3am to work out which one is the parent.
Author note
The thing I keep coming back to is that during a cascade, everything is behaving exactly as designed. The retries are correct. The alerts are correct. The thresholds are correct.
That is quite a specific kind of failure. Nothing malfunctioned. A set of individually reasonable behaviours combined into something none of them intended, and the fix is not to make any of them better.
Mohan Gopi, Associate DevOps Engineer at Frigga Cloud Labs, working across AWS, GCP and Azure. LinkedIn.