There's a category of outage that makes you doubt your monitoring.
Error rate is through the roof. Requests are timing out. And every panel you open looks fine. CPU is low. Memory is low. The database is up, responding, barely working at all. Nothing is red except the thing that's actually broken.
The first time you hit one of these, you assume the monitoring is lying. It isn't. You're just measuring the wrong thing, because the limit that broke isn't a resource. It's a number in a config file.
The pool, not the database
Connection pool exhaustion is the classic version of this and it catches people repeatedly.
Your app doesn't open a database connection per request, because that's expensive. It keeps a pool of them and hands them out. When every connection in the pool is checked out, new requests queue up and eventually time out.
The database itself is frequently healthy throughout. Spare CPU, spare memory, nothing to complain about. The bottleneck is connection availability, and that's capped by whichever is smaller: your app's pool size or the database's own max_connections.
Neither of those is a resource metric. They're both settings, and settings don't appear on a utilisation dashboard.
Read the error before you do anything else
This is the single most useful thing I've picked up on this, and it takes about five seconds.
The two errors look similar and point at completely different fixes.
too many connections, or remaining connection slots are reserved, means you've hit the database's max_connections. The database is refusing you.
A client-side pool timeout or unable to acquire connection means your application's pool ran dry first. The database never even saw the request.
One is a database limit. One is an app config. Raising the wrong one does nothing, and I've watched people spend twenty minutes on the wrong side of that line because the symptom was identical from the outside.
Defaults are usually wrong
The thing that makes this so common is that the default is almost never right, and it's documented as not being right.
HikariCP's own documentation says the default maximum pool size is 10 and that this is rarely sufficient for production. It's written down, in the docs, and it still takes services down because nobody reads pool configuration until production is on fire.
There's a related trap with horizontal scaling. Ten app instances each with a pool of 50 is 500 connections demanded from a database that might be capped at 150. Scaling out made it worse, not better, and the change that caused it looked completely routine.
It cascades faster than you'd expect
The reason a small mismatch becomes a total outage is retries.
A request times out waiting for a connection. The client retries. The retry asks for a connection from a pool that's already empty. That pushes the pool further behind, more requests time out, more retries fire. LinkedIn had a four-hour outage and Stripe saw payment failures from exactly this amplification pattern.
Within seconds, moderate load produces total failure. Which is why these incidents feel like they came out of nowhere. Nothing was gradually degrading. It was fine, and then the feedback loop closed.
Why I'd rather see limits next to usage
Coming from electronics, this is the bit that feels most familiar to me.
Measuring current through a circuit tells you nothing on its own. Two amps is fine or it's a fire, depending entirely on what the component is rated for. The reading is meaningless without the rating, and the rating is printed on the part rather than on your meter.
Software monitoring mostly shows you the meter. Connections in use, memory in bytes, requests per second. The ratings live somewhere else, in a Helm chart or an environment variable or a database parameter group, and joining the two is manual work that nobody does at two in the morning.
This is what I use Vörr for here. When I ask about a service it returns memory as a percentage of its configured limit rather than raw bytes, along with restart counts and how many pods are actually ready. That framing matters more than it sounds. Eight hundred megabytes means nothing to me under pressure because I don't remember what that service is allowed. Eighty-seven per cent of limit means something instantly.
The same applies to the resource configuration itself. Being able to pull what a database is actually configured with, next to what it's currently doing, without going to a console, is the difference between diagnosing this in two minutes and going round the houses on CPU first.
What I'd check
Read the exact error. Database limit or app pool. Everything downstream depends on which.
Count active connections against the cap. In Postgres, SELECT count(*) FROM pg_stat_activity compared to max_connections. In MySQL, SHOW STATUS LIKE 'Threads_connected'. Watch it trend under load rather than looking once.
Multiply your pool size by your replica count. Then compare that to the database cap. If the first number is bigger, you have a scheduled outage waiting for a traffic spike.
Look for connections held too long. Slow queries and external API calls inside a transaction both hold a connection open for the whole duration. Those turn a small slowdown into exhaustion.
Alert on pool utilisation, not just errors. Eighty per cent is a warning worth having. By the time errors appear, the retry cascade has already started.
The habit I'm trying to build is asking what the limit is before asking what the usage is. Usage on its own is a number without a meaning.
Author note
Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud Labs.
What I find slightly unnerving about this class of failure is how confident the dashboards are. Nothing is amber. Nothing is trending badly. The system is telling you everything is fine, accurately, about all the things it was asked to watch.
The thing that broke was a number somebody typed in a config file eighteen months ago, and no monitoring tool has an opinion about that.
If your team monitors configured limits rather than just consumption, I'd like to know how you've set it up. LinkedIn.