Picture a database going down for ninety seconds. Nothing dramatic, a failover, the kind of thing that should be invisible.
Every pod in your deployment runs a liveness probe that opens a connection to that database and runs a trivial query. Every one of those probes now fails. Kubernetes does exactly what you configured it to do and restarts every container in the fleet. They come back up, find the database still unavailable, fail again, and restart again.
The database recovers after ninety seconds. Your service does not, because it is now a thrashing fleet of cold containers all trying to reconnect at once. A brief dependency blip has become a full outage, and the mechanism that caused it was the one you added to improve reliability.
I have seen versions of this more than once, and the root cause is never the probe configuration. It is a misunderstanding about what a health check is.
A health check is an instruction, not a report
This is the reframe that makes everything else obvious.
We write health checks as though they answer a question about the service: am I healthy? But nothing consumes that answer as information. An orchestrator consumes it as a command, and the command is different depending on which probe asked.
A failing liveness probe means: restart this container. A failing readiness probe means: stop sending this container traffic. Those are not two ways of reporting the same condition. They are different verbs with different consequences, and the correct one depends entirely on whether restarting would actually help.
Run that test against the scenario above. The database is down. Would restarting the container help? Obviously not. The container is fine. So that check does not belong in liveness. It belongs in readiness, where the consequence is to stop taking traffic and wait, which is exactly what you want.
The question is never "is my service healthy". It is "what should happen when this particular thing is wrong".
The people who built it split the endpoint
If you want evidence that one endpoint cannot serve both purposes, look at what Kubernetes did to itself. The API server deprecated its own /healthz in v1.16 in favour of separate /livez and /readyz endpoints, for precisely this reason: a single conflated endpoint cannot tell an orchestrator when to restart versus when to stop routing.
Most services I look at still expose one /health wired into both probes. That is the default in almost every framework template, which is how it spreads.
Two opposite mistakes, both common
The failure has two mirror-image forms, and teams tend to overcorrect from one straight into the other.
The first is the check that always returns 200. It confirms an HTTP server is listening and nothing else. The process can be deadlocked on a thread pool, unable to reach any dependency, serving errors to every real request, and the probe stays green throughout. Your orchestrator believes everything is fine because you never asked it a question that could fail.
The second is the check that verifies everything. Database, cache, message broker, three downstream services, object storage. This feels rigorous and it manufactures outages. Every dependency you add to a health check becomes a single point of failure for the service doing the checking.
The Amazon Builders' Library makes this point sharply, and Lumigo's summary of it puts the mechanism plainly: a soft dependency is one you call only sometimes, and health-checking it converts it into a hard dependency. Your recommendation engine, which affects one panel on one page, goes down. Because it is in the health check, checkout goes down with it.
Load is when both mistakes collide
Under heavy traffic, probes compete with real requests for the same threads and connections. Response times stretch, probe timeouts trip, and containers get restarted precisely when you have the least capacity to spare.
The Kubernetes documentation is direct about the outcome: restarts under high load reduce your pod count, which increases load on the survivors, which makes their probes fail too. The mechanism you built to recover from failure becomes the thing propagating it.
The same holds for readiness with tight thresholds. A two hundred millisecond network blip fails one pod's check, Kubernetes pulls it from the endpoints list, the remaining pods absorb its traffic, their checks start timing out, and the cascade runs to completion. Failure thresholds exist for this. Requiring three consecutive failures before acting costs you a few seconds of reaction time and prevents an entire class of outage.
The Amazon guidance goes one step further and suggests servers should prioritise answering health checks over doing regular work when overloaded, on the grounds that being marked unhealthy while merely busy makes a bad situation considerably worse.
Deep checks are expensive at scale
There is a cost dimension people miss. Multiply your node count by your health check frequency by cross-zone configuration and the probe volume gets large quickly. If each probe runs a real database query, you have built a load generator pointed at your most critical dependency, running continuously, forever.
The AWS ELB guidance recommends a pattern I like a lot: run one background process that checks dependencies on a sensible interval and writes the result to a local file. The health endpoint then reads that file. Checks stay cheap regardless of how many probes arrive, and your dependencies see a constant, predictable load instead of one that scales with your fleet.
Fail open, and know that you are doing it
Worth knowing as a backstop rather than a strategy: an Application Load Balancer that finds every registered target unhealthy will route requests to all of them anyway.
That behaviour looks wrong until you consider the alternative, which is a bad health check taking an entire zone dark. A fleet of possibly-broken servers beats a fleet of definitely-unreachable ones. Do not design around it, but do know it is there, because it explains otherwise baffling incidents where traffic reaches instances your dashboard shows as failed.
What I would actually write
Keep liveness shallow. It should fail only when the process is genuinely unrecoverable and a restart is the correct response. No database calls. No downstream services. No dependencies of any kind outside the process.
Put dependencies in readiness. Failing there means traffic stops arriving while the process stays alive, which is almost always what you want during a dependency outage.
Use a startup probe for anything slow to initialise. It removes the need to guess at initialDelaySeconds, and it fixes the majority of CrashLoopBackOff situations where the application simply had not finished booting.
Set failure thresholds above one. Three consecutive failures is a reasonable default and it absorbs transient noise.
Return something useful in the body. A health endpoint that answers with ok and nothing else is a wasted opportunity. Include the build SHA, the version, and the instance identifier. During an incident, the question you will actually be asking is whether the fix has rolled out yet, and this is the cheapest possible way to answer it. Return 503 rather than 500 on failure, so that intermediaries treat it as unavailability rather than a bug.
Then test the probes under load, not at rest. Almost every probe configuration that causes an outage worked perfectly in staging, where nothing was competing for the thread pool.
Author note
I am Mohan Gopi, an Associate DevOps Engineer at Frigga Cloud Labs, working across AWS, GCP, and Azure with Kubernetes and Grafana in daily use. I wrote this because health checks are the rare piece of configuration where being more thorough makes you less reliable, and almost nothing in the tooling warns you about that. The pattern I keep seeing is teams wiring one endpoint into both probes because the framework scaffolded it that way, then adding dependency checks to it after an incident where something failed silently. Both decisions are reasonable on their own. Together they build a mechanism that converts a brief dependency wobble into a fleet-wide restart. Written with the consequence in mind, a probe tells your orchestrator exactly what to do and nothing more. Written as a status report, it will eventually tell your orchestrator to destroy a fleet that was working.
Let us connect on LinkedIn → Mohan Gopi