kubectl apply came back clean. Service created. Deployment rolled out. Pods running.
And every request returns 503.
The thing that took me a while to accept about Kubernetes is that a manifest being valid says almost nothing about whether it can function. A Service whose selector matches no pods anywhere in the cluster applies without complaint. An Ingress pointing at a service that doesn't exist applies without complaint. Kubernetes won't warn you about that mismatch at apply time.
The YAML is correct. The system it describes cannot work. Those are separate questions and only one of them gets checked.
502 and 503 are telling you different things
Before anything else, this distinction saves real time and I got it wrong for months.
A 502 means the proxy reached a backend and got something invalid back. The pod is there, it's reachable, and it's misbehaving. Crash loop, still starting, returned a TCP reset instead of HTTP.
A 503 means there was no backend to reach at all. Zero ready endpoints. The proxy tried to route and found an empty pool.
One sends you to the application. The other sends you to the wiring. They look identical in a browser and they mean completely different things.
Start with endpoints, always
For a 503, one command settles most cases:
kubectl get endpoints <service-name> -n <namespace>If ENDPOINTS shows <none>, you've confirmed there's nothing behind the Service and you can stop looking at the application entirely.
Then compare what the Service is asking for against what the pods actually have:
kubectl describe service <service-name> -n <namespace> | grep Selector
kubectl get pods -n <namespace> --show-labelsThe classic is a single character. app: myapp in the Service, app: my-app on the pods. Every key-value pair in the selector has to appear in the pod labels, and one typo means zero endpoints. It happens most often after a rename or a label change in a Deployment manifest, because somebody updated one file and not the other.
Worth knowing that EndpointSlices are what actually back this now:
kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service-name>The other four ways to get an empty pool
Selector mismatch is the most common, not the only one.
No readiness probe at all. A pod without one is considered ready the moment it starts, before the application has finished initialising. Traffic arrives at something that isn't listening yet. This is a leading cause of 503s during deploys specifically.
No overlap during a rolling update. Old pods terminate before new ones are marked ready, and for a few seconds the pool is empty. minReadySeconds and a sane maxUnavailable are the fix.
Ingress pointing at the wrong service name. Compare the backend in the Ingress spec against the services actually in the namespace. Again, nothing validates this at apply time.
Scaled to zero. If you're running HPA with a minimum of zero replicas, common with event-driven setups, there genuinely are no pods during quiet periods. That's working as designed and looks like an outage.
And if you're on a mesh, there's a fifth that isn't about your pods at all. Aggressive outlier detection can eject every endpoint after a brief error spike, turning a temporary problem into a complete one. Worth checking out lierDetection and trafficPolicy before assuming the workload is at fault.
What I want to see in one place
The reason this takes twenty minutes rather than two is that the answer is spread across objects that don't reference each other in any obvious way.
Ingress names a Service. Service selects on labels. Labels live on pods, which come from a Deployment, which came from a Helm chart, which was rendered by a pipeline run, which shipped a commit that changed a label. Every hop is a manual lookup and each one is a chance to check the wrong namespace.
This is what I use Vorr AI for here. Because it holds the relationships between services, workloads, infrastructure and the pipelines that deployed them, "what is this service actually made of and is anything behind it" is one question. It also connects back to the deploy that changed things, so when a selector stopped matching, finding out which rollout did it doesn't mean going through Helm diffs by hand.
The bit that matters most for this particular failure is being able to see configuration next to live state. A Service's selector is config. Whether endpoints exist is live state. The bug is always in the gap between those two, and most tooling shows you one or the other.
Things worth doing once
Add a smoke check that asserts endpoints exist after a deploy, not just that pods are running. A rollout that succeeds into an empty endpoint pool is a green pipeline and a broken service.
Always set a readiness probe, even a trivial one. The default behaviour of assuming readiness at start is almost never what you want.
Generate labels rather than typing them twice. If your Service selector and your pod labels are written by hand in two files, they will drift eventually. Helm values or Kustomize labels remove the typo class entirely.
Check the Ingress backend name in review. It's one line and nothing else will catch it.
The mental shift that helped me was to stop treating apply as validation. It confirms the manifest is well formed and that you had permission. It says nothing about whether the things you described can find each other, and finding each other is the entire job.
Author note
Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud, working on Vorr AI.
Coming from electronics, this one felt oddly familiar. A circuit can be built exactly to the schematic and still not work, because the schematic and the board are different artefacts and one of them has a dry joint.
Kubernetes manifests are the schematic. Endpoints are continuity. Checking the first tells you nothing about the second, and I spent a while assuming it did.