Every alert you have is built around something going wrong. A pod crashes, a request errors, a metric crosses a line. Something occurs, and the occurrence is what fires the alert.
Now think about a nightly backup that just stops running.
There's no crash. No error rate. No failing request. The thing that broke didn't produce an event, because the thing that broke was an event not happening. Your entire monitoring stack is watching for activity, and the failure is an absence.
This is not hypothetical
There's a documented case that gets cited a lot and deserves it. A backup job missed 24 days of runs and nobody noticed. kubectl get cronjobs looked completely normal. No alerts. The last successful run timestamp just sat there getting older.
The cause is a piece of Kubernetes behaviour that surprises most people the first time. If a CronJob misses more than 100 scheduled runs, the controller permanently stops scheduling it and logs a single error line. It doesn't retry. It doesn't alert. It just gives up.
The Kubernetes SIG-Network bot itself hit this threshold and ran silently dead until somebody eventually looked. It wasn't broken in any way a dashboard could show. It had stopped being scheduled, which looks exactly like a job that has nothing to do right now.
Exit code 0 is not success
The second failure mode is worse, because the job does run.
A container can exit 0 after connecting to a stale replica, processing an empty queue, or swallowing an exception. Kubernetes marks the Job successful, because as far as Kubernetes is concerned it was. The process ran and returned zero.
Your export exported nothing. Your sync synced nothing. Everything is green.
This is the one that catches heartbeat monitoring too. If your check only asks whether the job pinged, a job that pings with a count of zero passes. It ran. It succeeded by every technical measure. It did nothing useful.
And the evidence deletes itself
By the time you notice, the trail is usually gone.
Kubernetes keeps the last 3 successful and 1 failed Job by default, and deletes older ones along with their pods and their logs. Events expire in about an hour.
So the sequence goes: something stops working, weeks pass, someone asks why the data looks stale, and you go to investigate a failure whose logs were garbage collected long ago. You're left reconstructing from what the job was supposed to do rather than from what it did.
Two settings worth changing on day one, and they cost nothing:
spec:
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 10
failedJobsHistoryLimit: 5The deadline is the more valuable of the two, because it's what stops you hitting the 100-missed threshold after a period of controller downtime.
The worst version
There's a failure above all of these that's rare and genuinely nasty.
The cron daemon gets OOM-killed, or a system update disables it, or the controller pod goes unhealthy. Every scheduled job stops at once, and nothing alerts, because the thing responsible for running your alerting job is the thing that broke.
Any monitoring that lives inside the system it monitors has this problem. It's the reason the check has to be external.
Monitor for the absence of success
The pattern that fixes this inverts the question. Instead of alerting when something breaks, you alert when an expected signal goes missing.
The job pings an external endpoint on success, only on success. If the ping doesn't arrive within the expected window plus a grace period, you get paged. The job crashing, the node dying, the schedule being deleted, the controller giving up: all of them produce the same silence, and silence is what you're watching for.
0 2 * * * /usr/local/bin/backup.sh && curl -fsS "$HEARTBEAT_URL"The && matters. If the script fails, the ping never fires.
And if the job's usefulness depends on volume, assert on the output rather than the exit code. Ping with a count, and alert if it's below a threshold. That's what catches the export that ran perfectly and exported nothing.
Knowing what exists in the first place
There's a step before any of this that I skipped for longer than I should have.
You can't monitor jobs you don't know about. Scheduled work accumulates in a way that services don't. Somebody adds a CronJob for a migration and leaves it. A Jenkins job runs weekly for a report nobody reads. A Lambda fires on a schedule set up two years ago by someone who's left. None of it appears in a service catalogue, because none of it serves traffic.
This is where Vorr AI is useful to me here, and I want to be specific rather than broad about it. It enumerates the jobs that actually exist across the connected CI and deploy systems, along with when each one last ran and what happened. That's an inventory question, and it's the one that's genuinely hard to answer by hand once you have more than one system involved.
It doesn't replace the dead man's switch. External heartbeat monitoring is still the thing that pages you. What it answers is the prior question, which is what should be pinging in the first place, and that's the gap I keep finding when I go looking.
What I'd do this week
List every scheduled job you have, across every system. Kubernetes, CI, cloud schedulers, anything on a VM crontab. Most people are surprised by the length.
Set startingDeadlineSeconds on your CronJobs. One line, prevents the permanent-stop failure.
Add a heartbeat to anything whose absence would matter, starting with backups.
Then test one. Suspend a job deliberately and confirm you get paged. An untested dead man's switch is a belief, not a control.
The uncomfortable thing about this whole category is that nothing on this list is difficult. It's just that a job which stops running never asks for your attention, and everything else in your day does.
Author note
Manjunaathaa, Associate DevOps Development Engineer at Frigga Cloud, working on Vorr AI.
The idea that stuck with me here is that we've built monitoring almost entirely around presence. Something happened, measure it, alert on it. Absence needs a completely different mechanism and most teams have never built one.
Which means the failures we're best equipped to catch are the loud ones, and the quiet ones can run for weeks.