Kubernetes CronJob Monitoring with Cuckoo

A Kubernetes CronJob can stop doing its work without anyone noticing. The controller may skip a schedule, the Job may hit its backoffLimit, an image pull may fail, or someone may set suspend: true during an incident and forget about it. None of these send an email by default. This guide shows how to make each run report to Cuckoo so you get an alert when a run fails or simply does not happen.

How Cuckoo sees a CronJob

Cuckoo works the other way round from cluster metrics. Instead of watching the cluster, it waits for your job to call a ping URL:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start when the run begins,
• https://cuckoo.metastic.ch/ping/YOUR-UUID/0 (or any exit code) when it ends.

If no success ping arrives within the schedule plus grace time, the check goes down and you are alerted. A non-zero exit code is reported as a failure right away. This catches the case that is hardest to see from inside the cluster: no Job was created at all.

Why kube-state-metrics alone misses "never scheduled"

kube-state-metrics exposes useful series such as kube_job_status_failed and kube_cronjob_status_last_schedule_time. A typical "job failed" alert only fires when a Job object exists and fails. When the CronJob is suspended, when the controller gives up after too many missed schedules, when an admission webhook rejects the Pod, or when the whole cluster (including your Prometheus) is down, there is no failed Job to alert on. You can write staleness rules per CronJob, but they need a threshold per schedule and they share fate with the cluster. An external dead man's switch covers that gap with one check per job.

Step 1: store the ping URL in a Secret

Create a check in the dashboard, set its schedule, then store the URL:

kubectl create secret generic heartbeat-nightly-report \
  --from-literal=PING_URL='https://cuckoo.metastic.ch/ping/YOUR-UUID'

Keeping it in a Secret means the UUID is not in your Git manifests and can be rotated without editing the CronJob.

Step 2: the CronJob manifest

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "30 2 * * *"
  timeZone: "Europe/Zurich"        # stable since Kubernetes 1.27
  concurrencyPolicy: Forbid        # skip a run if the previous one is still active
  startingDeadlineSeconds: 600     # start late by at most 10 minutes, otherwise skip
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2              # up to 3 Pod attempts per run
      activeDeadlineSeconds: 3600  # kill the Job after 1 hour
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report
              image: registry.example.com/report:1.4.0
              envFrom:
                - secretRef:
                    name: heartbeat-nightly-report
              command: ["/bin/sh", "-c"]
              args:
                - |
                  curl -fsS -m 10 --retry 5 -o /dev/null "$PING_URL/start" || true
                  /app/run-report
                  code=$?
                  curl -fsS -m 10 --retry 5 -o /dev/null "$PING_URL/$code" || true
                  exit $code

The wrapper sends /start, runs the real command, reports its exit code and then exits with that same code, so Kubernetes still sees the failure and applies backoffLimit. The || true keeps a network hiccup on the ping from failing the job.

Notes on the fields:

• timeZone takes an IANA name. Without it, the schedule uses the time zone of the kube-controller-manager, which is usually UTC.
• concurrencyPolicy: Forbid prevents overlapping runs. A skipped run sends no ping, so a run that is stuck for hours will show up as late in Cuckoo.
• Without startingDeadlineSeconds, the controller refuses to start a Job after more than 100 missed schedules. Setting a deadline keeps that window small and predictable.
• activeDeadlineSeconds kills a hung Job. A killed Pod cannot send its exit code, which is why the next section uses max runtime.

Step 3: getting curl into the container

The wrapper needs curl (or wget) in the image. Options:

• Add it in your Dockerfile, for example RUN apk add --no-cache curl on Alpine or apt-get install -y --no-install-recommends curl on Debian.
• If the image is distroless, send the pings from the application code using its HTTP client.
• Send only the start ping from an init container based on curlimages/curl, and the final ping from the app.

          initContainers:
            - name: heartbeat-start
              image: curlimages/curl:latest
              envFrom:
                - secretRef:
                    name: heartbeat-nightly-report
              command: ["sh", "-c", "curl -fsS -m 10 --retry 5 -o /dev/null \"$PING_URL/start\" || true"]

Pin a specific tag instead of latest in production.

Schedule and grace advice

In Cuckoo, use the same cron expression and IANA time zone as the manifest (30 2 * * *, Europe/Zurich). Set the grace time to cover image pulls, scheduling delay and startingDeadlineSeconds, plus all retry attempts. For example, 30 to 60 minutes is reasonable for a nightly job. Set max runtime close to activeDeadlineSeconds so a Job that was killed mid-run (and so never sent an exit code) still goes down.

Troubleshooting

• Only start pings arrive: the container is probably killed by activeDeadlineSeconds or an OOM kill. Check kubectl describe job and the Pod events.
• No pings at all: run kubectl get cronjob nightly-report to check SUSPEND and LAST SCHEDULE, then kubectl get events for admission or quota errors. Test egress with kubectl run tmp --rm -it --image=curlimages/curl -- curl -I https://cuckoo.metastic.ch.
• Several fail pings per run: that is backoffLimit retrying. Each attempt reports its own result, and a later success brings the check back up.

FAQ

Does Cuckoo need access to my cluster?

No. The job makes outgoing HTTPS requests to the ping URL. You do not install an agent or open any inbound ports, and you do not give Cuckoo cluster credentials.

Should I use one check per CronJob?

Yes. Each CronJob has its own schedule and failure modes, so give each its own check and Secret. That way an alert tells you exactly which job is late or failing.

Can I send logs with the failure ping?

Yes. Send a POST with the log tail as the body, for example tail -c 10000 /tmp/run.log | curl -fsS -m 10 --data-binary @- "$PING_URL/$code". The first 10 KB are stored with the ping.