Cron Job Start and Fail Signals

A single success ping at the end of a job tells you that the job finished. It does not tell you how long it took, and it only reveals a failure after the grace time has passed. Two extra signals close those gaps: a /start ping when the job begins and a /fail ping when it breaks.

This guide explains how Cuckoo combines the three signals and gives you a small wrapper script that sends them reliably, including when the job is interrupted.

The problem with success-only pings

Consider a backup that normally runs for 40 minutes. With a success-only ping you have two bad options for the grace time: keep it short and get false alerts on slow nights, or make it long and learn about a crash hours later. You also cannot tell a job that hangs forever from one that never started.

How Cuckoo uses start, success and fail

When a success or fail ping follows a start ping, Cuckoo records the time between them as the run duration. If you set a max runtime on the check and a start ping is not followed by success or fail within that time, the check goes down. That is how a hung job gets caught even though it never exits.

The regular schedule still applies: if no ping arrives at all within period (or cron expression) plus grace, the check goes down too. The start ping covers "running too long", the schedule covers "did not run".

A reusable wrapper script

Save this as /usr/local/bin/hb-run and make it executable with chmod +x /usr/local/bin/hb-run:

#!/usr/bin/env bash
# Usage: hb-run PING_URL command [args...]
# Sends /start, then the command's exit code (0 = success) when it ends.
set -uo pipefail

if [ "$#" -lt 2 ]; then
  echo "usage: hb-run PING_URL command [args...]" >&2
  exit 64
fi

HB_URL="$1"; shift

hb_ping() {
  # A monitoring problem must never turn into a job failure.
  curl -fsS -m 10 --retry 5 -o /dev/null "$HB_URL$1" || true
}

finish() {
  local rc=$?
  hb_ping "/$rc"
  exit "$rc"
}

trap finish EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

hb_ping "/start"
"$@"

How it works:

• hb_ping "/start" announces the run before the command executes.
• The command runs as the last statement, so its exit status becomes the script's exit status.
• The EXIT trap fires however the script ends and reports that status as /ping/YOUR-UUID/<code>.
• The INT and TERM traps turn signals into exit codes 130 and 143, so an interrupted run is reported as a failure instead of silently disappearing.
• || true inside hb_ping keeps a network problem from changing the job's result.

Use it in the crontab:

0 1 * * * /usr/local/bin/hb-run https://cuckoo.metastic.ch/ping/YOUR-UUID /usr/local/bin/backup.sh

A minimal inline version

If you prefer not to install a script, the same idea fits on one crontab line:

0 1 * * * curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID/start; /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID/$?

It sends start and the exit code but has no signal handling. If the job is killed, Cuckoo still notices through the max runtime.

Using explicit /fail inside a script

Sometimes a command exits with 0 but the result is still wrong, for example an export that produced an empty file. Send /fail yourself in that case:

#!/usr/bin/env bash
set -euo pipefail
URL="https://cuckoo.metastic.ch/ping/YOUR-UUID"

curl -fsS -m 10 --retry 5 -o /dev/null "$URL/start" || true
/usr/local/bin/export-orders > /srv/export/orders.csv

if [ ! -s /srv/export/orders.csv ]; then
  curl -fsS -m 10 --retry 5 -o /dev/null "$URL/fail" || true
  exit 1
fi

curl -fsS -m 10 --retry 5 -o /dev/null "$URL" || true

Choosing max runtime and grace

• Max runtime: look at the durations Cuckoo records over a week or two and set the max runtime well above the slowest normal run, for example 2x the longest run you have seen.
• Grace time: with a start ping in place, the grace time only needs to cover start-up delays, such as cron running a few seconds late or a RandomizedDelaySec in systemd. Five to fifteen minutes is often enough.

Troubleshooting

• The check shows "running" forever. The final ping never arrived. Check that curl is reachable after the job, and that nothing like exec replaces the shell before the trap can fire.
• SIGKILL is not reported. kill -9 and the out-of-memory killer cannot be trapped. The max runtime catches these runs.
• Overlapping runs. If a job can take longer than its interval, protect it with flock -n /run/lock/job.lock, otherwise start and finish pings of different runs interleave.
• Wrong exit code. Make sure the command is the last statement in the wrapper; anything after it overwrites $?.

FAQ

Is the /start ping required?

No. It is optional. Without it, Cuckoo still alerts on missing and failed runs; you only lose duration tracking and the max runtime check for hung jobs.

What happens if a start ping is followed by another start ping?

That usually means the previous run never reported back, for example because it was killed. Treat it as a hint to check the logs and consider adding locking with flock if runs overlap.

Does the trap still fire when the script uses set -e?

Yes. When set -e aborts the script, bash runs the EXIT trap, and $? inside the trap holds the failing status, so the failure is reported with its exit code.