Cron Job Exit Code Monitoring

Every Unix process ends with an exit code: 0 for success, anything else for some kind of failure. Cron ignores it. A job can exit with status 1 every night for a month and cron will happily start it again the next day without telling anyone.

Cuckoo can read the exit code directly from the ping URL. Combined with the tail of the job's log in the request body, you get an immediate alert that already contains the error message.

How Cuckoo interprets exit codes

Append the exit code to the ping URL:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/0 marks the run as successful.
• https://cuckoo.metastic.ch/ping/YOUR-UUID/1 (or any value up to 255) marks it as failed and alerts right away.

If no ping arrives at all within the schedule plus grace time, the check goes down as usual. Exit codes add fast failure detection on top of missed-run detection.

The one-line version

In a crontab, $? holds the exit status of the previous command. Separate the job and the ping with ; (not &&) so the ping is always sent:

0 4 * * * /usr/local/bin/rotate-reports.sh; curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID/$?

$? is expanded by the shell before curl runs, so it still refers to rotate-reports.sh.

Attach the log tail

An alert that says "exit code 2" is useful. An alert that also shows the last lines of output is better. Cuckoo stores the first 10 KB of a POST body with the ping, so send the end of the log, not the beginning:

#!/usr/bin/env bash
# /usr/local/bin/rotate-reports-monitored
set -uo pipefail

LOG=$(mktemp)
trap 'rm -f "$LOG"' EXIT

/usr/local/bin/rotate-reports.sh > "$LOG" 2>&1
rc=$?

tail -c 10000 "$LOG" | curl -fsS -m 10 --retry 5 -o /dev/null \
  --data-binary @- \
  -H "Content-Type: text/plain; charset=utf-8" \
  "https://cuckoo.metastic.ch/ping/YOUR-UUID/$rc" || true

exit "$rc"

Details that matter:

• > "$LOG" 2>&1 captures both stdout and stderr; most error messages go to stderr.
• tail -c 10000 keeps the last 10,000 bytes, which fits the 10 KB limit and contains the final error.
• --data-binary @- reads the body from stdin and sends it unchanged. Plain -d would strip newlines, which makes multi-line logs hard to read.
• Passing a body makes curl use POST automatically.
• exit "$rc" preserves the original exit status for cron, systemd or whatever calls the script.

Use it in the crontab:

0 4 * * * /usr/local/bin/rotate-reports-monitored

Why you need set -o pipefail

By default, the exit status of a pipeline is the status of its last command. That hides failures:

pg_dump mydb | gzip > /backup/mydb.sql.gz
echo $?   # prints 0 even if pg_dump failed, because gzip succeeded

With set -o pipefail, the pipeline returns the status of the rightmost command that failed:

set -o pipefail
pg_dump mydb | gzip > /backup/mydb.sql.gz
echo $?   # non-zero if pg_dump or gzip failed

pipefail is supported by bash, zsh and ksh and was only recently added to the POSIX standard. Cron runs commands with /bin/sh, which is dash on Debian and Ubuntu, and the installed version may not support it. Put the logic in a script with #!/usr/bin/env bash or set SHELL=/bin/bash in the crontab.

Exit codes worth knowing

Codes above 128 mean "killed by signal N", where N = code - 128. Some tools define their own codes; restic uses 3 for an incomplete snapshot and borg uses 1 for warnings. Check the tool's documentation before deciding which codes are acceptable.

Treating some non-zero codes as success

If a tool uses an exit code for a harmless warning, map it before pinging:

/usr/local/bin/sync-mirror.sh > "$LOG" 2>&1
rc=$?
# rsync exit code 24 = some source files vanished during transfer; acceptable here
[ "$rc" -eq 24 ] && rc=0

Be deliberate about this. Every code you map to 0 is a failure you will not hear about.

Schedule and grace advice

With exit codes, failures are reported as soon as the job ends, so the grace time only needs to cover how long the job may take plus scheduling jitter. Use the job's cron expression and the server's IANA time zone on the check. Add a /start ping and max runtime if the job can hang.

Troubleshooting

• Every run reports 127. A command is not on cron's PATH. Use absolute paths.
• The stored body is empty. Check that output is redirected into the log file and that tail reads the right path.
• The log contains garbage characters. Cuckoo stores UTF-8 text; strip colour codes with sed 's/\x1b\[[0-9;]*m//g' before sending.
• Exit code is always 0. Something ran between the job and rc=$?, or a pipeline hid the error without pipefail.

FAQ

What is the difference between /fail and an exit code ping?

/ping/YOUR-UUID/fail always means failure. An exit code ping lets the job decide: 0 is success and any other value is failure, and the actual code is kept, which helps when you look at the history later.

Can I send more than 10 KB of logs?

Cuckoo stores the first 10 KB of the body, and anything beyond is not kept. That is why the examples send the tail. Keep full logs on the server or in your log system.

Does the exit code ping work with GET requests?

Yes. The exit code is part of the URL path, so GET, HEAD and POST all work. Use POST only when you want to attach output.