Monitor systemd Timers with Cuckoo

systemd timers are a solid replacement for cron: logs go to the journal, runs can catch up after downtime, and units have proper dependencies. What they do not do is tell you when a run failed or never happened. systemctl list-timers shows the next run, but nobody watches it at 3 a.m.

This guide adds Cuckoo signals to a timer-driven service so missed, failed and hung runs all raise an alert.

How Cuckoo sees the job

The service sends up to three pings to its check:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start when the run begins,
• https://cuckoo.metastic.ch/ping/YOUR-UUID when it succeeds,
• https://cuckoo.metastic.ch/ping/YOUR-UUID/fail (or an exit code like /1) when it fails.

On the check, use the timer's schedule as a cron expression plus the server's time zone, a grace time, and optionally a max runtime so a hanging run is detected.

Step 1: The service unit

/etc/systemd/system/backup.service:

[Unit]
Description=Nightly backup
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
Environment=HB_URL=https://cuckoo.metastic.ch/ping/YOUR-UUID
ExecStartPre=-/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null ${HB_URL}/start
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=-/usr/bin/curl -fsS -m 10 --retry 5 -o /dev/null ${HB_URL}

• Type=oneshot makes systemd wait for backup.sh to exit. ExecStartPost= then only runs if ExecStart= succeeded, so the success ping means the work really finished.
• The - prefix tells systemd to ignore a failing curl, so a network blip never marks the backup as failed.
• ${HB_URL} is expanded by systemd from Environment=.

Step 2: The timer unit

/etc/systemd/system/backup.timer:

[Unit]
Description=Run nightly backup

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target

• Persistent=true stores the last run time on disk. If the machine was off at 02:30, the service starts right after boot. Cuckoo then sees a late ping and sends a recovery once it arrives.
• RandomizedDelaySec=10m spreads start times by up to ten minutes. Add that delay to the grace time on the check.

Enable and test:

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers backup.timer
systemd-analyze calendar "*-*-* 02:30:00"
sudo systemctl start backup.service   # one manual run to see the pings arrive

Step 3a: Report failures with OnFailure=

OnFailure= starts another unit when the service enters the failed state. A small template unit can report the failure for any service. Put the ping URL for each service into its own file, /etc/heartbeat/backup.env:

HB_URL=https://cuckoo.metastic.ch/ping/YOUR-UUID

Then create /etc/systemd/system/heartbeat-fail@.service:

[Unit]
Description=Report failure of %i to Cuckoo

[Service]
Type=oneshot
EnvironmentFile=/etc/heartbeat/%i.env
ExecStart=/bin/sh -c 'journalctl -u %i.service -n 100 --no-pager | tail -c 10000 | curl -fsS -m 10 --retry 5 -o /dev/null --data-binary @- "$$HB_URL/fail"'

And reference it from backup.service:

[Unit]
OnFailure=heartbeat-fail@%N.service

%N is the unit name without the .service suffix, so the template looks for /etc/heartbeat/backup.env. $$ passes a literal $ to the shell, which reads HB_URL from the environment. The last 100 journal lines are attached to the fail ping (trimmed to the 10 KB Cuckoo stores).

Step 3b: Alternative with ExecStopPost=

ExecStopPost= runs after the service stops, whether it succeeded or not, and systemd passes $SERVICE_RESULT, $EXIT_CODE and $EXIT_STATUS to it. One helper script covers success and failure.

/usr/local/bin/heartbeat-stop-post:

#!/bin/sh
# Called from ExecStopPost=. systemd sets SERVICE_RESULT, EXIT_CODE and EXIT_STATUS.
url="${HB_URL:?HB_URL is not set}"

if [ "$SERVICE_RESULT" = "success" ]; then
  suffix=""
elif [ "$EXIT_CODE" = "exited" ] && [ "${EXIT_STATUS:-0}" != "0" ]; then
  suffix="/$EXIT_STATUS"          # report the real exit code
else
  suffix="/fail"                  # killed, timeout, signal, start-up failure
fi

curl -fsS -m 10 --retry 5 -o /dev/null "$url$suffix" || true

In the service, replace ExecStartPost= with:

ExecStopPost=/usr/local/bin/heartbeat-stop-post

Keep ExecStartPre= for the start ping. Use either 3a or 3b, not both, or failures will be reported twice.

Schedule and grace advice

OnCalendar=*-*-* 02:30:00 corresponds to the cron expression 30 2 * * *. Set it on the check with the zone shown by timedatectl. Grace time = normal run time + RandomizedDelaySec + margin, for example 30 minutes. For Type=oneshot services systemd applies no start timeout by default, so a hung run can block forever. Set a max runtime on the check, and consider TimeoutStartSec=3h in the service to stop runaway jobs.

Troubleshooting

• Inspect a run: journalctl -u backup.service -n 200 or journalctl -u backup.service --since today.
• Start ping arrives, success does not: ExecStart= failed or hung. Check systemctl status backup.service.
• Timer never fires: the timer is not enabled. systemctl list-timers --all shows inactive timers too.
• Ping fails at boot: the network is not up yet. Wants=/After=network-online.target waits for it.
• OnFailure= unit not started: check systemctl status heartbeat-fail@backup.service and that the env file exists.

FAQ

Do I need a timer per job, or can one service handle several?

Use one service and one timer per job, each with its own Cuckoo check. That keeps alerts, logs and schedules separate, and the OnFailure= template can be shared by all of them.

Does Persistent=true cause duplicate runs?

No. It triggers at most one catch-up run after downtime, no matter how many runs were missed. Cuckoo sees the late ping and closes the alert.

Can I use a user timer instead of a system timer?

Yes. Put the units in ~/.config/systemd/user/ and use systemctl --user. Enable lingering with loginctl enable-linger USER so the timer runs while you are logged out.