Dead Man's Switch Monitoring Explained

Some failures are loud: a website returns errors, a server stops answering. Others are quiet: a backup script stops running, a queue consumer silently stalls, a nightly sync exits early. Nothing returns an error because nothing is there to ask. Those quiet failures are what dead man's switch monitoring is for.

Where the name comes from

A dead man's switch is a control that must be actively held or reset by an operator. Train controls are the classic example: if the driver stops pressing the pedal, the train brakes. The safe state is triggered by the absence of a signal, not by an alarm.

Applied to software, the idea is the same. A job regularly reports "I'm alive and I finished". The monitor does not need to know anything about how the job works. It only needs to notice when the reports stop.

Push vs pull monitoring

There are two basic ways to monitor something:

Pull monitoring cannot see a cron job, because there is nothing to request: the job exists for a few minutes a day and then disappears. It also cannot see work behind a firewall or NAT without opening ports. Push monitoring handles both, because the job reaches out on its own.

The two complement each other. An uptime monitor tells you the web shop is reachable; a heartbeat check tells you the nightly order export to the warehouse actually happened.

How Cuckoo implements it

In Cuckoo, every check has a unique ping URL and an expected schedule:

1. You create a check with a period (for example every 1 h) or a cron expression and IANA time zone, plus a grace time.
2. Your job calls https://cuckoo.metastic.ch/ping/YOUR-UUID when it finishes successfully.
3. If no ping arrives before the schedule plus grace has passed, the check goes down and alerts go out by email, Slack, Discord or signed webhook.
4. When pings resume, you get a recovery message.

Beyond the basic switch, a job can be more explicit:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start at the beginning, to track duration and catch hung runs with a max runtime.
• https://cuckoo.metastic.ch/ping/YOUR-UUID/fail when it knows it failed, for an immediate alert.
• https://cuckoo.metastic.ch/ping/YOUR-UUID/2 to report an exit code (0 is success, anything else a failure).
• A POST body of up to 10 KB, such as the last lines of the log, stored with the ping.

The smallest possible integration is one line in a crontab:

0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID

Or, from application code, for example in Python:

import urllib.request

def ping(suffix: str = "") -> None:
    url = "https://cuckoo.metastic.ch/ping/YOUR-UUID" + suffix
    try:
        urllib.request.urlopen(url, timeout=10)
    except OSError:
        pass  # never let monitoring break the job

ping("/start")
run_nightly_export()
ping()

Typical use cases

• Backups: restic, borg, pg_dump, mysqldump, snapshot scripts. A backup that silently stopped is often only discovered when a restore is needed.
• Scheduled maintenance: log rotation, certificate renewal, cache warm-up, cleanup of temporary files.
• Data pipelines: imports, exports, ETL jobs, report generation, invoice runs.
• Long-running workers: a queue consumer can ping every few minutes from its main loop; if it deadlocks, the pings stop.
• Machines behind NAT: home servers, branch office devices or IoT gateways that cannot be reached from outside but can make outbound requests.
• CI and scheduled pipelines: nightly builds or scheduled GitHub Actions and GitLab pipelines.

Designing a good switch

• Ping at the end, after the real work. A ping at the start only proves the job started.
• Ping from the job, not from a separate cron entry. A separate "I'm alive" cron line proves cron works, not that the job does.
• Keep the schedule honest. Use the same cron expression the job uses, in the same time zone.
• Size the grace time to real run times. Add /start and a max runtime for jobs with variable duration.
• Do not let the ping break the job. Use timeouts and ignore ping errors where the job continues afterwards.

Troubleshooting

• Check stays down although the job runs. Run the ping command manually on the same host to rule out DNS, proxy or firewall issues for outbound HTTPS.
• Alerts arrive too late. Report failures actively with /fail or exit codes instead of waiting for the grace time.
• Too many false alarms. The schedule or time zone does not match the job, or the grace time is too short.

FAQ

Is heartbeat monitoring the same as a dead man's switch?

In practice, yes. Both describe a monitor that expects regular signals and alerts when they stop. "Cuckoo" usually refers to the periodic signal itself, "dead man's switch" to the alerting principle behind it.

Do I still need uptime monitoring?

For anything that serves requests, such as websites or APIs, yes. Pull checks prove that users can reach the service. Cuckoo checks cover the background work that no user request ever touches.

Can a dead man's switch detect a job that produces wrong results?

Only if the job checks its own result. Add a sanity check, such as a minimum file size or row count, and send /fail when it does not pass. The monitor then alerts on bad output as well as on missing runs.