Cuckoo

Monitor Certbot Certificate Renewal

Updated

Let's Encrypt certificates are short-lived (90 days for the default profile today), and certbot renews them automatically. Until it doesn't: a firewall change blocks the HTTP-01 challenge, a DNS API token expires, the timer is disabled after an OS upgrade, or a web server reload hook fails. Renewal attempts start weeks before expiry, so a broken setup gives you a few weeks of warning, but only if something tells you. This guide reports every certbot renew run to Cuckoo so you hear about the first failed attempt, not the expired certificate.

How Cuckoo sees the renewal job

The check's ping URL is called after each renewal run:

  • https://cuckoo.metastic.ch/ping/YOUR-UUID/0 when certbot renew exited with 0,
  • https://cuckoo.metastic.ch/ping/YOUR-UUID/1 (any non-zero code) when it failed.

If no result arrives within the schedule plus grace time, for example because the timer no longer runs, the check goes down and you are alerted.

Why not ping from a certbot hook?

Certbot offers three hooks, and none of them fits a dead man's switch:

  • --deploy-hook runs once for each certificate that was successfully renewed. Most runs of certbot renew renew nothing, because no certificate is due yet, so the hook does not run and the check would look late for weeks.
  • --pre-hook and --post-hook run only if at least one certificate was due and a renewal was attempted. On most days they do not run either, and --post-hook also runs after a failed attempt, so it cannot tell success from failure.

The useful signal is the exit code of certbot renew itself. It exits with 0 when every certificate is either not yet due or was renewed, and with a non-zero code when any renewal failed. So ping after the command, on every run, with that exit code. Keep the deploy hook for what it is good at, such as reloading nginx.

Option A: systemd timer

Packages from Debian and Ubuntu ship certbot.timer and certbot.service; the snap ships snap.certbot.renew.timer and snap.certbot.renew.service. Check which you have:

systemctl list-timers | grep -i certbot

Create a small reporting script:

sudo tee /usr/local/bin/certbot-heartbeat >/dev/null <<'EOF'
#!/bin/sh
# Called by systemd via ExecStopPost; SERVICE_RESULT and EXIT_STATUS are set by systemd
URL="https://cuckoo.metastic.ch/ping/YOUR-UUID"
if [ "$SERVICE_RESULT" = "success" ]; then
  suffix=0
else
  case "$EXIT_STATUS" in
    ''|*[!0-9]*) suffix=fail ;;   # killed by a signal or no exit code
    *) suffix="$EXIT_STATUS" ;;
  esac
fi
curl -fsS -m 10 --retry 5 -o /dev/null "$URL/$suffix" || true
EOF
sudo chmod 755 /usr/local/bin/certbot-heartbeat

Then add it to the service with a drop-in, without editing the packaged unit:

sudo systemctl edit certbot.service    # or snap.certbot.renew.service
[Service]
ExecStopPost=/usr/local/bin/certbot-heartbeat

ExecStopPost= runs after the service stopped, whether it succeeded or failed, which is why it is used here instead of ExecStartPost=. Test it once by hand:

sudo systemctl start certbot.service
journalctl -u certbot.service -n 20

Option B: cron

If you run certbot from cron, put the ping directly after the command:

# /etc/cron.d/certbot-renew
17 3,15 * * * root certbot renew -q; curl -fsS -m 10 --retry 5 -o /dev/null "https://cuckoo.metastic.ch/ping/YOUR-UUID/$?" || true

$? is the exit code of certbot renew. Make sure this is the only cron entry or timer running certbot, so you do not renew twice. When certbot runs without a terminal, it may sleep for a random few minutes first to spread load on Let's Encrypt, which is normal.

acme.sh

acme.sh installs its own cron entry that runs acme.sh --cron. Wrap that line the same way:

23 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null; curl -fsS -m 10 --retry 5 -o /dev/null "https://cuckoo.metastic.ch/ping/YOUR-UUID/$?" || true

acme.sh also has built-in notification hooks (--set-notify) that report renewal errors. Those depend on the cron job running at all; the Cuckoo ping covers the case where it stopped. Force a failure once, for example with a wrong DNS API key on a test domain, to confirm the exit code is non-zero on your version.

Schedule and grace advice

The Debian/Ubuntu certbot.timer fires twice a day with a randomised delay of up to 12 hours, so two runs can be up to about a day apart. Use a period of 1 day with a grace time of 12 hours for timer-based setups. For the fixed cron example above, use 17 3,15 * * * with the server's time zone and a grace time of 1 hour. Since renewal starts well before expiry, a generous grace time still gives you plenty of time to react.

Troubleshooting

  • Fail pings with exit code 1: run sudo certbot renew --dry-run to see the error. Common causes are port 80 blocked for HTTP-01, a changed webroot path or an expired DNS provider token.
  • No pings at all: run systemctl status certbot.timer and systemctl cat certbot.service to check that the timer is active and the drop-in is loaded.
  • Pings arrive but certificates still expire: the renewal works, but the server was not reloaded. Add --deploy-hook "systemctl reload nginx" to the renewal config.

FAQ

Does a successful ping mean my certificate was renewed?

No. It means certbot renew ran and nothing failed. On most days no certificate is due, so nothing is renewed. Certbot only attempts a renewal inside the renewal window, traditionally the last 30 days; newer versions also follow the CA's ARI hints.

Should I also monitor the certificate expiry date?

It is a useful extra layer. An expiry check on the live site catches cases such as a server serving an old certificate because it was never reloaded. The renewal ping tells you why the renewal itself broke.

Can I include certbot's output in the ping?

Yes. Capture the output and POST it, for example certbot renew 2>&1 | tail -c 10000 > /tmp/cb.log, then send it with curl --data-binary @/tmp/cb.log. Use PIPESTATUS in bash to keep certbot's exit code.

Related guides

All guides · Compare Cuckoo with other tools

Send feedback

Ideas, bugs, questions: we read every message. Or write to hello@metastic.ch.