Restic Backup Monitoring

restic is a fast, encrypted backup tool that works with local disks, SFTP, S3-compatible storage and many other backends. Like any backup, it is only useful if it keeps running. An expired storage key, a full repository or a stale lock can stop backups for weeks, and you usually find out on the day you need a restore.

This guide wraps restic in a small script that reports start, result and log tail to Cuckoo, so a missed or failed backup produces an alert.

How Cuckoo sees the backup

The script pings https://cuckoo.metastic.ch/ping/YOUR-UUID/start before the backup and https://cuckoo.metastic.ch/ping/YOUR-UUID/<exit code> afterwards, with the last 10 KB of output in the POST body. Exit code 0 marks the run as successful; anything else marks it as failed and alerts immediately. If the script does not run at all, the check goes down once its schedule plus grace time has passed.

Restic exit codes

Recent restic versions (0.17 and later) document these exit codes for restic backup:

Code 3 deserves attention. The snapshot exists, but it is missing files, typically because of permission errors or files that vanished during the run. Treating it as a failure is the safe default; the log tail shows which paths were affected.

Step 1: Keep secrets in an env file

Create /etc/restic/backup.env, owned by root with mode 600:

RESTIC_REPOSITORY=s3:https://s3.example.com/my-backups
RESTIC_PASSWORD_FILE=/etc/restic/password
AWS_ACCESS_KEY_ID=REPLACE_ME
AWS_SECRET_ACCESS_KEY=REPLACE_ME
HB_URL=https://cuckoo.metastic.ch/ping/YOUR-UUID

sudo chmod 600 /etc/restic/backup.env /etc/restic/password

Step 2: The backup script

Save as /usr/local/bin/restic-backup and make it executable:

#!/usr/bin/env bash
set -uo pipefail

set -a; . /etc/restic/backup.env; set +a

LOG=$(mktemp)
trap 'rm -f "$LOG"' EXIT

hb() { curl -fsS -m 10 --retry 5 -o /dev/null "$@" || true; }

hb "$HB_URL/start"

run() {
  echo "### restic $*"
  restic "$@"
}

{
  run backup --verbose=1 --exclude-caches --one-file-system \
      --exclude-file /etc/restic/excludes.txt \
      /etc /home /srv \
  && run forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune \
  && run check
} > "$LOG" 2>&1
rc=$?

tail -c 10000 "$LOG" | hb --data-binary @- "$HB_URL/$rc"
cat "$LOG"          # keep the full output for cron mail or the journal
exit "$rc"

What it does:

• set -a exports every variable from the env file so restic picks up the repository and credentials.
• backup, forget --prune and check run in order; && stops at the first failure and rc holds its exit code.
• --exclude-caches skips directories marked with a CACHEDIR.TAG; --one-file-system avoids crossing into other mounts.
• forget --prune applies the retention policy and removes unreferenced data in one step.
• check verifies the repository structure. Add --read-data-subset=5% once a week to also read back part of the actual data.
• The log tail goes to Cuckoo with the exit code, and the full log stays local.

If you prefer to accept incomplete snapshots, map code 3 explicitly after rc=$? with [ "$rc" -eq 3 ] && rc=0. The forget and check steps do not run in that case, because && stopped at the backup.

Step 3: Schedule it

With cron (sudo crontab -e):

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
30 1 * * * /usr/local/bin/restic-backup > /var/log/restic-backup.log 2>&1

Or with a systemd timer: a Type=oneshot service with ExecStart=/usr/local/bin/restic-backup and a timer with OnCalendar=*-*-* 01:30:00 and Persistent=true.

On the Cuckoo check, use the cron expression 30 1 * * *, the server's IANA time zone and a grace time of around 30 minutes. Set a max runtime of about twice the longest backup you have seen, so a run stuck on a slow backend is caught.

Troubleshooting

• Exit code 11 (lock failed): another restic process is running, or a crashed run left a stale lock. Check with restic list locks; remove stale locks with restic unlock only when no other process uses the repository.
• Exit code 12 (wrong password): the password file is missing or unreadable for the user running the job.
• Exit code 3 every night: some paths cannot be read. Run the backup as root or exclude them.
• Exit code 10: the repository was never initialised (restic init) or the URL is wrong.
• Backups get slower over time: forget --prune every night can be expensive on large repositories. Run forget nightly and prune weekly as separate scheduled jobs, each with its own check.

FAQ

Should forget, prune and check run in the same job as backup?

For small and medium repositories, one job is simplest and gives a single signal. For large repositories, split them so a long prune does not delay the next backup; give each job its own Cuckoo check.

Does restic check read all my data?

No. By default it verifies the repository structure and metadata. --read-data reads everything, and --read-data-subset reads a fraction, which is a good weekly compromise.

Can I see which files failed in an incomplete snapshot?

Yes. restic prints an error line per unreadable file, and those lines end up in the log tail stored with the ping, so they are visible in the dashboard and in the alert context.