Borg Backup Monitoring

BorgBackup (borg) deduplicates, compresses and encrypts backups efficiently, which makes it a common choice for servers and workstations. But a borg job that stops running gives no sign at all. The SSH key to the backup host gets rotated, the repository fills up, or a lock stays behind after a crash, and the newest archive slowly gets older.

This guide shows two ways to report borg runs to Cuckoo: a plain shell script, and borgmatic's built-in monitoring hook.

How Cuckoo sees the backup

The job pings https://cuckoo.metastic.ch/ping/YOUR-UUID/start when it begins and then either the plain URL (success), /fail, or an exit code such as https://cuckoo.metastic.ch/ping/YOUR-UUID/2. If no ping arrives within the schedule plus grace time, the check goes down. A POST body (first 10 KB) is stored with the ping, so the log tail is attached to the alert.

Borg exit codes

Borg 1.x returns these codes by default:

Since borg 1.4, setting BORG_EXIT_CODES=modern makes borg return more specific codes: 3–99 for errors and 100–127 for warnings. The default is still legacy (0/1/2). Cuckoo treats any non-zero code as a failure, so decide whether warnings should alert you; the script below lets you choose.

Option 1: Plain shell script

/etc/borg/backup.env (mode 600):

BORG_REPO=ssh://borg@backup.example.com/./server1
BORG_PASSCOMMAND="cat /etc/borg/passphrase"
HB_URL=https://cuckoo.metastic.ch/ping/YOUR-UUID
ALERT_ON_WARNING=yes

/usr/local/bin/borg-backup:

#!/usr/bin/env bash
set -uo pipefail
set -a; . /etc/borg/backup.env; set +a

LOG=$(mktemp)
trap 'rm -f "$LOG"' EXIT
hb() { curl -fsS -m 10 --retry 5 -o /dev/null "$@" || true; }

hb "$HB_URL/start"
max_rc=0
step() {
  echo "### borg $*" >> "$LOG"
  borg "$@" >> "$LOG" 2>&1
  local rc=$?
  [ "$rc" -gt "$max_rc" ] && max_rc=$rc
  return "$rc"
}

step create --stats --compression zstd,6 --exclude-caches \
     --exclude-from /etc/borg/excludes.txt \
     "::{hostname}-{now}" /etc /home /srv

# Only prune and compact if create did not fail hard (legacy exit codes, the default)
if [ "$max_rc" -lt 2 ]; then
  step prune --list --glob-archives '{hostname}-*' \
       --keep-daily 7 --keep-weekly 4 --keep-monthly 6
  step compact
fi

rc=$max_rc
[ "$rc" -eq 1 ] && [ "${ALERT_ON_WARNING:-yes}" != "yes" ] && rc=0

tail -c 10000 "$LOG" | hb --data-binary @- "$HB_URL/$rc"
cat "$LOG"
exit "$max_rc"

Notes:

• borg prune only marks archives for deletion; since borg 1.2, borg compact actually frees the space.
• --glob-archives '{hostname}-*' restricts pruning to this host's archives, which matters if several machines share a repository.
• max_rc keeps the worst result of all steps, so a warning in create is not hidden by a clean compact.
• The ping reports the result; the script still exits with borg's real code for cron or systemd.

Schedule it in root's crontab:

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
15 2 * * * /usr/local/bin/borg-backup > /var/log/borg-backup.log 2>&1

Run borg check on its own schedule, for example weekly, with its own Cuckoo check, since it can take long on big repositories.

Option 2: borgmatic's healthchecks hook

borgmatic, the popular wrapper around borg, has a monitoring hook named healthchecks. It sends a ping with /start appended when a run begins, the plain URL when it finishes, and /fail on error, and can POST the log in the body. Those are the same URL conventions Cuckoo uses, so you can point the hook at your Cuckoo ping URL.

In /etc/borgmatic/config.yaml (borgmatic 1.8 and later; older versions put it under a hooks: section):

source_directories:
    - /etc
    - /home
    - /srv

repositories:
    - path: ssh://borg@backup.example.com/./server1
      label: offsite

keep_daily: 7
keep_weekly: 4
keep_monthly: 6

healthchecks:
    ping_url: https://cuckoo.metastic.ch/ping/YOUR-UUID
    send_logs: true
    ping_body_limit: 10000
    states:
        - start
        - finish
        - fail

• Use the full URL. borgmatic treats a value that does not start with http as a UUID for its default service.
• send_logs: true attaches logs to finish and fail pings; it defaults to off since borgmatic 2.1.
• ping_body_limit: 10000 matches what Cuckoo stores.
• Listing the states explicitly leaves out the separate log state, which uses a /log suffix that Cuckoo does not define.

The hook runs for the create, prune, compact and check actions. Schedule borgmatic with cron or the systemd timer that ships with many distribution packages.

Schedule and grace advice

Use the job's cron expression (15 2 * * *) and the server's IANA time zone on the check. Backups vary in duration, so keep the grace time moderate (for example 30 minutes) and set a max runtime of roughly twice the longest run you have seen. A stalled SSH connection is then caught without waiting a full day.

Troubleshooting

• "Failed to create/acquire the lock": a previous run is still active or crashed. Check for running borg processes before using borg break-lock.
• Exit code 1 every night: usually files changing during the backup, such as logs or databases. Exclude them or back up a database dump instead.
• Passphrase prompts under cron: BORG_PASSCOMMAND or BORG_PASSPHRASE is not set in the job's environment.
• Host key prompts: connect once interactively as the user that runs the job so the SSH host key is known.

FAQ

Should borg warnings trigger an alert?

By default the script reports them, because a warning can mean files were skipped. If warnings are routine and you review logs anyway, set ALERT_ON_WARNING=no so only errors alert.

Can I use borgmatic and the plain script together?

There is no need. Pick one: borgmatic if you want configuration in YAML and built-in hooks, the script if you prefer to see every borg command directly.

Does Cuckoo show how long the backup took?

Yes, when the run starts with a /start ping. Both the script and the borgmatic hook send it, so the duration between start and finish is recorded.