Monitor Litestream SQLite Backups

Litestream continuously replicates a SQLite database to S3-compatible storage, a file system or another replica target. That makes it different from a nightly backup script: there is no cron job that finishes and could report success. Replication just keeps running, until a credential expires, the bucket policy changes, the disk fills up or the process is not restarted after a deploy. The database keeps working the whole time, so nobody notices until they need a restore. This guide sets up a scheduled restore test that proves the replica is usable and reports to Cuckoo.

How Cuckoo sees a continuous process

Since Litestream itself has no "run", you monitor the outcome instead: a small verification job runs on a schedule, restores the latest replica to a temporary file, checks it, and pings:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start before the restore,
• https://cuckoo.metastic.ch/ping/YOUR-UUID/0 if the restored copy is intact and recent,
• https://cuckoo.metastic.ch/ping/YOUR-UUID/1 (or another non-zero code) if any check failed.

If the verification job itself stops running, no ping arrives and the check goes down after the grace time. One check therefore covers replication, the storage target and the verification job.

Step 1: confirm your Litestream version and commands

The CLI changed with v0.5, which moved to the LTX file format. Check what you have:

litestream version

On v0.5.x, litestream ltx lists the LTX files of a database and litestream restore recovers it. The older snapshots and generations commands from v0.3 are not part of the current CLI. The examples below use v0.5 flags:

• -o PATH writes the restored database to a different path,
• -config PATH points to the config file (default /etc/litestream.yml),
• -if-replica-exists exits 0 when no backup exists (do not use it here; a missing backup should fail),
• -integrity-check quick|full runs a SQLite check after restoring (added in v0.5.10).

Step 2: the verification script

Save as /usr/local/bin/litestream-verify and make it executable:

#!/usr/bin/env bash
set -uo pipefail

DB=/var/lib/app/app.db                  # path as configured in litestream.yml
HB=https://cuckoo.metastic.ch/ping/YOUR-UUID
MAX_AGE_SECONDS=3600                    # newest restored row must be younger than this

tmp=$(mktemp -d)
log="$tmp/verify.log"
trap 'rm -rf "$tmp"' EXIT

curl -fsS -m 10 --retry 5 -o /dev/null "$HB/start" || true

verify() {
  litestream restore -config /etc/litestream.yml -o "$tmp/restore.db" "$DB" || return 10

  result=$(sqlite3 "$tmp/restore.db" 'PRAGMA integrity_check;') || return 11
  [ "$result" = "ok" ] || { echo "integrity_check: $result"; return 12; }

  # Freshness: adapt the query to a table your app writes to regularly
  newest=$(sqlite3 "$tmp/restore.db" "SELECT CAST(strftime('%s', MAX(created_at)) AS INTEGER) FROM events;") || return 13
  age=$(( $(date +%s) - ${newest:-0} ))
  echo "newest row is ${age}s old"
  [ "$age" -le "$MAX_AGE_SECONDS" ] || return 14
}

verify >"$log" 2>&1
code=$?

tail -c 10000 "$log" | curl -fsS -m 10 --retry 5 --data-binary @- -o /dev/null "$HB/$code" || true
exit $code

What each part catches:

• litestream restore fails if the replica is missing, the credentials do not work or files are corrupt. On v0.5.10 and later you can add -integrity-check full to the restore command as well.
• PRAGMA integrity_check returns the single row ok for a healthy database. Anything else is reported.
• The freshness query catches the quiet failure where old data restores fine but replication stopped hours ago. Use a timestamp column your application updates often. If the app is idle at night, use a larger MAX_AGE_SECONDS or a small table the app touches on a timer.

Distinct return codes (10 to 14) make the failed step visible in the Cuckoo dashboard, and the log tail is stored with the ping.

To restore straight from a replica URL without a config file, pass the URL instead of the database path, for example litestream restore -o "$tmp/restore.db" s3://my-bucket/app.db, with the credentials in environment variables.

Step 3: schedule it

With a systemd timer:

# /etc/systemd/system/litestream-verify.service
[Unit]
Description=Verify Litestream replica

[Service]
Type=oneshot
ExecStart=/usr/local/bin/litestream-verify

# /etc/systemd/system/litestream-verify.timer
[Unit]
Description=Hourly Litestream replica check

[Timer]
OnCalendar=hourly
RandomizedDelaySec=300
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now litestream-verify.timer

Run the verification on a different machine from the database if you can. Then it also tests that the replica is reachable from somewhere else, which is the situation you are in during a real restore.

Schedule and grace advice

For an hourly timer, use a period of 1 hour and a grace time of 15 to 30 minutes, covering the random delay and the restore time. Restore time grows with the database size, so time a manual run first. Set max runtime slightly above the longest restore you expect so a hung download is reported.

Troubleshooting

• Exit code 10: the restore failed. Run the restore command manually and check credentials, bucket name and litestream ltx /var/lib/app/app.db output.
• Exit code 14 only: restore works, but data is stale. Check systemctl status litestream on the database host and look for upload errors in its logs.
• No pings: check systemctl list-timers litestream-verify.timer and that sqlite3 is installed.

FAQ

Why not monitor the Litestream process with systemd instead?

A running process does not prove that uploads succeed. Process monitoring is useful, but a restore test checks the result you actually depend on: a recent, intact copy you can restore.

How often should I run the restore test?

Hourly is a common balance between detection time and the cost of downloading the replica. Very large databases may need a daily full test combined with a more frequent, cheaper freshness check.

Does the restore test affect the live database?

No. It reads only from the replica and writes to a temporary directory that is deleted afterwards. The live database and the running Litestream process are not touched.