Monitor Go Cron Jobs and robfig/cron Schedules

Go jobs usually run in one of two ways: a standalone binary started by cron or a Kubernetes CronJob, or a long-running service that schedules work with robfig/cron/v3. Both can stop without anyone noticing. The crontab gets lost, the binary panics on startup after a config change, or the service is redeployed with the scheduler disabled. Logs only help if someone reads them.

A heartbeat check makes the job report in. If the report is late or says "failed", Cuckoo alerts you.

How Cuckoo sees a Go job

Create one check per job, with the job's schedule and time zone. Each check has a URL:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start when the job starts
• https://cuckoo.metastic.ch/ping/YOUR-UUID when it succeeds
• https://cuckoo.metastic.ch/ping/YOUR-UUID/fail when it fails (a POST body of up to 10 KB is stored)

Step 1: A ping helper

The standard library is enough. Use a client with a timeout, pass a context, and never return an error that the job would have to handle:

package heartbeat

import (
	"context"
	"io"
	"log"
	"net/http"
	"strings"
	"time"
)

var client = &http.Client{Timeout: 10 * time.Second}

// Ping sends url+suffix ("" = success, "/start", "/fail"). A non-empty body is POSTed (first 10 KB kept).
func Ping(ctx context.Context, url, suffix, body string) {
	if url == "" {
		return
	}
	ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
	defer cancel()

	method, r := http.MethodGet, io.Reader(nil)
	if body != "" {
		if len(body) > 10_000 {
			body = body[:10_000]
		}
		method, r = http.MethodPost, strings.NewReader(body)
	}
	req, err := http.NewRequestWithContext(ctx, method, url+suffix, r)
	if err != nil {
		log.Printf("heartbeat: %v", err)
		return
	}
	resp, err := client.Do(req)
	if err != nil {
		log.Printf("heartbeat: %v", err)
		return
	}
	resp.Body.Close()
}

// Run wraps a job: /start, then success or /fail with the error text.
func Run(ctx context.Context, url string, job func(context.Context) error) error {
	Ping(ctx, url, "/start", "")
	err := job(ctx)
	// Fresh context: the job context may already be cancelled or past its deadline.
	if err != nil {
		Ping(context.Background(), url, "/fail", err.Error())
		return err
	}
	Ping(context.Background(), url, "", "")
	return nil
}

Run returns the job's error unchanged, so the caller decides about logging and exit codes. Ping errors are only logged.

Option A: A standalone binary run by cron

For a binary started by cron, systemd timers or a Kubernetes CronJob, wrap main and make the exit code reflect the result:

func main() {
	ctx := context.Background()
	url := os.Getenv("CUCKOO_URL") // https://cuckoo.metastic.ch/ping/YOUR-UUID
	if err := heartbeat.Run(ctx, url, exportOrders); err != nil {
		log.Printf("export failed: %v", err)
		os.Exit(1)
	}
}

A panic is not caught by Run. Add defer with recover() in the job if you want panics reported as /fail. Otherwise the missing success ping triggers the alert after the grace time. If you prefer to keep the Go code free of monitoring, wrap the binary in the crontab instead:

0 3 * * * /usr/local/bin/export-orders; curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID/$?

Option B: robfig/cron/v3 inside a service

robfig/cron/v3 accepts a CRON_TZ= prefix per schedule, or a default location for all of them via cron.WithLocation. SkipIfStillRunning prevents overlapping runs, and Recover turns panics into log lines instead of crashing the process:

c := cron.New(cron.WithChain(
	cron.Recover(cron.DefaultLogger),
	cron.SkipIfStillRunning(cron.DefaultLogger),
))

nightly := os.Getenv("CUCKOO_NIGHTLY") // https://cuckoo.metastic.ch/ping/YOUR-UUID
c.AddFunc("CRON_TZ=Europe/Zurich 0 3 * * *", func() {
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
	defer cancel()
	_ = heartbeat.Run(ctx, nightly, rebuildSearchIndex)
})

c.Start()
defer c.Stop()

A skipped run sends no ping, so a job stuck for hours becomes a late check. If the whole service dies, every check fed by it goes late. You get that signal for free with an external heartbeat.

Schedule and grace advice

• Mirror the schedule exactly: CRON_TZ=Europe/Zurich 0 3 * * * becomes cron 0 3 * * * with time zone Europe/Zurich. Without a time zone, robfig/cron uses time.Local, which in containers is usually UTC.
• robfig/cron/v3 uses five fields by default. If you enabled seconds with cron.WithSeconds(), drop the seconds field in Cuckoo.
• Set the grace time to the normal runtime plus a margin. Set max runtime to the context timeout, so a run that started but never reported is flagged.

Troubleshooting

• x509: certificate signed by unknown authority in scratch images: copy CA certificates into the image (for example from alpine or gcr.io/distroless/static).
• Pings only from one replica: expected if you use leader election. Without it, every replica runs the schedule and pings, so use a single scheduler instance.
• Context deadline exceeded on /start: the context passed to Run was already expired. Create the job's timeout context right before calling Run.

FAQ

Should the ping use the job's context?

For /start, yes. For the final ping, a fresh context with its own 10-second timeout is safer, because the job context may already be cancelled when the job ends because of a timeout.

Does a failing ping slow down or fail the job?

The helper waits at most 10 seconds per ping and only logs errors. The job's result and exit code never depend on whether the ping got through.

Can I monitor goroutines that run in a loop rather than on a schedule?

Yes. Ping the success URL at the end of every loop iteration and give the check a simple period slightly longer than the loop interval. If the loop stalls or the goroutine exits, the check goes late.