Monitor APScheduler Jobs in Python with Cuckoo

APScheduler runs jobs inside your Python process: a BackgroundScheduler in a web app, or a BlockingScheduler as a dedicated worker. When a job raises, APScheduler logs the traceback and moves on. When a run is skipped because the previous one is still busy, or because it was too late to start, you get a warning in the log. When the process dies, the schedule dies with it. None of this alerts anyone by default.

A heartbeat check makes each run report in. If a report is late or says "failed", Cuckoo sends an alert.

How Cuckoo sees an APScheduler job

Create one check per job, using the job's cron expression and time zone. Each check has a URL:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start when the job starts
• https://cuckoo.metastic.ch/ping/YOUR-UUID when it succeeds
• https://cuckoo.metastic.ch/ping/YOUR-UUID/fail when it raises (a POST body of up to 10 KB is stored)

Step 1: A ping helper

# heartbeat.py
import requests

def ping(url, suffix="", body=None):
    if not url:
        return
    try:
        if body is None:
            requests.get(url + suffix, timeout=10)
        else:
            requests.post(url + suffix, data=body.encode()[:10_000], timeout=10)
    except requests.RequestException:
        pass  # monitoring must never break the job

Step 2: Schedule the job (APScheduler 3.x)

import os
import traceback
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger
from heartbeat import ping

NIGHTLY = os.environ.get("CUCKOO_NIGHTLY")  # https://cuckoo.metastic.ch/ping/YOUR-UUID

def nightly_export():
    ping(NIGHTLY, "/start")
    try:
        export_orders()
    except Exception:
        ping(NIGHTLY, "/fail", traceback.format_exc()[-10_000:])
        raise
    ping(NIGHTLY)

scheduler = BlockingScheduler(timezone="Europe/Zurich")
scheduler.add_job(
    nightly_export,
    CronTrigger.from_crontab("0 3 * * *", timezone="Europe/Zurich"),
    id="nightly-export",
    misfire_grace_time=600,  # may start up to 10 min late, e.g. after a restart
    coalesce=True,           # several missed runs collapse into one
    max_instances=1,         # never run two at once
    replace_existing=True,
)
scheduler.start()

In a web app, use BackgroundScheduler the same way and call scheduler.start() once at startup. Re-raising inside the job keeps APScheduler's own error logging.

What these options mean for monitoring:

• max_instances=1: if the previous run is still busy, APScheduler skips the new one and logs a warning. No ping is sent, so the check goes late. That's the signal you want for a stuck job.
• misfire_grace_time: a run that can't start within this window counts as missed and doesn't run. Again no ping, again a late check.
• coalesce=True: after downtime, only one catch-up run happens. Cuckoo gets one success ping and sends a recovery message.

Alternative: event listeners

If you have many jobs, keep the job functions clean and report from a listener. APScheduler emits an event for each execution, each error and each missed run:

from apscheduler.events import EVENT_JOB_EXECUTED, EVENT_JOB_ERROR, EVENT_JOB_MISSED
from heartbeat import ping

PINGS = {"nightly-export": os.environ.get("CUCKOO_NIGHTLY")}

def on_job_event(event):
    url = PINGS.get(event.job_id)
    if event.code == EVENT_JOB_EXECUTED:
        ping(url)
    elif event.code == EVENT_JOB_ERROR:
        ping(url, "/fail", f"{event.exception!r}\n{event.traceback or ''}")
    elif event.code == EVENT_JOB_MISSED:
        ping(url, "/fail", f"missed run scheduled for {event.scheduled_run_time}")

scheduler.add_listener(on_job_event, EVENT_JOB_EXECUTED | EVENT_JOB_ERROR | EVENT_JOB_MISSED)

Listeners run in the scheduler's thread, so keep them fast. The 10-second timeout keeps a ping from blocking the scheduler for long. Listeners don't give you a /start ping, so if you want a max runtime, ping /start at the top of the job.

Reporting EVENT_JOB_MISSED as /fail gets you an immediate alert, rather than waiting for the grace time to run out. Skips caused by max_instances emit EVENT_JOB_MAX_INSTANCES instead. Add it to the mask if you want those reported too. Otherwise they show up as a late check.

APScheduler 4

APScheduler 4 is a rewrite with a different API: Scheduler/AsyncScheduler, add_schedule() instead of add_job(), and new event classes. The ping helper and the pattern (ping from the task, or from an event subscriber) carry over. The code above targets the 3.x series.

Schedule and grace advice

• Use the same cron expression and time zone in Cuckoo as in CronTrigger. Without an explicit time zone, APScheduler 3 uses the local time zone of the host.
• Make the Cuckoo grace time larger than misfire_grace_time plus the normal runtime, so a late but valid start isn't reported as down.
• Run the scheduler in one process only. With Gunicorn and several workers, every worker starts its own BackgroundScheduler, and every job runs and pings several times.

Troubleshooting

• Jobs run several times: see above. Move the scheduler to a dedicated process or start it in only one worker.
• "Run time was missed" warnings: the process was busy or asleep. Raise misfire_grace_time or check for blocking code in other jobs.
• Listener pings missing: the job id is not in your PINGS map. Set explicit ids in add_job.

FAQ

Should I ping from the job or from a listener?

From the job if you have a few jobs and want /start and max runtime. From a listener if you have many jobs and want monitoring in one place. You can combine them, with /start in the job and the result from the listener.

What happens if the process running APScheduler crashes?

No more pings are sent. Cuckoo notices that the next expected ping is missing and alerts you once the grace time has passed. That is the main reason to monitor in-process schedulers from the outside.

Does a slow Cuckoo response delay my job?

At most by the 10-second timeout per ping. Network errors are caught, so the job's own result is never affected.