Monitor Django and Celery Beat Periodic Tasks

A Celery periodic task can fail in several places. The beat process can die, so nothing is scheduled. The worker can be down or busy, so messages sit in the broker. Or the task runs and raises. Flower shows you the last case if you happen to look, but nothing reports the first two. Celery doesn't know that a message which was never sent is missing.

A heartbeat check covers all three: the task has to report back on time, from the worker, or you get an alert.

How Cuckoo sees a periodic task

Create one check per periodic task in Cuckoo. Each check has a ping URL:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start when the task starts
• https://cuckoo.metastic.ch/ping/YOUR-UUID when it succeeds
• https://cuckoo.metastic.ch/ping/YOUR-UUID/fail when it fails (a POST body of up to 10 KB is stored, so send the error text)

If no success or failure arrives within the schedule plus the grace time, Cuckoo marks the check down. That happens whether the cause was beat, the broker, the worker or the code.

Option 1: Ping from inside the task

This is the most explicit approach. A small helper must never raise, so a monitoring outage can't fail your task:

# myapp/heartbeat.py
import requests

def ping(url, suffix="", body=None):
    try:
        if body is None:
            requests.get(url + suffix, timeout=10)
        else:
            requests.post(url + suffix, data=body.encode()[:10_000], timeout=10)
    except requests.RequestException:
        pass  # never let monitoring break the job

# myapp/tasks.py
import traceback
from celery import shared_task
from django.conf import settings
from .heartbeat import ping

@shared_task
def cleanup_sessions():
    url = settings.CUCKOO_URLS["cleanup_sessions"]  # https://cuckoo.metastic.ch/ping/YOUR-UUID
    ping(url, "/start")
    try:
        do_cleanup()
    except Exception:
        ping(url, "/fail", traceback.format_exc()[-10_000:])
        raise
    ping(url)

The schedule stays as usual, in CELERY_BEAT_SCHEDULE or in django_celery_beat's database tables:

from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    "cleanup-sessions": {
        "task": "myapp.tasks.cleanup_sessions",
        "schedule": crontab(hour=3, minute=0),
    },
}
CELERY_TIMEZONE = "Europe/Zurich"

Option 2: Celery signals for many tasks

To avoid editing every task, connect to Celery's signals once and map task names to ping URLs:

# myapp/celery_monitoring.py
import traceback
from celery.signals import task_prerun, task_success, task_failure
from django.conf import settings
from .heartbeat import ping

URLS = settings.CUCKOO_URLS_BY_TASK  # {"myapp.tasks.cleanup_sessions": "https://cuckoo.metastic.ch/ping/YOUR-UUID"}

@task_prerun.connect
def _start(sender=None, **kwargs):
    if sender.name in URLS:
        ping(URLS[sender.name], "/start")

@task_success.connect
def _success(sender=None, **kwargs):
    if sender.name in URLS:
        ping(URLS[sender.name])

@task_failure.connect
def _failure(sender=None, exception=None, einfo=None, **kwargs):
    if sender.name in URLS:
        ping(URLS[sender.name], "/fail", str(einfo or exception))

Import this module where the worker loads it, for example in your celery.py after app.autodiscover_tasks(). The signals run in the worker process. task_failure fires only for the final failure: a task that calls self.retry() sends task_retry instead, so retries don't trigger fail alerts.

Django management commands under cron

Not everything runs through Celery. For manage.py commands started by cron, wrap the call in the crontab line:

0 4 * * * cd /srv/app && curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID/start; .venv/bin/python manage.py clearsessions; curl -fsS -m 10 --retry 5 -o /dev/null https://cuckoo.metastic.ch/ping/YOUR-UUID/$?

$? is the exit code of manage.py, so 0 is reported as success and anything else as failure. Raise CommandError in your command to exit non-zero.

When beat itself dies

Beat is a single process and easy to forget: a container restart without celery beat, or a stale celerybeat.pid file, and nothing is scheduled any more. Because the pings come from the worker after beat sent the message, a dead beat process shows up as late checks. If you only want to watch beat, schedule a trivial task every five minutes that only pings, and give its check a 5-minute period with 5 minutes of grace.

Schedule and grace advice

• Use the same cron expression and time zone in Cuckoo as in crontab(...) and CELERY_TIMEZONE.
• Add queue wait time to the grace time. A task that waits 20 minutes behind a backlog is late by your definition only if the grace time is shorter than that.
• Set max runtime when hard time limits are not configured, so a hung task gets flagged.

Troubleshooting

• Signals never fire: the module with the receivers isn't imported in the worker. Import it from celery.py.
• sender is a string: task_success and task_failure pass the task object as sender. Use sender.name.
• Duplicate runs: run exactly one beat instance. Two beats send every task twice, which shows up as two pings per period.

FAQ

Do I need both the in-task pings and the signals?

No. Pick one per task. In-task pings are explicit and easy to review. Signals keep monitoring out of the task code and work well when you have many periodic tasks.

Does this work with django-celery-beat's DatabaseScheduler?

Yes. The scheduler only decides when beat sends a message. The pings come from the worker, so they work the same with the default scheduler, django_celery_beat, or any other beat scheduler.

Can I use the Sentry SDK's Celery beat monitoring instead?

Yes. Cuckoo accepts Sentry cron check-ins, and CeleryIntegration(monitor_beat_tasks=True) can send them. See the guide on Sentry crons for Python for the setup and its trade-offs.