Monitor Google Cloud Run Jobs and Cloud Scheduler

Cloud Run jobs run containers to completion, and Cloud Scheduler can start them on a cron schedule. The setup has several places where a nightly run can disappear quietly: the scheduler job can be paused, its service account can lose the permission to run the job, a new image can crash on start, or one task out of ten can fail after all retries. This guide shows how to report each execution to Cuckoo exactly once, so you get one clear alert when something goes wrong.

How Cuckoo sees the job

Each check has a ping URL:

• https://cuckoo.metastic.ch/ping/YOUR-UUID/start marks the start of a run,
• https://cuckoo.metastic.ch/ping/YOUR-UUID (or /0) reports success,
• https://cuckoo.metastic.ch/ping/YOUR-UUID/fail (or a non-zero exit code) reports failure.

If no success or failure ping arrives within the schedule plus grace time, the check goes down. That catches the case where Cloud Scheduler never started the job.

Step 1: trigger the job from Cloud Scheduler

Cloud Scheduler calls the Cloud Run Admin API with a service account that has the Cloud Run Invoker role (roles/run.invoker) on the job:

gcloud scheduler jobs create http nightly-report-trigger \
  --location=europe-west6 \
  --schedule="30 2 * * *" \
  --time-zone="Europe/Zurich" \
  --http-method=POST \
  --uri="https://europe-west6-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT_ID/jobs/nightly-report:run" \
  --oauth-service-account-email=scheduler@PROJECT_ID.iam.gserviceaccount.com

Cloud Scheduler uses standard five-field cron with an explicit time zone, so you can copy both values into the Cuckoo check.

Step 2: understand tasks and retries

A Cloud Run job execution consists of one or more tasks. Each task gets environment variables:

• CLOUD_RUN_TASK_INDEX: from 0 to task count minus one,
• CLOUD_RUN_TASK_COUNT: the number of tasks in this execution,
• CLOUD_RUN_TASK_ATTEMPT: 0 on the first attempt, then incremented on each retry.

--max-retries sets how often a failed task is retried. The execution fails if any task still fails after its last retry. This matters for pings: if every task attempt sent its own result, a task that fails once and then succeeds would produce a false fail alert, and ten tasks would send ten success pings.

Step 3a: single-task jobs

For a job with one task, wrap the command and send a fail ping only on the final attempt. Pass the retry limit in yourself, because Cloud Run does not expose it as a variable:

gcloud run jobs deploy nightly-report \
  --image=europe-west6-docker.pkg.dev/PROJECT_ID/jobs/report:1.4.0 \
  --region=europe-west6 \
  --tasks=1 \
  --max-retries=2 \
  --task-timeout=30m \
  --set-env-vars=MAX_RETRIES=2,CUCKOO_URL=https://cuckoo.metastic.ch/ping/YOUR-UUID

#!/bin/sh
# entrypoint.sh
HB="$CUCKOO_URL"
attempt="${CLOUD_RUN_TASK_ATTEMPT:-0}"

[ "$attempt" = "0" ] && curl -fsS -m 10 --retry 5 -o /dev/null "$HB/start" || true

/app/run-report
code=$?

if [ "$code" -eq 0 ]; then
  curl -fsS -m 10 --retry 5 -o /dev/null "$HB" || true
elif [ "$attempt" -ge "${MAX_RETRIES:-0}" ]; then
  curl -fsS -m 10 --retry 5 -o /dev/null "$HB/$code" || true
fi
exit $code

Intermediate failures are silent, because Cloud Run will retry them. Only the final failed attempt reports. For a sensitive URL, store it in Secret Manager and use --set-secrets=CUCKOO_URL=heartbeat-url:latest instead.

Step 3b: multi-task jobs

With several tasks, no single task knows whether the others succeeded. Two workable options:

1. Report from task 0 only. Use the same script, but skip all pings unless CLOUD_RUN_TASK_INDEX is 0. This tells you the execution ran, but not whether the other tasks failed.
2. Report from a final step. Start the job from Cloud Workflows instead of directly from Cloud Scheduler. The Cloud Run connector waits for the execution to finish and raises an error if it failed:

main:
  steps:
    - init:
        assign:
          - hb: ${sys.get_env("CUCKOO_URL")}
          - project: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
    - start:
        call: http.get
        args:
          url: ${hb + "/start"}
          timeout: 10
    - run_job:
        try:
          call: googleapis.run.v2.projects.locations.jobs.run
          args:
            name: ${"projects/" + project + "/locations/europe-west6/jobs/nightly-report"}
            body: {}
            connector_params:
              timeout: 7200
        except:
          as: e
          steps:
            - report_fail:
                call: http.get
                args:
                  url: ${hb + "/fail"}
                  timeout: 10
            - reraise:
                raise: ${e}
    - report_success:
        call: http.get
        args:
          url: ${hb}
          timeout: 10

Deploy it with --set-env-vars=CUCKOO_URL=... and point the Cloud Scheduler job at the workflow's executions API. The fail ping then fires only once, after Cloud Run has used up all retries. In this snippet an unreachable ping URL would fail the workflow step, so wrap the http.get calls in try if you want pings to be strictly best-effort.

Schedule and grace advice

Use the Cloud Scheduler cron expression and time zone for the check. The grace time should cover container start-up, the normal runtime and all retries: with --task-timeout=30m and --max-retries=2, a run can legitimately take up to about 90 minutes. Set max runtime to the same worst case so a run that sent /start and then vanished still goes down.

Troubleshooting

• No executions at all: run gcloud scheduler jobs describe nightly-report-trigger --location=europe-west6 and check the state and last attempt status. A PERMISSION_DENIED usually means the service account lacks roles/run.invoker on the job.
• Executions, but no pings: check the task logs. Jobs with only internal egress through a VPC connector need Cloud NAT to reach the internet.
• Fail alert, then success: you are pinging on intermediate attempts; check the attempt condition.

FAQ

Why send the fail ping only on the last attempt?

Cloud Run retries failed tasks up to --max-retries times. A failure on an earlier attempt may recover, so reporting it would cause an alert followed by a recovery for a run that ended up fine.

Do I need Cloud Workflows to monitor a Cloud Run job?

No. For single-task jobs, the entrypoint wrapper is enough. Workflows is useful when an execution has many tasks and you want one result for the whole execution.

Can I send the job output with the ping?

Yes. Send a POST with the log tail as the body, for example tail -c 10000 /tmp/out.log | curl -fsS -m 10 --data-binary @- "$HB/$code". The first 10 KB are stored with the ping.