Overlapping cron runs: what skip, allow and queue actually do

A 5-minute sync that sometimes takes six will double-run unless something stops it. Measured behaviour of Infrai's three overlap policies, plus the handler guard.

Overlap is a scheduling problem with a data-integrity blast radius, and it wants two fixes at two layers. At the scheduler, pick an overlap_policy — Infrai defaults to skip, which refuses to start a run while a previous one is still in flight and records the refusal. In your handler, take a lock anyway, because the scheduler can only serialise the runs it knows about.

The corruption you already hit almost certainly came from the second layer being absent. Infrai’s default would have prevented the specific case you describe; it would not have prevented a retry racing the next tick, and no scheduler will.

What each policy does, measured

We created three identical jobs against a deliberately slow endpoint — roughly a six-second response — and fired two runs at each within the same millisecond. The results are not what the enum names imply on first read.

Note the asymmetry that matters for alerting: a skipped run is not silence. It’s a durable row you can count. Most crontab-plus-flock setups throw that information away — the second process exits 1 and nobody hears about it.

Set the policy

curl -X POST https://api.infrai.cc/v1/cron/create \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "inventory-sync",
    "task": "https://example.com/hooks/inventory-sync",
    "cron_expr": "*/5 * * * *",
    "timezone": "UTC",
    "overlap_policy": "skip",
    "timeout_seconds": 240,
    "retry": 1
  }'

Note timeout_seconds sitting below the interval. A 5-minute schedule with a 240-second ceiling means a wedged run gets cut loose before the next tick is due, so overlap becomes an exception rather than a standing state. The ceiling is 900 seconds; you can’t ask for more.

Changing your mind later doesn’t require a rebuild:

curl -X PATCH https://api.infrai.cc/v1/cron/update/cron_baJarz5HYghX90u9Y8Ckc7BU \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"overlap_policy": "queue", "timeout_seconds": 240}'

What a refused run looks like from both sides

The caller — including a manual POST /v1/cron/trigger/{id} — gets a clear 400 rather than a queued acknowledgement:

{
  "ok": false,
  "error": {
    "code": "INVALID_ARGUMENT",
    "http_status": 400,
    "message": "cron job 'cron_baJarz5HYghX90u9Y8Ckc7BU' has an in-flight run; skipped",
    "retryable": false
  }
}

And the history keeps the evidence:

{
  "run_id": "cronrun_HWG3hSTW4zWojm65kWOEUUFL",
  "status": "skipped",
  "skipped_reason": "overlap_skip",
  "http_status": null,
  "duration_ms": null,
  "is_manual_trigger": true
}

skipped_reason has three values worth branching on — overlap_skip, max_runs_reached and disabled — so an alert can say why the job didn’t do anything, which is the question you actually have at 9am.

Turn skips into a signal

A job that skips once a week is healthy. A job skipping a third of its ticks has outgrown its schedule and is silently falling behind, which looks identical to “working fine” on every dashboard that only counts failures.

curl https://api.infrai.cc/v1/cron/runs/list/cron_baJarz5HYghX90u9Y8Ckc7BU \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Here’s the whole audit as a Node 22 script — it walks every job on the account and reports the skip rate, so you find the ones you’d forgotten about too.

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

async function api(path) {
  const res = await fetch(`https://api.infrai.cc${path}`, {
    headers: { authorization: `Bearer ${KEY}` },
  });
  const json = await res.json();
  if (!res.ok || json.ok === false) {
    throw new Error(`GET ${path} -> ${res.status} ${json?.error?.message ?? ""}`);
  }
  return json.data;
}

const { items: jobs } = await api("/v1/cron/list");

for (const job of jobs) {
  const { items: runs } = await api(`/v1/cron/runs/list/${job.job_id}`);
  if (!runs.length) continue;
  const overlapped = runs.filter((r) => r.skipped_reason === "overlap_skip").length;
  const rate = overlapped / runs.length;
  const slowest = Math.max(...runs.map((r) => r.duration_ms ?? 0));
  const verdict = rate > 0.1 ? "SCHEDULE TOO TIGHT" : "ok";
  console.log(
    `${job.name.padEnd(28)} policy=${job.overlap_policy.padEnd(5)} ` +
      `skipped=${overlapped}/${runs.length} slowest=${slowest}ms ${verdict}`,
  );
}

Ten per cent is an arbitrary threshold — pick your own — but the shape of the check is the point. Compare slowest against the interval and the answer usually writes itself.

The lock your handler still needs

Scheduler-level policy is scoped to one job on one schedule. It doesn’t cover a manual trigger fired by a colleague, a retry from the previous tick arriving late, a staging job pointed at production by accident, or a second job that touches the same rows for a different reason. Every one of those has corrupted somebody’s data.

So take an advisory lock at the top of the handler and return 200 quickly when you can’t get it — a fast, honest “already running” beats a slow duplicate.

import os
import sys

import psycopg

LOCK_KEY = 8842001  # any stable 64-bit int; one per logical job

with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT pg_try_advisory_lock(%s)", (LOCK_KEY,))
        got_lock = cur.fetchone()[0]

    if not got_lock:
        print("another inventory-sync run holds the lock; exiting cleanly")
        sys.exit(0)

    try:
        with conn.cursor() as cur:
            cur.execute("SELECT id FROM inventory WHERE synced_at IS NULL LIMIT 500")
            rows = cur.fetchall()
        print(f"syncing {len(rows)} rows")
        # ... perform the sync, committing in batches ...
        conn.commit()
    finally:
        with conn.cursor() as cur:
            cur.execute("SELECT pg_advisory_unlock(%s)", (LOCK_KEY,))

The version with the sharpest teeth is the one where the sync work itself is idempotent — keyed on a natural id and an ON CONFLICT DO UPDATE — because then a double-run is merely wasteful rather than destructive, and you stop needing to be right about locking at all.

A caveat that has bitten people

A timeout status means Infrai stopped waiting after timeout_seconds. It does not mean your server stopped working. If your handler ignores client disconnects — most WSGI and Express handlers do — the run is recorded as finished while the process keeps writing, and the next tick starts against a job the scheduler believes is idle. That’s the exact shape of the corruption you described, and no overlap_policy value fixes it. Make the handler honour cancellation, or make the work idempotent, or both.

Also worth flagging: run history is a read, not a stream. There’s no push notification when a run is skipped, so the audit above wants to be a scheduled job of its own. Set on_failure_webhook at create time if you want failures pushed to you, and note that a skip is not a failure — it won’t fire.

Where a different tool suits better

crontab with flock -n is still the right answer on a single box you already run: it’s zero dependencies and it’s been correct for twenty years. Quartz’s @DisallowConcurrentExecution gives you the same guarantee inside a JVM, cluster-wide, if your job is already Java. And if the sync is genuinely a multi-step pipeline where a partial run needs compensating, Temporal’s durable execution model is a better fit than any HTTP scheduler, including this one — Infrai fires requests and records outcomes, it doesn’t own your job’s state machine.

Cron, queue and error tracking sit on one Infrai key, which is the practical reason to keep the scheduler here: the fix for a sync that keeps overrunning is usually to have the tick enqueue work instead of doing it, and that’s another route on the same account rather than another vendor. Every route in this namespace — create, update, list, trigger, run history — is free and rate-limited, and doesn’t consume new-account trial credit. Check it yourself:

curl "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | python3 -c "import json,sys; print([c['billing'] for c in json.load(sys.stdin)['capabilities'] if c['id']=='cron.update'])"

Rates here move down over time, so treat that output as authoritative over any prose.

References

• https://docs.infrai.cc/en/api/cron/
• https://docs.infrai.cc/errors/CRON_OVERLAP_SKIPPED/
• https://cronitor.io/guides/prevent-duplicate-cron-jobs
• https://ma.ttias.be/prevent-cronjobs-from-overlapping-in-linux/