Cron jobs, workers and the failures that never throw an exception

Background work fails three ways: it throws, it finishes wrong, or it never runs. How to record each with Infrai's errors API, plus a watchdog for the runs that never happened.

An exception tracker sees one of the three ways a background job dies. It sees the throw. It doesn’t see the reconcile that finished in 40ms because the query returned zero rows, and it certainly doesn’t see the job that never started because the scheduler container was evicted an hour ago. Infrai’s errors namespace gives you a route for each of the first two — POST /v1/errors/capture for thrown exceptions, a free message route for anomalies that don’t throw — and the third needs a watchdog reading from outside the process.

That third case is why “cron monitoring” is a separate product category. Purpose-built services like Cronitor model your schedule and page you when a run is late; Sentry’s cron monitors do the same thing next to your exceptions. What’s below is the version you can build on read routes that cost nothing, and an honest account of where it’s thinner than the dedicated tools.

Three failure modes, three signals

The middle row is where most real damage hides. A payout batch that processes zero payments looks identical to a quiet Tuesday, and by the time finance notices, you’re reconstructing four days of history from database timestamps.

Wrapping the job

const API = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
const ENVIRONMENT = process.env.APP_ENV ?? "production";
const RELEASE = process.env.APP_RELEASE ?? "dev";
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const headers = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

async function note(text, level) {
  try {
    const res = await fetch(`${API}/v1/errors/message`, {
      method: "POST",
      headers,
      body: JSON.stringify({ text, level, environment: ENVIRONMENT, release: RELEASE }),
      signal: AbortSignal.timeout(5000),
    });
    if (!res.ok) console.error(`note rejected: HTTP ${res.status}`);
  } catch (err) {
    console.error(`note unreachable: ${err.message}`);
  }
}

async function failure(job, err) {
  try {
    const res = await fetch(`${API}/v1/errors/capture`, {
      method: "POST",
      headers,
      body: JSON.stringify({
        message: String(err?.stack ?? err).slice(0, 8000),
        exception: err?.name ?? "Error",
        fingerprint: `job:${job}:${err?.name ?? "Error"}`,
        environment: ENVIRONMENT,
        release: RELEASE,
      }),
      signal: AbortSignal.timeout(5000),
    });
    if (!res.ok) console.error(`capture rejected: HTTP ${res.status}`);
  } catch (sendErr) {
    console.error(`capture unreachable: ${sendErr.message}`);
  }
}

export async function runJob(job, fn, { expectWork = true } = {}) {
  const started = Date.now();
  try {
    const result = await fn();
    const processed = Number(result?.processed ?? 0);
    if (expectWork && processed === 0) {
      await note(`job:${job} completed with 0 processed items`, "warning");
    }
    await note(`heartbeat job:${job} ok processed=${processed} ms=${Date.now() - started}`, "info");
    return result;
  } catch (err) {
    await failure(job, err);
    throw err;
  }
}

Three choices in there are deliberate. The fingerprint is job:<name>:<ErrorClass> — never the job id, never a row id, or one broken worker becomes ten thousand groups. The zero-work check is opt-out per job, because plenty of jobs legitimately find nothing to do and you don’t want a nightly false alarm. And the heartbeat goes out on the success path only, which is what makes its absence meaningful.

Heartbeats and warnings are free. Only the exception capture is metered, so instrumenting a worker this thoroughly costs the same as instrumenting it badly.

The watchdog for runs that never happened

Nothing inside a job that didn’t run can tell you it didn’t run. Something else has to notice the silence:

const API = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const EXPECTED = [
  { job: "nightly-reconcile", maxAgeMinutes: 90 },
  { job: "invoice-sweep", maxAgeMinutes: 30 },
];

async function lastHeartbeat(job) {
  const q = encodeURIComponent(`heartbeat job:${job}`);
  const res = await fetch(`${API}/v1/errors/search?q=${q}&limit=20`, {
    method: "GET",
    headers: { Authorization: `Bearer ${KEY}` },
    signal: AbortSignal.timeout(10_000),
  });
  if (!res.ok) throw new Error(`search failed: HTTP ${res.status}`);
  const { data } = await res.json();
  const stamps = (data.items ?? []).map((e) => Date.parse(e.timestamp)).sort((a, b) => b - a);
  return stamps[0] ?? null;
}

let late = 0;
for (const { job, maxAgeMinutes } of EXPECTED) {
  try {
    const seen = await lastHeartbeat(job);
    const ageMin = seen === null ? Infinity : (Date.now() - seen) / 60_000;
    if (ageMin > maxAgeMinutes) {
      late += 1;
      console.error(`LATE ${job}: last heartbeat ${seen === null ? "never" : `${Math.round(ageMin)}m ago`}`);
    } else {
      console.log(`ok ${job}: ${Math.round(ageMin)}m ago`);
    }
  } catch (err) {
    console.error(`watchdog could not check ${job}: ${err.message}`);
    late += 1;
  }
}
process.exitCode = late ? 1 : 0;

Run this somewhere your application isn’t — a different host, a different provider, ideally a different scheduler. A watchdog sharing a failure domain with the thing it watches is decoration.

The same search from the shell, which is the fastest way to confirm your markers are landing:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/errors/search?q=heartbeat&limit=5" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Search takes q and ignores everything else you pass alongside it, so filter on environment or level in your own code after the fact. An empty query is rejected with a 400 rather than returning everything.

Triage when a worker does blow up

curl -sS "https://api.infrai.cc/v1/errors/group_detail/errgrp_17E05u607XMVSasuMpsrMX7w" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

The group carries count, first_seen_at, last_seen_at and a representative_event — enough to tell a one-off from a loop without paging through every occurrence. Unknown ids come back as ERROR_GROUP_NOT_FOUND, which is a useful assertion in a test.

Closing one out is a single POST:

curl -sS -X POST "https://api.infrai.cc/v1/errors/resolve/{error_group_id}" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{}'

That returns the updated group with is_resolved flipped to true — we checked against the live API, where the field is is_resolved rather than the status string you might expect from other trackers.

What it costs to instrument every job

Captures bill at $0.00005 each, verified 2026-07-26; heartbeats, warnings and all reads are free. A fleet emitting a marker every five minutes across ten jobs — about 86,000 messages a month — still bills $0.00 for that half. New accounts also start with $2 of free credit against the metered side. Confirm both:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id | startswith("errors.")) | {id, free: .billing.free}'

Rates in this catalogue tend to move down, so treat the figure as a ceiling rather than a promise. What’s structural is the split: writes that carry an exception are metered, everything else in this namespace isn’t.

Where the dedicated tools win

This is not schedule-aware. There’s no cron expression to register, no concept of a grace period, no automatic detection of a run that started and hung, and no UI listing your jobs by health. You’re encoding the expected interval in the watchdog’s own config, which drifts from the real crontab the moment someone edits one and not the other — that’s the honest drawback of building it this way. If job monitoring is the problem you’re solving rather than a side effect of error tracking, Cronitor and Healthchecks.io model it properly, and Sentry ships cron monitors beside its exception store. Datadog can cover it too if you’re already paying for the agents — a monitor on a custom metric your job emits does the same job as the watchdog above, with a schedule you configure in one place.

The case for doing it here is narrower and still real: your workers already need somewhere to send exceptions, the extra signals cost nothing, and one API key covers the queue, the schedule and the store instead of three.

References

• Infrai errors API reference
• ERROR_GROUP_NOT_FOUND error reference
• Sentry cron monitoring
• Cronitor
• Healthchecks.io cron monitoring