Threshold alerting from a cron job: when is 5xx noise an incident?

Polling an errors API is easy; picking the firing rule isn't. Windows, counts, cooldowns and a stdlib Python evaluator that pages you on rate, not presence.

By infrai-platform-engineeringPublished 2026-07-26API verified 2026-07-26

Polling is the easy half. A cron entry every five minutes reads GET /v1/errors/list on Infrai, buckets the events by group, and decides whether anything crossed a line — reads in this namespace are free and rate-limited, so 288 polls a day costs nothing. The hard half is the line itself: alert on the first occurrence and you’ll mute the channel by Thursday, alert on a daily digest and you’ll find out about the outage from a customer.

So this page is mostly about firing rules. It assumes you already have failed background jobs and 5xx exceptions being captured somewhere, and that you’d rather run sixty lines of Python on a schedule than adopt a monitoring vendor. If you want server-side alert rules with escalation policies and a maintained mobile pager, Datadog monitors and Sentry alerts both do that properly and this isn’t a fair fight.

Rate, not presence

One ECONNRESET against a payment provider at 3am is weather. Forty in ten minutes is an incident. The difference isn’t the error — it’s identical text in both cases — it’s the derivative, and any rule that ignores the time axis will either page on weather or sleep through the incident.

Three numbers turn a poll into a rule: a window (how far back you count), a threshold (how many inside it), and a cooldown (how long before the same group can page again). Get those three right and the channel stays credible.

Four signals, four different thresholds

Signal	Where it comes from	Threshold that behaves	What the naive version does
A failure shape nobody has seen	`is_new_group: true` on capture	1 — page immediately	Nothing; it’s buried under the known noise
A burst of 5xx from one endpoint	count per group inside the window	5 in 15 minutes	One flaky retry wakes someone
A slow burn that never spikes	group `count` delta between polls	20/hour, sustained twice	Never fires at all
A queue worker that stopped running	not visible in error data	needs a heartbeat, not a threshold	Silence reads as health

The last row is a genuine limitation of this design and the reason to be honest about it: an errors API can only tell you about jobs that ran and failed. A job that never started emits nothing, so no threshold over captured events will ever catch it — that needs a separate heartbeat signal, which we cover in cron heartbeat monitoring.

Reading the window

The list route returns events newest-first, 100 at a time, with an offset cursor. There’s no server-side time filter, so the loop pages until it walks off the back of your window:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/errors/list?environment=production&level=error&limit=3" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "items": [
      {
        "event_id": "evt_err_qSu4p8MkHPJGV4Ux5FM94r3F",
        "error_group_id": "errgrp_YnTmNG9cTD39ePhpUojeCT0a",
        "timestamp": "2026-07-26T01:15:47.278790Z",
        "level": "error",
        "title": "alert delivery failed: Slack webhook returned HTTP 500 after 3 attempts",
        "environment": "production",
        "release": "ops-2026.07.3",
        "tags": {}
      }
    ],
    "next_cursor": "1",
    "total": 91
  }
}

environment, level and release are honoured here. On the groups route they aren’t — that one filters by status and quietly ignores everything else, which is a trap worth knowing about before you build a rule on a filter that isn’t applied. The neighbouring guide on poll-based alerting measured that asymmetry in detail.

The evaluator

Stdlib only — no pip install, runs on Python 3.11 or newer, and safe to drop straight into a crontab:

#!/usr/bin/env python3
"""Fire a Slack alert when one error group crosses a rate threshold."""
import json
import os
import pathlib
import urllib.error
import urllib.request
from datetime import datetime, timedelta, timezone

API_KEY = os.environ["INFRAI_API_KEY"]
SLACK_URL = os.environ["SLACK_WEBHOOK_URL"]
BASE = "https://api.infrai.cc"
STATE = pathlib.Path(os.environ.get("ALERT_STATE", "/var/tmp/infrai-alert-state.json"))

WINDOW = timedelta(minutes=15)
THRESHOLD = 5
COOLDOWN = timedelta(minutes=30)


def api_get(path):
    req = urllib.request.Request(
        f"{BASE}{path}", headers={"authorization": f"Bearer {API_KEY}"}
    )
    with urllib.request.urlopen(req, timeout=15) as resp:
        return json.loads(resp.read())["data"]


def events_since(cutoff):
    """Page the list route newest-first until events fall outside the window."""
    cursor, collected = None, []
    for _ in range(20):  # hard page cap: 2,000 events is plenty for 15 minutes
        path = "/v1/errors/list?environment=production&level=error&limit=100"
        if cursor:
            path += f"&cursor={cursor}"
        data = api_get(path)
        for item in data["items"]:
            stamp = datetime.fromisoformat(item["timestamp"].replace("Z", "+00:00"))
            if stamp < cutoff:
                return collected
            collected.append(item)
        cursor = data.get("next_cursor")
        if not cursor:
            return collected
    return collected


def notify(group_id, title, count):
    body = json.dumps(
        {"text": f":rotating_light: {count} errors in 15 min — {title}\n{group_id}"}
    ).encode()
    req = urllib.request.Request(
        SLACK_URL, data=body, headers={"content-type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return resp.status


def main():
    now = datetime.now(timezone.utc)
    state = json.loads(STATE.read_text()) if STATE.exists() else {}

    counts, titles = {}, {}
    for item in events_since(now - WINDOW):
        gid = item["error_group_id"]
        counts[gid] = counts.get(gid, 0) + 1
        titles.setdefault(gid, item.get("title", "")[:120])

    fired = 0
    for gid, count in counts.items():
        if count < THRESHOLD:
            continue
        last = state.get(gid)
        if last and now - datetime.fromisoformat(last) < COOLDOWN:
            continue
        try:
            notify(gid, titles[gid], count)
        except (urllib.error.URLError, TimeoutError) as exc:
            print(f"delivery failed for {gid}: {exc}")
            continue
        state[gid] = now.isoformat()
        fired += 1

    STATE.write_text(json.dumps(state))
    print(f"{len(counts)} active groups, {fired} alerts sent at {now.isoformat()}")


if __name__ == "__main__":
    main()

Crontab line, with output kept so a failing job leaves a trail:

*/5 * * * * INFRAI_API_KEY=your_infrai_api_key SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/xxxx /usr/bin/python3 /opt/ops/alert.py >> /var/log/infrai-alert.log 2>&1

Picking the delivery channel

Channel	Latency to a human	Survives a 3am outage?	Good for
Slack incoming webhook	Seconds, if someone’s watching	No — nobody is looking	Daytime engineering signal
Transactional email	Minutes	Sometimes	Digests, threshold-crossed summaries
SMS	Seconds	Yes	The two or three rules that justify waking someone
A queue you drain elsewhere	Whatever your worker does	Depends	Fanning one alert to several places

The same Infrai key that reads the error groups also sends the email and the SMS, and pushes the queue message — that’s the practical argument for keeping alerting on the account you already have, rather than the per-event price. One credential, one bill, no second SDK for the notification leg.

Watching the watcher

An alerting script that dies silently is worse than none. Capture its own failures with a fixed fingerprint, so repeated delivery problems land in one group instead of a hundred:

curl -sS -X POST "https://api.infrai.cc/v1/errors/capture" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "alert delivery failed: Slack webhook returned HTTP 500 after 3 attempts (group errgrp_Aareim7M8ldNcRlLaDr0WL4L)",
    "exception": "AlertDeliveryError",
    "fingerprint": "alerting:slack-webhook:delivery-failed",
    "environment": "production",
    "release": "ops-2026.07.3"
  }'

{
  "ok": true,
  "data": {
    "event_id": "evt_err_qSu4p8MkHPJGV4Ux5FM94r3F",
    "fingerprint": "48240d533c19eafc7697fd219a26977cdec6dd8291b0bc1083335662d89da896",
    "error_group_id": "errgrp_YnTmNG9cTD39ePhpUojeCT0a",
    "is_new_group": true,
    "dashboard_url": "https://infrai.cc/projects/proj_local/errors/evt_err_qSu4p8MkHPJGV4Ux5FM94r3F"
  }
}

The exception string is accepted and normalised — the store keeps your message text, not a parsed exception object — so put anything you want to read later into message.

What the loop costs

Reads are free. Only capture is metered, at $0.00005 per event, and you can check what you’re really paying rather than trusting a number in a document:

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | python3 -c "import json,sys; d=json.load(sys.stdin)['data']; print([b for b in d['breakdown'] if b['key']=='errors.capture'])"

On the account we tested from, that returned 84 captures for $0.0042 in the trailing 30 days — the arithmetic holds. Prices in this direction tend to fall rather than rise, so read it live; the durable facts are that reads don’t bill and capture bills per event.