Metrics-based failure alerting for a SaaS API: rule shapes that don't flap

Threshold, ratio, two-window burn rate, absence — the four alert rules worth writing, and a TypeScript evaluator that polls Infrai's free metrics query route.

Getting numbers out of your API is the easy half. Infrai gives you POST /v1/metrics/report to write a counter and a free GET /v1/metrics/query to read an aggregate back over a time window, and after an hour you have job.run, job.failed and a p99 latency gauge. The half that decides whether anyone trusts the alerts is the rule you evaluate against those numbers.

Most homegrown alerting fails the same way. Someone writes if (failures > 5) page(), it fires during a deploy, it fires again nine minutes later, and within a fortnight the channel is muted. The fix isn’t a better monitoring vendor — it’s picking a rule shape that matches the failure you’re afraid of.

Four shapes, and when each is the right one

Absence is a different enough problem that it’s covered separately in the cron heartbeat guide at https://docs.infrai.cc/en/guides/metrics/answers/nextjs-nodejs-cron-job-heartbeat-monitoring-missed-run/. The other three are what follows.

You need a denominator

A ratio rule is only as good as the total it divides by, and the mistake is instrumenting failures alone. Count every attempt, tag it with the outcome:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/metrics/report" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "job.run",
    "value": 1,
    "type": "counter",
    "tags": {"job": "sync-subscriptions", "outcome": "failed", "env": "production"}
  }'

{
  "ok": true,
  "data": { "accepted": true, "metric_id": "metric_BTCkNRxQICuQN29ZNd9mLIdF" },
  "metadata": { "request_id": "req_b1917e74d0004e668edef805", "latency_ms": 24 }
}

One metric name, one outcome tag, two questions answered. In a worker that looks like this:

const API = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

type Tags = Record<string, string>;

async function count(name: string, tags: Tags): Promise<void> {
  try {
    const res = await fetch(`${API}/v1/metrics/report`, {
      method: "POST",
      headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
      body: JSON.stringify({ name, value: 1, type: "counter", tags }),
    });
    if (!res.ok) console.error(`metric ${name}: HTTP ${res.status}`);
  } catch (err) {
    console.error(`metric ${name} unreachable`, err);
  }
}

export async function runJob<T>(job: string, work: () => Promise<T>): Promise<T> {
  try {
    const out = await work();
    await count("job.run", { job, outcome: "ok", env: "production" });
    return out;
  } catch (err) {
    await count("job.run", { job, outcome: "failed", env: "production" });
    throw err;
  }
}

Reading either side is a tag filter plus a window, and reads are free:

curl -sS "https://api.infrai.cc/v1/metrics/query?name=job.run&agg=sum&window=1h&tag.job=sync-subscriptions&tag.outcome=failed" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

curl -sS "https://api.infrai.cc/v1/metrics/query?name=job.run&agg=sum&window=1h&tag.job=sync-subscriptions" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "name": "job.run",
    "agg": "sum",
    "points": [
      { "ts": "2026-07-26T08:00:00Z", "value": 118.0 },
      { "ts": "2026-07-26T09:00:00Z", "value": 131.0 }
    ]
  }
}

since and until pin an absolute range instead, which is what you want when you’re reconstructing an incident timeline rather than watching live. A typo in agg comes back as a 400 rather than a plausible-looking number.

The evaluator

Because the read side takes a window, the burn-rate rule is just two queries at two resolutions — no ring buffer, no history file, no drift when the box reboots. The only state worth persisting is the fact that a page has already gone out.

import { readFileSync, writeFileSync, existsSync } from "node:fs";

const API = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const STATE = "/var/tmp/alert-state.json";
const FAST = "5m";        // "is it broken right now"
const SLOW = "1h";        // "has it been broken long enough to matter"
const FAST_RATIO = 0.2;
const SLOW_RATIO = 0.05;
const MIN_SAMPLE = 20;    // ignore ratios computed from a handful of runs
const COOLDOWN_MS = 30 * 60 * 1000;

interface State { lastPagedAt: number; firing: boolean; }

async function total(window: string, outcome?: string): Promise<number> {
  const url = new URL(`${API}/v1/metrics/query`);
  url.searchParams.set("name", "job.run");
  url.searchParams.set("agg", "sum");
  url.searchParams.set("window", window);
  url.searchParams.set("tag.job", "sync-subscriptions");
  if (outcome) url.searchParams.set("tag.outcome", outcome);
  const res = await fetch(url, { headers: { authorization: `Bearer ${KEY}` } });
  if (!res.ok) throw new Error(`query failed: HTTP ${res.status}`);
  const body = await res.json();
  return body.data.points.reduce((n: number, p: { value: number }) => n + p.value, 0);
}

const state: State = existsSync(STATE)
  ? JSON.parse(readFileSync(STATE, "utf8"))
  : { lastPagedAt: 0, firing: false };

const [fastFailed, fastAll, slowFailed, slowAll] = await Promise.all([
  total(FAST, "failed"), total(FAST), total(SLOW, "failed"), total(SLOW),
]);

const ratio = (failed: number, all: number) => (all >= MIN_SAMPLE ? failed / all : 0);
const fast = ratio(fastFailed, fastAll);
const slow = ratio(slowFailed, slowAll);

// Hysteresis: fire when both windows agree, clear only when the fast window
// has fully recovered. Without the asymmetry the alert oscillates on the edge.
const shouldFire = fast > FAST_RATIO && slow > SLOW_RATIO;
const shouldClear = state.firing && fast < FAST_RATIO / 2;

if (shouldFire && Date.now() - state.lastPagedAt > COOLDOWN_MS) {
  console.error(`PAGE sync-subscriptions fast=${(fast * 100).toFixed(1)}% slow=${(slow * 100).toFixed(1)}%`);
  state.lastPagedAt = Date.now();
  state.firing = true;
} else if (shouldClear) {
  console.log("RESOLVED sync-subscriptions");
  state.firing = false;
}

writeFileSync(STATE, JSON.stringify(state, null, 2));

Three things in there are the difference between an alert people act on and one they mute. MIN_SAMPLE stops a 1-in-2 failure at 03:00 from claiming a 50% error rate. The two windows have to agree, so a single bad deploy minute doesn’t page. And COOLDOWN_MS means an ongoing incident produces one message every 30 minutes rather than one every poll.

Pick the two windows deliberately. A 5m/1h pair catches a fast burn in about five minutes and tolerates a blip; 30m/6h is the shape you want for a slow leak you’d rather investigate on Monday. Running both pairs against the same counters is fine — the reads cost nothing.

Wire it to whatever runs on a schedule:

# every 5 minutes, from a host that is not the host being monitored
*/5 * * * * app INFRAI_API_KEY=your_infrai_api_key /usr/bin/node /srv/ops/evaluate-alerts.mjs >> /var/log/alerts.log 2>&1

What the polling costs

Nothing on the read side: GET /v1/metrics/query is free, though rate-limited per account, and a five-minute cadence sits comfortably inside that. Writes are the billable half — POST /v1/metrics/report and POST /v1/metrics/batch at $0.001 per metric point, verified 2026-07-26.

The denominator is the expensive half of a ratio rule, and it’s worth sizing before you instrument. A worker doing 5,000 jobs a day that reports every single attempt writes 5,000 points, or about $5 a day. Counting in process and emitting one point per (job, outcome) pair every five minutes writes 576 — call it 58 cents — and the ratio it yields is identical, because a ratio only ever needed the sums. Batching many points into one metrics/batch request saves round trips rather than money; the rollup is what changes the bill. New accounts get $2 in credit to try the shape.

Read today’s figure rather than this paragraph — these rates have moved down more than once:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.namespace == "metrics") | {id, method, path, price: .billing.price_usd, unit: .billing.unit}'

What this approach doesn’t do

There are no server-side alert rules. Nothing on the platform evaluates a threshold for you, there’s no notification routing, no escalation policy and no on-call schedule — so if the machine running your evaluator dies, your alerting dies silently with it. Prometheus with Alertmanager, or Grafana’s alerting, or a Datadog monitor with PagerDuty behind it, all solve that properly and it’s worth paying for once more than one person carries a pager.

The other limitation is arithmetic the query route won’t do for you. It aggregates one metric name at a time, so a ratio is two calls and a division in your code, and there are no funnels, no joins across metric names and no recording rules to precompute anything. That’s fine for a dozen rules on one box; past that you want a query language, and PromQL is the one to want.

The reason to start here anyway is that the alert is rarely the end of the workflow. The same key that answers the query also sends the mail, captures the exception with its stack trace, stores the incident artefact and runs the cron — so the follow-on work doesn’t need a new account, a new SDK or a second invoice at the end of the month.

References

• https://docs.infrai.cc/en/api/metrics/
• https://docs.infrai.cc/errors/RATE_LIMIT_ACCOUNT/
• https://sre.google/workbook/alerting-on-slos/
• https://prometheus.io/docs/practices/alerting/