Auto-rollback a bad release in Node: error-rate check, then flip the flag

Tag counters with the release, read the ratio back from Infrai's metrics API, and roll back by rewriting the flag's default value — with the version guard that makes it safe.

A release gate is two jobs stapled together: decide whether the new code is worse, and turn it off if it is. Both halves live on one Infrai key — counters go to POST /v1/metrics/report, the verdict comes from GET /v1/metrics/query, and the rollback is POST /v1/flags/set with the flag’s current version. No agent, no scrape target, no second vendor.

The half that trips people up is the second one, because there are two ways to turn a flag off and they mean different things. POST /v1/flags/toggle/{key} with {"enabled": false} is the kill switch: it leaves default_value where it is and GET /v1/flags/get_value/{key} starts answering with the off value straight away. POST /v1/flags/set rewrites default_value itself. A gate that might have to hand the flag back to a human should use set, because the value it wrote is the value a colleague reads.

The release is the window, not the clock

Most rollback tutorials reach for a clock: “error rate over the last 10 minutes.” GET /v1/metrics/query will do that — it takes name, an agg from avg, sum, count, p50 or p99, tag filters as tag.<key>=<value>, and since/until to bound the window, which really do bound it. Ask for a window in 2020 and points comes back empty. Ask for an agg that isn’t in the list and you get a 400 naming the five that are, rather than a plausible-looking zero.

Use it anyway, and you’ve still picked the wrong axis.

You don’t want “the last 10 minutes”. You want “this release”, and a release is a label you already have. Tag every counter with the build you deployed, and the sum for tag.release=web@2026.07.26-a3f1c2 is by definition the numbers that build produced — no clock skew, no window straddling the deploy boundary, no argument about whether the bake started when CI went green or when the last pod rolled. A time window is an approximation of the question; the release tag is the question.

Emitting the two counters

You need a numerator and a denominator. A raw error count says nothing: 37 errors is a catastrophe at 200 requests and noise at 200,000. Accumulate both in process and flush them together, so the whole minute is one HTTP call.

// gate-metrics.mjs — Node 22, no dependencies
const BASE = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
const RELEASE = process.env.RELEASE ?? "dev";
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

let requests = 0;
let errors = 0;
export const observe = (ok) => { requests++; if (!ok) errors++; };

export async function flush() {
  if (requests === 0) return;
  const points = [
    { name: "checkout_requests_total", value: requests, type: "counter", tags: { release: RELEASE, service: "checkout" } },
    { name: "checkout_errors_total", value: errors, type: "counter", tags: { release: RELEASE, service: "checkout" } },
  ];
  requests = 0; errors = 0;
  const res = await fetch(`${BASE}/v1/metrics/batch`, {
    method: "POST",
    headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
    body: JSON.stringify({ points }),
  });
  if (!res.ok) console.error("[gate] flush failed", res.status, await res.text());
}

setInterval(() => { flush().catch((e) => console.error("[gate]", e.message)); }, 60_000).unref();

The same two points sent by hand, so you can see the wire format:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST https://api.infrai.cc/v1/metrics/report \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "checkout_errors_total",
    "value": 37,
    "type": "counter",
    "tags": {"release": "web@2026.07.26-a3f1c2", "service": "checkout"}
  }'

POST /v1/metrics/batch checks every point before it accepts any of them. Leave name off the second entry of a two-point payload and the whole call comes back 400 with param: "points[1].name" — the index is in the error, so a mistake in your flush loop shows up in your own logs on the first deploy instead of as a gap in a dashboard you’d started trusting.

Reading the ratio back

Two free reads, one division.

curl -sS -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  "https://api.infrai.cc/v1/metrics/query?name=checkout_errors_total&agg=sum&tag.release=web@2026.07.26-a3f1c2"

{
  "ok": true,
  "data": {
    "name": "checkout_errors_total",
    "agg": "sum",
    "points": [{ "ts": "2026-07-26T05:50:18.362558Z", "value": 37.0 }]
  }
}

points always carries exactly one element. Against 812 requests for the same release that’s a 4.6% error rate, which for a checkout path is a rollback.

The rollback, and the guard around it

set is an upsert with optimistic locking: send the key, the current version, and the new default_value. The server bumps the version and every subsequent get_value returns the new answer.

curl -sS -X POST https://api.infrai.cc/v1/flags/set \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"key": "kb_release_gate_demo", "version": 1, "default_value": false}'

{
  "ok": true,
  "data": {
    "key": "kb_release_gate_demo",
    "type": "bool",
    "default_value": false,
    "enabled": true,
    "version": 2,
    "updated_at": "2026-07-26T05:50:17.377431Z"
  }
}

Two things about that response are worth reading carefully. enabled stays true, because set and toggle move different levers — you’ve changed what the flag serves, not whether it’s switched on. And if you pass a stale version you get an HTTP 409 instead of an overwrite:

{
  "ok": false,
  "error": {
    "code": "FLAG_VERSION_CONFLICT",
    "http_status": 409,
    "message": "expected version 1, current 3; refetch and retry",
    "retryable": false
  }
}

That’s the guard you want when a human is disabling the flag in one window while the gate is disabling it in another — one of them loses, loudly, instead of resurrecting the release.

For the read-back afterwards, GET /v1/flags/is_enabled/{key} is the cheapest check for a boolean: it answers with the effective value your app would see, so it reads {"enabled": true, "value": true} while the release is live and {"enabled": false, "value": false} the moment the gate has rolled it back, and it returns 404 FLAG_NOT_FOUND if CI is pointed at a key that was renamed. For a multi-variant flag read GET /v1/flags/get/{key} instead and inspect default_value yourself.

The gate, end to end

// release-gate.mjs — run from CI after the bake window
const BASE = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
const RELEASE = process.env.RELEASE;
const FLAG = process.env.FLAG_KEY ?? "kb_release_gate_demo";
const THRESHOLD = Number(process.env.ERROR_BUDGET ?? 0.02);
const MIN_SAMPLE = 200;
if (!KEY || !RELEASE) throw new Error("INFRAI_API_KEY and RELEASE are required");

const auth = { authorization: `Bearer ${KEY}`, "content-type": "application/json" };

async function call(path, init = {}) {
  const res = await fetch(`${BASE}${path}`, { headers: auth, ...init });
  const json = await res.json();
  if (!json.ok) throw new Error(`${path} -> ${res.status} ${json.error?.code ?? "unknown"}`);
  return json.data;
}

const sum = async (metric) => {
  const q = new URLSearchParams({ name: metric, agg: "sum", "tag.release": RELEASE });
  const data = await call(`/v1/metrics/query?${q}`);
  return data.points[0]?.value ?? 0;
};

const [requests, errors] = await Promise.all([
  sum("checkout_requests_total"),
  sum("checkout_errors_total"),
]);

if (requests < MIN_SAMPLE) {
  console.log(`hold: ${requests} requests is not enough to judge ${RELEASE}`);
  process.exit(75);
}

const rate = errors / requests;
console.log(`${RELEASE}: ${errors}/${requests} = ${(rate * 100).toFixed(2)}%`);
if (rate <= THRESHOLD) {
  console.log("release stays on");
  process.exit(0);
}

const flag = await call(`/v1/flags/get/${FLAG}`);
const rolled = await call("/v1/flags/set", {
  method: "POST",
  body: JSON.stringify({ key: FLAG, version: flag.version, default_value: false }),
});
console.error(`rolled back ${FLAG} at version ${rolled.version}`);
process.exit(1);

Exit code 75 is EX_TEMPFAIL, so a CI step can retry the bake instead of treating thin traffic as a pass.

What this gate doesn’t do

It doesn’t do statistical significance. LaunchDarkly’s guarded rollouts run a real comparison between variation cohorts and can stop a rollout on a regression a fixed threshold would miss — if you need that, buy it. This is a threshold on a ratio, and thresholds flap when traffic is low, which is what MIN_SAMPLE is for.

It also can’t show you the shape of the failure. One aggregate per call means no trend line, no burn-rate curve, no rate() over a sliding window of the kind Prometheus gives you.

The trade-off is easy to read off that table: this is the cheapest thing that can both decide and act, and the weakest thing at explaining. In practice a lot of teams keep Grafana for the explaining and use a gate like this only as the actuator.

What the gate costs to run

Verified 2026-07-26: POST /v1/metrics/report and every point inside POST /v1/metrics/batch cost $0.001, while all reads — metrics/query, every route under flags — are free. So the gate’s own cost is the flush loop, not the polling: one two-point batch a minute is about $2.88 a day per service, and dropping to one flush every 5 minutes takes that under $0.60. New accounts start with $2 of credit. Rates drift downward over time and campaigns run, so read today’s:

curl -sS -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  "https://api.infrai.cc/v1/account/usage"

That breakdown is per capability, which is the fastest way to notice that someone instrumented a per-request counter.

The structural point survives any repricing: writes cost, reads don’t, so aggregate in your process and let the gate poll as often as it likes.

References

• https://docs.infrai.cc/en/api/metrics/
• https://docs.infrai.cc/en/api/flags/
• https://docs.infrai.cc/errors/FLAG_VERSION_CONFLICT/
• https://launchdarkly.com/docs/home/releases/guarded-rollouts
• https://prometheus.io/docs/prometheus/latest/querying/functions/