Feature flag toggles, retries and duplicate writes: making rollouts idempotent

Why absolute-state toggle and rollout calls survive retries, which failures are worth retrying, and how to record duplicate-write incidents on Infrai's errors API.

The duplicate-write problem with feature flags is almost always a modelling problem, not a concurrency one. A call that says “set checkout-v2 to enabled” or “set rollout to 25%” can be replayed a hundred times and the flag ends up in the same state; a call that says “bump the rollout by 10%” cannot. Model every toggle and rollout write as absolute state and the retry question mostly dissolves. Infrai’s errors namespace is where the leftovers go — the cases where a timeout left you genuinely unsure whether the write landed, which is a real incident worth recording with POST /v1/errors/capture and a stable fingerprint.

That leftover set is small but nasty, because a client timeout tells you nothing about the server. This page covers the retry policy, the read-back check that resolves the ambiguity, and how to audit the incidents afterwards.

Absolute state retries cleanly. Deltas don’t.

The last row is the one that bites teams who did everything else right. The flag write is idempotent, the audit record next to it isn’t, and after a retry storm the change log shows the same toggle applied four times by the same person in the same second.

An idempotency key fixes that half: generate one per logical intent — not per attempt — and have the receiving side store it with a unique constraint.

Which failures are worth retrying

A connection error, a request timeout, an HTTP 429 and a 502 or 503 are all worth retrying, because the request may never have been processed or the service is asking you to slow down. An HTTP 400, 401, 403, 404 or 422 is a statement about your request, and replaying it just burns quota — the second attempt will fail the same way.

The ambiguous one is a client-side timeout on an absolute-state write. The server may have applied it, may not have, and you can’t tell from where you’re standing. With absolute state that’s fine: retry, and if both attempts land, the flag ends up in exactly the state you asked for.

Retry with jitter, not a fixed backoff, or your whole fleet retries in lockstep and you’ve built a synchronised thundering herd.

const FLAG_API = process.env.FLAG_API_URL;
if (!FLAG_API) throw new Error("FLAG_API_URL is not set");

const RETRYABLE_STATUS = new Set([408, 425, 429, 500, 502, 503, 504]);
const BASE_DELAY_MS = 250;
const MAX_ATTEMPTS = 3;

const sleep = (ms) => new Promise((done) => setTimeout(done, ms));
const backoff = (attempt) => Math.round(BASE_DELAY_MS * 2 ** attempt * (0.5 + Math.random() / 2));

export async function setFlagState({ key, desired, idempotencyKey }) {
  let lastProblem = null;
  for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) {
    try {
      const res = await fetch(`${FLAG_API}/flags/${encodeURIComponent(key)}`, {
        method: "PUT",
        headers: {
          "Content-Type": "application/json",
          "Idempotency-Key": idempotencyKey,
        },
        body: JSON.stringify(desired),
        signal: AbortSignal.timeout(2000),
      });
      if (res.ok) return { applied: true, attempts: attempt + 1 };
      if (!RETRYABLE_STATUS.has(res.status)) {
        return { applied: false, attempts: attempt + 1, status: res.status, retryable: false };
      }
      lastProblem = `HTTP ${res.status}`;
    } catch (err) {
      lastProblem = err.name === "TimeoutError" ? "client timeout" : err.message;
    }
    if (attempt < MAX_ATTEMPTS - 1) await sleep(backoff(attempt));
  }
  return { applied: null, attempts: MAX_ATTEMPTS, uncertain: true, lastProblem };
}

applied: null is the interesting return value. Not true, not false — unknown, and the caller has to do something about it.

Resolving the unknown with a read-back

import { setFlagState } from "./flags.js";

const FLAG_API = process.env.FLAG_API_URL;

async function readFlag(key) {
  const res = await fetch(`${FLAG_API}/flags/${encodeURIComponent(key)}`, {
    signal: AbortSignal.timeout(2000),
  });
  if (!res.ok) throw new Error(`read-back failed: HTTP ${res.status}`);
  return res.json();
}

export async function setFlagChecked({ key, desired, idempotencyKey }) {
  const result = await setFlagState({ key, desired, idempotencyKey });
  if (result.applied === true) return result;

  const observed = await readFlag(key);
  const matches = Object.entries(desired).every(([field, want]) => observed[field] === want);
  return { ...result, applied: matches, observed, drift: matches ? null : desired };
}

Read-back turns “I don’t know” into “I know now” for the price of one extra GET. It’s the cheapest reconciliation you’ll ever write, and it only runs on the unhappy path.

Recording what actually went wrong

When the read-back disagrees with what you asked for — or when your audit table shows two rows for one idempotency key — that’s an incident, and it belongs in the error store with a fingerprint that groups every occurrence of the same defect:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/errors/capture" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "duplicate write detected: retry of toggle checkout-v2 applied twice (idempotency_key=tgl_9c41b2, attempt=3)",
    "exception": "DuplicateWrite",
    "fingerprint": "flags:toggle:duplicate-write",
    "environment": "production",
    "release": "2026.07.6"
  }'

{
  "ok": true,
  "data": {
    "event_id": "evt_err_6y9BJioIO0B6lIQdsHCh0vgn",
    "fingerprint": "45f817427c733507a40fe33e2c4b8f7249ae19be09dcf38634430dca598392ef",
    "error_group_id": "errgrp_tjwxLYEb285OvXUa1LhWeKBQ",
    "is_new_group": true,
    "dashboard_url": "https://infrai.cc/projects/proj_local/errors/evt_err_6y9BJioIO0B6lIQdsHCh0vgn"
  }
}

Keep the idempotency key in the message, never in the fingerprint. The key is unique per intent, so putting it in the grouping string would produce one group per incident and destroy the count that tells you how often this happens.

Auditing the pattern afterwards

curl -sS "https://api.infrai.cc/v1/errors/group_detail/errgrp_tjwxLYEb285OvXUa1LhWeKBQ" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

count is your duplicate-write rate, first_seen_at dates the regression, and the releases array names the deploy that introduced it. For the individual occurrences — which flag, which key, which attempt number — read the events:

curl -sS "https://api.infrai.cc/v1/errors/events/errgrp_tjwxLYEb285OvXUa1LhWeKBQ?limit=50" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.data.items[] | {timestamp, message, release}'

Free-text lookup across everything, when you only remember the flag name, runs through GET /v1/errors/search with a q parameter.

The catch: capture itself isn’t deduplicated

Worth flagging before you wire this into a retry loop — the capture route has no idempotency key of its own. We sent the identical body twice against the live API and got two distinct event_id values in one group, with is_new_group false on the second and the group count incremented to 2. So a report emitted inside your own retry loop inflates the count of the very thing you’re measuring.

Two ways to handle it, both cheap. Report once per logical incident, after the retries are exhausted, rather than once per attempt — that’s what the wrapper above does by returning a single result. Or accept the inflation and read the trend instead of the absolute number, since the fingerprint keeps everything in one group either way.

Cost, and where flags themselves live

Capture is billable at $0.00005 per event, verified 2026-07-26; the read routes (list, search, get, groups, group_detail, events) are free and rate-limited, and a new account carries $2 of credit. Duplicate-write incidents are rare by definition, so this is a rounding error on any bill — pull the live figures rather than trusting the paragraph:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '[.capabilities[] | select(.namespace == "errors" or .namespace == "flags") | {id, method, path, billable: .billing.is_billable}]'

That filter answers the other half of the question too. Flag storage, toggles and percentage rollouts live in their own namespace on the same key and the same bill, and the manifest is generated from the same source the router uses — so the routes it lists are the routes that exist today, which is a better contract than any list transcribed into an article. Rates in this catalogue drift downward over time and discount campaigns run, so what you read may well be lower than what’s printed here.

Where this stops

The errors API doesn’t do alerting, on-call routing or anomaly detection, so nothing here pages you when the duplicate-write group starts growing — you’re polling, or you’re wiring the capture response into an email or SMS call on the same key. Sentry and Datadog both do that part properly and are the right answer if alert rules are what you’re missing. For flag management itself — approval workflows, audit trails, environment promotion, targeting rules — a dedicated platform like Unleash or LaunchDarkly does far more than a key-value toggle store, and if governance around who can flip production flags is the actual requirement, that’s where to look.

What the pattern here buys you is smaller and, in our testing, underrated: retries that can’t corrupt state, one extra GET to resolve the ambiguous case, and a grouped, countable record of the times it went wrong anyway.

References

• Infrai errors API reference
• Infrai documentation home
• MDN: idempotent
• Timeouts, retries and backoff with jitter — Amazon Builders’ Library
• Unleash: feature toggles