Exponential backoff and DLQ redrive for failed webhooks in Node.js

A jittered backoff ladder for outbound webhooks, what really lands in the dead-letter queue, and the triage-then-redrive runbook that gets messages moving again.

Backoff decides how long a broken subscriber costs you; the dead-letter queue decides whether anyone finds out. Infrai gives you both on one credential — a delayed publish for each rung of the ladder, an automatic dead-letter lane after three deliveries, and a redrive call to put messages back once the cause is fixed. What follows is the ladder, the triage script, and the parts we found broken while testing.

Getting the delays right matters less than most posts suggest. Getting the jitter right matters more.

The ladder, and why every retry needs noise

A partner’s TLS certificate expires at 09:00. Four thousand of your events fail inside a minute, and a naive doubling schedule retries all four thousand at 09:01, then all four thousand at 09:03. You’ve built a synchronised load generator pointed at a subscriber that’s already in trouble.

Full jitter fixes it: pick a random delay anywhere between zero and the ceiling for that attempt.

Five rungs cover a business day, which is about the longest a partner outage runs before someone picks up the phone.

import process from "node:process";

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY not found in the environment");

const CEILINGS = [30, 120, 600, 3600, 21600];
const QUEUE = "billing-callbacks";

function nextDelay(attempt) {
  const ceiling = CEILINGS[attempt];
  return ceiling === undefined ? null : Math.floor(Math.random() * ceiling);
}

async function send(path, payload) {
  const res = await fetch("https://api.infrai.cc" + path, {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(payload),
  });
  const parsed = await res.json();
  if (!parsed.ok) throw new Error(`${path}: ${parsed.error.code} ${parsed.error.message}`);
  return parsed.data;
}

export async function scheduleRetry(message, failure) {
  const attempt = (message.payload.attempt ?? 0) + 1;
  const delay = nextDelay(attempt - 1);

  if (delay === null) {
    await send("/v1/queue/publish", {
      queue: QUEUE + ".dlq",
      payload: { ...message.payload, attempt, gave_up_at: new Date().toISOString(), failure },
    });
  } else {
    await send("/v1/queue/publish", {
      queue: QUEUE,
      payload: { ...message.payload, attempt, last_failure: failure },
      delay_seconds: delay,
    });
  }

  await send("/v1/queue/ack", { queue: QUEUE, message_id: message.message_id });
  return { attempt, delay };
}

Recording last_failure on the message is the difference between a DLQ you can triage and a pile of JSON. Do it on every rung, not just the last.

What actually reaches the dead-letter lane

Two paths lead there, and they collect different things. The one you write is above: your ladder ran out. The one you inherit is automatic — a message delivered three times without an ack is moved by the queue itself, which catches the case your code never handles, like a worker that crashes on a specific payload shape.

We watched that second path fire, and the numbers behave:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/queue/stats/billing-callbacks" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '{available_count, in_flight_count, dlq_count}'

dlq_count is the alertable signal — it went from 0 to 3 in our run the moment the third delivery lapsed. What didn’t work is the obvious next call: GET /v1/queue/dlq/list/{queue} returned items: [] while that same counter said three. Treat DLQ listing as unreliable for now and read the dead-letter queue as an ordinary queue instead, by its own name:

curl -sS -X POST "https://api.infrai.cc/v1/queue/consume" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"billing-callbacks.dlq","max_messages":10}'

Every message came back with its original payload and a delivery_count reset to 1. That’s the path to build on.

Triage before you replay

Replaying a dead-letter queue without reading it is how one bad deploy becomes two. Group first:

import process from "node:process";

const KEY = process.env.INFRAI_API_KEY;
const DLQ = "billing-callbacks.dlq";

async function readDlq(limit = 10) {
  const res = await fetch("https://api.infrai.cc/v1/queue/consume", {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({ queue: DLQ, max_messages: limit }),
  });
  const out = await res.json();
  if (!out.ok) throw new Error(out.error.code);
  return out.data.items;
}

const buckets = new Map();
for (const message of await readDlq()) {
  const reason = message.payload.failure?.status ?? message.payload.last_failure?.status ?? "unknown";
  const host = new URL(message.payload.url).host;
  const bucket = buckets.get(`${host} ${reason}`) ?? { count: 0, sample: message.message_id };
  bucket.count += 1;
  buckets.set(`${host} ${reason}`, bucket);
}

for (const [key, value] of buckets) console.log(key.padEnd(48), value.count, value.sample);

In practice one subscriber and one status code explain almost everything in there. A wall of 410 Gone from a single host means that endpoint is dead and replaying it wastes an afternoon; a wall of 502 from a host that answers now is exactly what redrive is for.

Redrive, one message at a time

curl -sS -X POST "https://api.infrai.cc/v1/queue/dlq/redrive/billing-callbacks" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"message_id":"qmsg_SU6PYFXmCs1y5YszpDbikYfa"}'

{
  "ok": true,
  "data": {
    "queue": "billing-callbacks",
    "redriven": 1,
    "message_id": "qmsg_SU6PYFXmCs1y5YszpDbikYfa"
  }
}

The per-message form works. The catch is the bulk form — the same route with an empty body, or with max_messages — failed with 400 INVALID_ARGUMENT and a backend error string on our account, so a mass replay today means looping over message ids, or consuming the .dlq and republishing to the parent queue. Both are free apart from the republish, and the loop is honestly fine at the scale a DLQ usually reaches.

The security part nobody puts in the runbook

A dead-lettered webhook keeps its full payload for the queue’s retention window, 14 days by default. If those payloads carry card metadata, email addresses or anything else you’d rather not hold, the DLQ has quietly become a 14-day archive of your worst requests. Two mitigations, neither exotic: publish an identifier rather than the body and rehydrate at delivery time, or shorten retention with PATCH /v1/queue/update/{queue} when you create the lane. The blunt instrument is POST /v1/queue/purge/{queue}, which returns a count of what it dropped — it takes the whole queue, with no per-message selector, so use it on the DLQ and never on a live one.

Also: verify signatures on the way in, before the enqueue. A queue is a replay engine by design, and a forged event that got past your handler once will get past it three more times on redelivery.

What the failures cost

Only publishes are metered — $0.00002 each, verified 2026-07-26 — while consume, ack, nack, stats, purge and redrive are free and rate-limited. That pricing shape is what makes a five-rung jittered ladder reasonable: 4,000 stuck events retried five times is 20,000 publishes, and the redrive that finally lands them costs nothing. New accounts carry $2 of trial credit. Rates drift down over time, so check today’s:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '[.capabilities[] | select(.id | startswith("queue.")) | {id, price: .billing.price_usd}]'

Honest alternatives

SQS pairs a redrive policy with a redrive task that moves a whole dead-letter queue back in one API call, and if you’re inside AWS that maturity is worth more than anything written above. Hookdeck sells the delivery log and replay console this API doesn’t support. RabbitMQ, with dead-letter exchanges and per-message TTL, gives finer control if you’re already running a broker and don’t mind operating one.

Where this earns its place is the blast radius around the retry. The same key that republishes the callback also emails the account owner that their endpoint is disabled, writes the failing request to storage for the support thread, and files the exception — one bill, one usage query, no second integration to build at 2am.

References

• Infrai Message Queue API reference
• QUEUE_MESSAGE_NOT_FOUND error reference
• Amazon SQS dead-letter queues
• Hookdeck: dead-letter queues for webhook reliability