The webhook callback queue pattern: delay, retry, dead-letter

Where to keep retry state for failed outbound callbacks: a delayed republish ladder instead of nack, a three-strike DLQ, and the Node.js dispatcher that runs it.

A failed callback needs three things: somewhere to wait, an attempt counter that survives the wait, and an exit. The pattern that gives you all three is a queue where each retry is a new message published with a delay, the attempt number rides inside the payload, and the last failure lands in a dead-letter queue. On Infrai that’s POST /v1/queue/publish with delay_seconds, plus a worker loop you already know how to write.

The hard part isn’t the backoff arithmetic. It’s deciding which counter is authoritative — and most implementations quietly keep two.

Two counters, and only one of them is yours

Every message carries delivery_count, incremented each time a consumer receives it. Publishing to a name that doesn’t exist yet creates the queue with defaults, and those defaults include max_receive_count: 3, so the third delivery that ends without an ack is dead-lettered automatically. That counter belongs to the queue. It resets when a message is redriven, and in our testing PATCH /v1/queue/update/{queue} accepted a new max_receive_count and left the stored value at 3 — a limitation you should design around rather than fight.

Treat it as a circuit breaker, not a retry policy.

Your policy is the attempt integer you put in the payload yourself. It survives a republish, it can differ per subscriber (a partner with a flaky staging host gets a longer ladder than your own internal consumer), and it’s what your on-call dashboard groups by. The queue’s counter stops runaway loops; yours decides when to give up.

Here’s the message you publish, as a file so the shape is obvious:

{
  "queue": "webhook-out",
  "payload": {
    "url": "https://partner.example.com/hooks/infrai",
    "event": { "id": "evt_8812", "type": "invoice.paid", "amount_cents": 4900 },
    "attempt": 0
  },
  "delay_seconds": 0
}

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  --data @callback.json

delay_seconds runs from 0 to 604800 — seven days of ladder if you want one. Push past that ceiling and the API answers 400 INVALID_ARGUMENT; the message it returns talks about the queue already existing, which is misleading, so read the delay bound first when a publish starts failing.

The dispatcher

One worker, one pass, no framework:

import process from "node:process";

const BASE = "https://api.infrai.cc";
const QUEUE = "webhook-out";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

// 1m, 5m, 30m, 2h, 12h — five attempts, spread over about fifteen hours.
const LADDER = [60, 300, 1800, 7200, 43200];

async function call(path, payload) {
  const res = await fetch(`${BASE}${path}`, {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(payload),
  });
  const out = await res.json();
  if (!out.ok) throw new Error(`${path} → ${out.error.code}: ${out.error.message}`);
  return out.data;
}

async function deliver({ url, event }) {
  try {
    const res = await fetch(url, {
      method: "POST",
      headers: { "Content-Type": "application/json", "X-Event-Id": event.id },
      body: JSON.stringify(event),
      signal: AbortSignal.timeout(10_000),
    });
    return res.status >= 200 && res.status < 300;
  } catch (err) {
    console.warn(`${url} unreachable: ${err.message}`);
    return false;
  }
}

export async function pass() {
  const { items } = await call("/v1/queue/consume", { queue: QUEUE, max_messages: 10 });
  for (const msg of items) {
    const { url, event, attempt } = msg.payload;
    const delivered = await deliver({ url, event });

    if (!delivered && attempt < LADDER.length) {
      await call("/v1/queue/publish", {
        queue: QUEUE,
        payload: { url, event, attempt: attempt + 1 },
        delay_seconds: LADDER[attempt],
      });
    } else if (!delivered) {
      await call("/v1/queue/publish", {
        queue: `${QUEUE}.dlq`,
        payload: { url, event, attempts: attempt, reason: "ladder exhausted" },
      });
    }

    await call("/v1/queue/ack", { queue: QUEUE, message_id: msg.message_id });
  }
  return items.length;
}

Note what happens on every path: the original message is acked and the retry exists as a separate, delayed message. That’s deliberate. Holding a message for two hours while you sleep is a message that gets redelivered to a second worker the moment its visibility timeout expires (300 seconds by default), and the ack that eventually follows comes back as QUEUE_MESSAGE_NOT_IN_FLIGHT.

A consume returns at most 10 messages — the API rejects max_messages: 11 outright — so a pass is bounded and you loop.

What comes back

{
  "ok": true,
  "data": {
    "items": [
      {
        "message_id": "qmsg_bJaMmFE86ddHMBmNe51YQ2HF",
        "queue": "webhook-out",
        "payload": { "url": "https://partner.example.com/hooks/infrai", "attempt": 2 },
        "status": "in_flight",
        "delivery_count": 1,
        "published_at": "2026-07-26T00:30:06.714543Z"
      }
    ],
    "next_cursor": null
  }
}

The dead-letter queue is an ordinary queue named after its parent, so nothing new is needed to drain it:

curl -sS -X POST "https://api.infrai.cc/v1/queue/consume" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"webhook-out.dlq","max_messages":10}'

And the number your alert should watch lives in stats:

curl -sS "https://api.infrai.cc/v1/queue/stats/webhook-out" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

dlq_count moving is nearly always one cause with many victims — a rotated signing secret, an expired certificate, a partner who changed hostname without telling anyone.

Which pattern for which failure

What the ladder costs

Publishing is the only metered call in this design: $0.00002 per message, verified 2026-07-26. Consume, ack, nack, stats and dead-letter reads are free and rate-limited rather than billed, which is why a five-rung ladder costs five publishes and a redelivery costs nothing at all. A subscriber that goes dark for a day, taking 4,000 events with it and retrying each five times, is 20,000 publishes. New accounts start with $2 of credit, roughly 99,999 publishes.

Rates drift downward and discount campaigns run, so read today’s number rather than trusting this paragraph:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '[.capabilities[] | select(.id | startswith("queue.")) | {id, billable: .billing.is_billable, price: .billing.price_usd}]'

Where this isn’t the right tool

If delayed outbound HTTP is the entire problem — no consumers, no fan-out, no other backend work — QStash is built for exactly that and you’d be better off there. SQS with a redrive policy is the obvious answer inside an AWS account that already has one. Temporal is what you want when a failed callback has to unwind three other steps, because a queue has no idea what a compensation is.

The argument for doing it on Infrai is what happens after the retry. The same key that publishes the message also sends the “your endpoint is disabled” email, stores the request/response pair for the support ticket, files the error, and attributes all of it to one tenant on one bill. That’s the second question, and it’s already answered on the account you have.

References

• Infrai Message Queue API reference
• QUEUE_MESSAGE_NOT_IN_FLIGHT error reference
• BullMQ: retrying failing jobs
• Hookdeck: dead-letter queues for webhook reliability