One throttled customer shouldn't stall your whole webhook queue

A multi-tenant sender that keeps a per-destination cooldown, honours Retry-After, and defers throttled events to a slow-lane queue instead of sleeping the worker.

A 429 from one customer’s endpoint says nothing about the other four hundred. So the fix isn’t await sleep(retryAfter) in the consume loop — that punishes every destination for one destination’s limit. Keep a per-host cooldown map, skip messages bound for a host that’s cooling, and move the deferred ones to a second Infrai queue that a slower worker drains. Consuming, acking and nacking are free routes, so the extra bookkeeping doesn’t show up on the bill.

The pattern below runs one process against a outbound-webhooks lane and a outbound-webhooks-retry lane, and it degrades per-tenant rather than globally.

Three ways to wait, and what each costs you

Most implementations only ever use the first row.

The second row is free and automatic, and it’s the right answer when Retry-After is small. Past a couple of minutes the third row wins, because a message that keeps coming back every five minutes will exhaust max_receive_count — three deliveries by default — and dead-letter itself while the vendor is still perfectly willing to accept it in an hour.

Reading the instruction the endpoint gave you

A throttled response usually carries the answer in a header, sometimes in the body, and occasionally not at all.

{
  "error": "rate_limited",
  "message": "too many requests for this endpoint",
  "retry_after_seconds": 120
}

Treat a missing Retry-After as 60 seconds rather than retrying immediately — and cap whatever the endpoint claims, because a hostile or broken destination will happily tell you to come back in a week.

The dispatcher

import process from "node:process";
import { setTimeout as idle } from "node:timers/promises";

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const MAIN = "outbound-webhooks";
const SLOW = "outbound-webhooks-retry";
const MAX_DEFER_SECONDS = 3600;

const auth = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };
const cooldown = new Map(); // hostname -> epoch ms it becomes usable again

async function queueCall(route, payload) {
  const r = await fetch(`https://api.infrai.cc${route}`, { method: "POST", headers: auth, body: JSON.stringify(payload) });
  const j = await r.json();
  if (j.ok !== true) throw new Error(`${route}: ${j.error.code} ${j.error.message}`);
  return j.data;
}

function coolFor(response) {
  const header = response.headers.get("retry-after");
  if (!header) return 60;
  const asSeconds = Number(header);
  const seconds = Number.isFinite(asSeconds) ? asSeconds : (Date.parse(header) - Date.now()) / 1000;
  return Math.min(MAX_DEFER_SECONDS, Math.max(1, Math.round(seconds)));
}

for (;;) {
  const { items } = await queueCall("/v1/queue/consume", { queue: MAIN, max_messages: 10 });
  if (items.length === 0) { await idle(3000); continue; }

  for (const message of items) {
    const event = message.payload;
    const host = new URL(event.url).hostname;
    const cooling = cooldown.get(host) ?? 0;

    if (cooling > Date.now()) {
      const waitSeconds = Math.ceil((cooling - Date.now()) / 1000);
      await queueCall("/v1/queue/publish", {
        queue: SLOW,
        body: { ...event, not_before: new Date(cooling).toISOString(), attempt: (event.attempt ?? 0) + 1 },
      });
      await queueCall("/v1/queue/ack", { queue: MAIN, receipt_handle: message.message_id });
      console.log(`deferred ${host} for ${waitSeconds}s`);
      continue;
    }

    let response;
    try {
      response = await fetch(event.url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(event.body) });
    } catch (e) {
      console.error(`${host} unreachable: ${e.message}`);
      continue; // no ack: the queue redelivers after the visibility window
    }

    if (response.status === 429) {
      const seconds = coolFor(response);
      cooldown.set(host, Date.now() + seconds * 1000);
      await queueCall("/v1/queue/publish", {
        queue: SLOW,
        body: { ...event, not_before: new Date(Date.now() + seconds * 1000).toISOString(), attempt: (event.attempt ?? 0) + 1 },
      });
      await queueCall("/v1/queue/ack", { queue: MAIN, receipt_handle: message.message_id });
      console.warn(`${host} throttled; cooling ${seconds}s`);
      continue;
    }

    if (response.ok) {
      await queueCall("/v1/queue/ack", { queue: MAIN, receipt_handle: message.message_id });
    } else {
      console.error(`${host} answered ${response.status}; leaving for redelivery`);
    }
  }
}

The cooldown map is in memory, which is fine — it’s an optimisation, not a source of truth. Lose it on restart and the first message to that host discovers the limit again.

The slow lane drains itself

The deferred worker uses the visibility timeout as its clock. A message whose not_before hasn’t arrived is simply left unacked, and it comes back around later.

import process from "node:process";
import { setTimeout as idle } from "node:timers/promises";

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");
const auth = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

async function queueCall(route, payload) {
  const r = await fetch(`https://api.infrai.cc${route}`, { method: "POST", headers: auth, body: JSON.stringify(payload) });
  const j = await r.json();
  if (j.ok !== true) throw new Error(`${route}: ${j.error.code} ${j.error.message}`);
  return j.data;
}

for (;;) {
  const { items } = await queueCall("/v1/queue/consume", { queue: "outbound-webhooks-retry", max_messages: 10 });
  if (items.length === 0) { await idle(15_000); continue; }

  for (const message of items) {
    const event = message.payload;
    if (Date.parse(event.not_before) > Date.now()) continue; // not yet; let the lease lapse

    if ((event.attempt ?? 0) >= 6) {
      console.error(`giving up on ${event.url} after ${event.attempt} attempts`);
      await queueCall("/v1/queue/ack", { queue: "outbound-webhooks-retry", receipt_handle: message.message_id });
      continue;
    }

    await queueCall("/v1/queue/publish", { queue: "outbound-webhooks", body: event });
    await queueCall("/v1/queue/ack", { queue: "outbound-webhooks-retry", receipt_handle: message.message_id });
  }
}

Deferring by hand like this exists for a reason: setting delivery_delay_seconds on a queue had no effect in our testing on 2026-07-26, and published messages were immediately available. Until that changes, the slow lane is the mechanism.

Push one in by hand to check the wiring:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"outbound-webhooks-retry","body":{"url":"https://hooks.example.com/acme","not_before":"2026-07-26T02:00:00Z","attempt":1,"body":{"type":"invoice.paid"}}}'

Watching both lanes

curl -sS "https://api.infrai.cc/v1/queue/stats/outbound-webhooks-retry" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "queue": "outbound-webhooks-retry",
    "message_count": 3,
    "available_count": 3,
    "in_flight_count": 0,
    "delayed_count": 0,
    "dlq_count": 0,
    "oldest_message_age_seconds": 128
  }
}

A slow lane that grows and never drains means one destination is permanently throttling you — that’s a support conversation, not an engineering fix.

What a throttled hour costs

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id == "queue.publish" or .id == "queue.consume") | {id, price_usd: .billing.price_usd, free: .billing.free}'

Each deferral is a republish, and publishing is the only metered call at $0.00002 each, verified 2026-07-26. A hundred thousand events with a 5% deferral rate is $2.10 rather than $2.00 — the retries are noise against the base volume, which is the point. Consume, ack and nack are free and rate-limited. New accounts get $2 of free credit. These rates trend downward, so read them live instead of from this page.

Limits worth knowing, and the alternatives

Two limitations to weigh for a US/EU SaaS. The queue routes are served from western and China regions, so if you need EU-only processing of webhook payloads for a compliance story, keep identifiers in the message and fetch the body inside your own region. And there’s no server-side rate limiter — every line of pacing above is yours to maintain, which is a real trade-off against a product that ships one.

If you’d rather not maintain it, qstash offers per-destination rate limiting as a product feature and is the better buy when outbound webhooks are the only queue you have. bullmq, if you’re already running Redis, gives you real delayed jobs and rate-limiter groups out of the box. What this stack buys instead is everything around the send sitting on one credential — the parked-message triage described in the dead-letter backlog guide, the alert email, the per-tenant cost attribution.

References

• Infrai queue API reference
• QUEUE_DELAY_INVALID error reference
• Hookdeck: webhook retry best practices
• Zapier: Webhooks by Zapier rate limits