Exponential backoff for failed jobs when your queue only retries three times

Build a real backoff ladder on a queue with fixed redelivery: republish with a per-message delay, carry the attempt count, and redrive from the DLQ when it's fixed.

Infrai’s queue redelivers an unacked message three times and then dead-letters it. We measured those attempts landing about six seconds apart on 2026-07-26, which covers roughly an eighteen-second window — fine for a transient socket error, useless for an image CDN that’s been down since lunchtime. If you want minutes and hours of backoff, you build the ladder yourself by acking the failed message and republishing it with a delay.

That sounds like more work than it is: it’s one function and an integer in the payload.

What redelivery gives you, and what it doesn’t

The visibility timeout is the only backoff the queue does on its own. Don’t ack, wait for the lease to lapse, get the message again. The interval is the timeout, not a curve, and the attempt budget is three regardless of what you set max_receive_count to — changing that field didn’t change the behaviour in our testing.

So there are three honest strategies, and they’re not exclusive.

The middle row is the one that actually implements exponential backoff, and it’s the one this piece is about.

Per-message delay is the primitive

POST /v1/queue/publish accepts an optional delay in seconds alongside the queue and the payload. The message is accepted, counted, and simply not handed to any consumer until the delay expires.

export INFRAI_API_KEY="your_infrai_api_key"

PAYLOAD='{"queue":"render-jobs","body":{"job":"thumbnail","asset_id":"as_9142","attempt":2},"delay_seconds":120}'

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "$PAYLOAD"

There’s a wrinkle you need to know about before you trust it. The publish response reports "status": "available" even for a message that’s been held back — the echo is misleading. What tells the truth is the queue’s own counters:

curl -sS "https://api.infrai.cc/v1/queue/stats/render-jobs" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Consume immediately after that publish and you get an empty items array, while the message sits under delayed_count:

{
  "ok": true,
  "data": {
    "queue": "render-jobs",
    "message_count": 0,
    "available_count": 0,
    "in_flight_count": 0,
    "delayed_count": 1,
    "dlq_count": 0,
    "oldest_message_age_seconds": 0
  }
}

An out-of-range value comes back as QUEUE_DELAY_INVALID, so clamp your computed delay rather than letting a runaway attempt counter generate a nonsense number. Setting a queue-wide delivery delay through the update route was silently ignored when we tried it, so keep the delay per message.

The ladder

Four rungs with jitter covers most third-party outages: half a minute, two minutes, ten minutes, an hour. After that the job has earned a look from a person.

import process from "node:process";
import { setTimeout as sleep } from "node:timers/promises";

const BASE = "https://api.infrai.cc";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is not set");
const headers = { Authorization: `Bearer ${key}`, "Content-Type": "application/json" };
const QUEUE = "render-jobs";
const LADDER = [30, 120, 600, 3600];

async function call(path, payload) {
  const res = await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(payload) });
  const out = await res.json();
  if (out.ok === false) throw new Error(`${path}: ${out.error.code} — ${out.error.message}`);
  return out.data;
}

async function republish(job, delaySeconds) {
  const message = { queue: QUEUE, body: job };
  if (delaySeconds > 0) message.delay_seconds = delaySeconds;
  const res = await fetch(`${BASE}/v1/queue/publish`, {
    method: "POST",
    headers,
    body: JSON.stringify(message),
  });
  const out = await res.json();
  if (out.ok === false) throw new Error(`republish: ${out.error.code}`);
  return out.data.message_id;
}

async function process_image(job) {
  const res = await fetch(`https://images.example.com/resize/${job.asset_id}`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ width: job.width ?? 640 }),
    signal: AbortSignal.timeout(15000),
  });
  if (res.status >= 400 && res.status < 500) throw Object.assign(new Error(`permanent ${res.status}`), { permanent: true });
  if (!res.ok) throw new Error(`transient ${res.status}`);
}

async function onFailure(job, err) {
  if (err.permanent) {
    console.error(`asset ${job.asset_id}: ${err.message} — not retrying`);
    return;
  }
  const attempt = job.attempt ?? 1;
  if (attempt > LADDER.length) {
    console.error(`asset ${job.asset_id} exhausted ${LADDER.length} retries; leaving it to dead-letter`);
    throw err;
  }
  const base = LADDER[attempt - 1];
  const delay = Math.round(base * (0.75 + Math.random() * 0.5));
  const id = await republish({ ...job, attempt: attempt + 1 }, delay);
  console.warn(`asset ${job.asset_id} retry ${attempt + 1} scheduled in ${delay}s as ${id}`);
}

let running = true;
process.on("SIGTERM", () => { running = false; });

while (running) {
  const { items } = await call("/v1/queue/consume", { queue: QUEUE, max_messages: 10 });
  if (items.length === 0) { await sleep(2000); continue; }
  for (const msg of items) {
    try {
      await process_image(msg.payload);
      await call("/v1/queue/ack", { queue: QUEUE, receipt_handle: msg.message_id });
    } catch (err) {
      try {
        await onFailure(msg.payload, err);
        await call("/v1/queue/ack", { queue: QUEUE, receipt_handle: msg.message_id });
      } catch {
        console.warn(`leaving ${msg.message_id} unacked (delivery ${msg.delivery_count}) so it dead-letters`);
      }
    }
  }
}

Read the control flow carefully, because the acks are the whole trick. A successful job acks. A permanently broken job acks — a 422 from the resize service won’t get better on the ninth attempt. A retryable job also acks, but only after its replacement has been published with a delay, so the work is never in two places or in none. And a job past the last rung deliberately isn’t acked, which hands it back to the built-in redelivery path and lets it dead-letter naturally.

The jitter is a factor between 0.75 and 1.25. Without it, a thousand jobs that failed together retry together, and you re-DDoS the service the moment it comes back up.

Attempt counts live in the payload

delivery_count from consume counts deliveries of one particular message. Your ladder creates a new message each rung, so delivery_count resets to zero and only attempt in the payload knows the real history. Redrive resets it too. Keep both in your logs — one tells you about the queue’s behaviour, the other about the job’s.

When the ladder runs out

An hour into an outage, the job dead-letters into the companion queue. Read it as an ordinary queue:

curl -sS -X POST "https://api.infrai.cc/v1/queue/consume" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"render-jobs.dlq","max_messages":10}'

The documented DLQ listing route came back empty for us even with messages demonstrably dead-lettered, so consume by name is the reliable read. Once the downstream is healthy, push each message back individually — bulk redrive with an empty body didn’t work in our testing:

curl -sS -X POST "https://api.infrai.cc/v1/queue/dlq/redrive/render-jobs" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"message_id":"qmsg_PdBdedHTNrlbrJpiwXlbFxde"}'

What the ladder costs

Each rung is a publish, and publish is the only billable call here at $0.00002 per message, verified 2026-07-26. A four-rung ladder on a job that never succeeds costs five publishes — a tenth of a cent per hundred jobs. Consume, ack, nack, stats, DLQ reads and redrive are all free and rate-limited. New accounts get $2 of free credit, which is roughly 99,999 publishes.

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

That’s your actual spend rather than a price list, and prices themselves trend downward with discount periods running, so read live before you model it.

Where a purpose-built retry engine wins

SQS has this as configuration rather than code: a redrive policy on the queue, a maxReceiveCount you control, and a console-driven DLQ redrive that moves batches back. If your retry policy needs to be an infrastructure setting that ops owns, that’s the better fit. Temporal goes further and models the whole workflow, retries included, with durable state — the right answer for multi-step jobs where “retry step three” means something. BullMQ ships backoff strategies as a first-class option in its retry guide if you’re already running Redis.

The drawback of the approach here is plain: your backoff policy is code in the worker rather than config on the queue, and a worker deployed with the wrong ladder is a bug, not a setting. In exchange you get a policy that can depend on the payload — a paying tenant’s job can climb a longer ladder than a free one — and no broker to operate.

References

• Infrai queue API reference
• QUEUE_DELAY_INVALID error reference
• Configure a dead-letter queue redrive in Amazon SQS
• Retrying failing jobs — BullMQ