Exponential backoff for failed jobs when redelivery is a flat retry budget

Build a real backoff ladder on a queue with fixed redelivery: republish with a per-message delay, carry the attempt count, and redrive from the DLQ when it's fixed.

Infrai’s queue redelivers an unacked message up to max_retries times — three by default — and then dead-letters it. Those redeliveries are spaced by the visibility timeout, so a queue with the default lease covers a window measured in seconds: fine for a transient socket error, useless for an image CDN that’s been down since lunchtime. If you want minutes and hours of backoff, you build the ladder yourself by acking the failed message and republishing it with a delay.

That sounds like more work than it is: it’s one function and an integer in the payload.

What redelivery gives you, and what it doesn’t

The visibility timeout is the only backoff the queue does on its own. Don’t ack, wait for the lease to lapse, get the message again. The interval is the timeout, not a curve. The attempt budget is adjustable — send max_retries on POST /v1/queue/create or PATCH /v1/queue/update/{queue} and it comes back in the queue record as max_receive_count — but a flat budget of N identical waits still isn’t a curve. Worth knowing which name goes in which direction, because the request field and the response field are spelled differently.

So there are three honest strategies, and they’re not exclusive.

The middle row is the one that actually implements exponential backoff, and it’s the one this piece is about.

Per-message delay is the primitive

POST /v1/queue/publish accepts an optional delay in seconds alongside the queue and the payload. The message is accepted, counted, and simply not handed to any consumer until the delay expires.

export INFRAI_API_KEY="your_infrai_api_key"

PAYLOAD='{"queue":"render-jobs","payload":{"job":"thumbnail","asset_id":"as_9142","attempt":2},"delay_seconds":120}'

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "$PAYLOAD"

The publish response carries available_at, which is when the message becomes visible — that’s the field to assert on in a test. The queue’s own counters are the other half of the picture:

curl -sS "https://api.infrai.cc/v1/queue/stats/render-jobs" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Consume immediately after that publish and you get an empty items array, while the message sits under delayed_count:

{
  "ok": true,
  "data": {
    "queue": "render-jobs",
    "message_count": 0,
    "available_count": 0,
    "in_flight_count": 0,
    "delayed_count": 1,
    "dlq_count": 0,
    "oldest_message_age_seconds": 0
  }
}

An out-of-range value comes back as QUEUE_DELAY_INVALID with retryable: false, so clamp your computed delay rather than letting a runaway attempt counter generate a nonsense number — the ceiling is 604800 seconds. There’s a queue-wide default too, delivery_delay_seconds on the update route, which persists and applies to publishes that don’t set their own. A backoff ladder wants the per-message field, since every rung asks for a different number.

The ladder

Four rungs with jitter covers most third-party outages: half a minute, two minutes, ten minutes, an hour. After that the job has earned a look from a person.

import process from "node:process";
import { setTimeout as sleep } from "node:timers/promises";

const BASE = "https://api.infrai.cc";
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is not set");
const headers = { Authorization: `Bearer ${key}`, "Content-Type": "application/json" };
const QUEUE = "render-jobs";
const LADDER = [30, 120, 600, 3600];

async function call(path, payload) {
  const res = await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(payload) });
  const out = await res.json();
  if (out.ok === false) throw new Error(`${path}: ${out.error.code} — ${out.error.message}`);
  return out.data;
}

async function republish(job, delaySeconds) {
  const message = { queue: QUEUE, payload: job };
  if (delaySeconds > 0) message.delay_seconds = delaySeconds;
  const res = await fetch(`${BASE}/v1/queue/publish`, {
    method: "POST",
    headers,
    body: JSON.stringify(message),
  });
  const out = await res.json();
  if (out.ok === false) throw new Error(`republish: ${out.error.code}`);
  return out.data.message_id;
}

const RESIZER = process.env.RESIZER_BASE_URL;
if (!RESIZER) throw new Error("RESIZER_BASE_URL is not set");

async function process_image(job) {
  const res = await fetch(`${RESIZER}/resize/${job.asset_id}`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ width: job.width ?? 640 }),
    signal: AbortSignal.timeout(15000),
  });
  if (res.status >= 400 && res.status < 500) throw Object.assign(new Error(`permanent ${res.status}`), { permanent: true });
  if (!res.ok) throw new Error(`transient ${res.status}`);
}

async function onFailure(job, err) {
  if (err.permanent) {
    console.error(`asset ${job.asset_id}: ${err.message} — not retrying`);
    return;
  }
  const attempt = job.attempt ?? 1;
  if (attempt > LADDER.length) {
    console.error(`asset ${job.asset_id} exhausted ${LADDER.length} retries; leaving it to dead-letter`);
    throw err;
  }
  const base = LADDER[attempt - 1];
  const delay = Math.round(base * (0.75 + Math.random() * 0.5));
  const id = await republish({ ...job, attempt: attempt + 1 }, delay);
  console.warn(`asset ${job.asset_id} retry ${attempt + 1} scheduled in ${delay}s as ${id}`);
}

let running = true;
process.on("SIGTERM", () => { running = false; });

while (running) {
  const { items } = await call("/v1/queue/consume", { queue: QUEUE, max_messages: 10 });
  if (items.length === 0) { await sleep(2000); continue; }
  for (const msg of items) {
    try {
      await process_image(msg.payload);
      await call("/v1/queue/ack", { queue: QUEUE, message_id: msg.message_id });
    } catch (err) {
      try {
        await onFailure(msg.payload, err);
        await call("/v1/queue/ack", { queue: QUEUE, message_id: msg.message_id });
      } catch {
        console.warn(`leaving ${msg.message_id} unacked (delivery ${msg.delivery_count}) so it dead-letters`);
      }
    }
  }
}

Read the control flow carefully, because the acks are the whole trick. A successful job acks. A permanently broken job acks — a 422 from the resize service won’t get better on the ninth attempt. A retryable job also acks, but only after its replacement has been published with a delay, so the work is never in two places or in none. And a job past the last rung deliberately isn’t acked, which hands it back to the built-in redelivery path and lets it dead-letter naturally.

The jitter is a factor between 0.75 and 1.25. Without it, a thousand jobs that failed together retry together, and you re-DDoS the service the moment it comes back up.

Attempt counts live in the payload

delivery_count from consume counts deliveries of one particular message. Your ladder creates a new message each rung, so delivery_count resets to zero and only attempt in the payload knows the real history. Redrive resets it too. Keep both in your logs — one tells you about the queue’s behaviour, the other about the job’s.

When the ladder runs out

An hour into an outage, the job dead-letters. Inspect the parked messages without consuming them:

curl -sS "https://api.infrai.cc/v1/queue/dlq/list/render-jobs?limit=10" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

The count that route returns agrees with dlq_count from stats, so those two are safe to alert on together. Once the downstream is healthy, send the whole lot back in one call — an empty body redrives the queue’s dead letters and tells you how many moved:

curl -sS -X POST "https://api.infrai.cc/v1/queue/dlq/redrive/render-jobs" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "ok": true,
  "data": { "queue": "render-jobs", "redriven": 14 }
}

Pass a message_id instead if you want to move one poison message back after fixing it by hand.

What the ladder costs

Each rung is a publish, and publish is the only billable call here — $0.00002 per message on today’s reading. A four-rung ladder on a job that never succeeds costs five publishes; consume, ack, nack, stats, DLQ reads and redrive are all free and rate-limited. Rather than multiplying that unit price out on a page that can go stale, ask the account:

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

That’s your actual spend rather than a price list. Read it live before you model anything.

Where a purpose-built retry engine wins

SQS has this as configuration rather than code: a redrive policy on the queue, a maxReceiveCount that ops owns, and a console flow that moves batches back. Buy that if your retry policy genuinely belongs in infrastructure-as-code rather than in the worker. Temporal goes further and models the whole workflow, retries included, with durable state — the right answer for multi-step jobs where “retry step three” means something. BullMQ ships backoff strategies as a first-class option if you’re already running Redis and don’t mind operating it.

The drawback of the approach here is plain: your backoff policy is code in the worker rather than config on the queue, and a worker deployed with the wrong ladder is a bug, not a setting.

What you get back is that the ladder is portable. The whole design is plain REST over one credential — publish with an integer, consume, ack, dlq/redrive — with no SDK to install, no broker to operate and no proprietary retry DSL encoding your policy. There’s no lock-in in LADDER = [30, 120, 600, 3600]; it’s four numbers in a file you own, and the same worker moves to any queue that accepts a per-message delay. The policy can also depend on the payload, which config-on-the-queue can’t do: a paying tenant’s job climbs a longer ladder than a free one, decided at runtime.

References

• Infrai queue API reference
• QUEUE_DELAY_INVALID error reference
• Retrying failing jobs — BullMQ