A dead-letter redrive runbook for small Node.js SaaS teams

How to find dead background jobs, replay them one at a time, and confirm they cleared — plus the two redrive paths that are broken today and what to use instead.

Redrive is the boring half of a queue and the half you’ll actually be woken up for. A partner’s certificate expires, four hundred jobs exhaust their attempts, the certificate gets fixed, and now someone has to put those four hundred jobs back. On Infrai that recovery is three calls — read the count, read the dead messages, replay them by id — and none of the three costs anything. What follows is the runbook, including the two paths that don’t work yet, because a runbook that omits those is worse than none.

For a team of five running a handful of background jobs, this is about as small as the operation gets.

Step one: know the count before you know the cause

Every queue gets a companion dead-letter queue. By default it’s named after the parent with a .dlq suffix; pass dlq at creation to choose the name yourself:

curl -sS -X POST "https://api.infrai.cc/v1/queue/create" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name":"billing-jobs","type":"standard","dlq":"billing-jobs-dead"}'

The count you page on lives in the stats route:

curl -sS "https://api.infrai.cc/v1/queue/stats/billing-jobs" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "queue": "billing-jobs",
    "message_count": 1,
    "available_count": 0,
    "in_flight_count": 1,
    "delayed_count": 0,
    "dlq_count": 12,
    "oldest_message_age_seconds": 40
  }
}

A non-zero dlq_count with a flat message_count is the signature of a systemic failure — one broken dependency, many identical victims. A slowly-climbing dlq_count while throughput stays healthy is the other story entirely: a few malformed payloads that will never succeed and shouldn’t be replayed.

Step two: read the dead messages — by queue name, not the list route

Here’s the trap. GET /v1/queue/dlq/list/{queue} returns an empty array right now even when stats report a dozen dead messages; we reproduced it with dlq_count at 1 and at 2, on freshly created queues. Until that’s fixed, consume the dead-letter queue by its own name — it’s a normal queue and consuming it works exactly as you’d expect:

curl -sS -X POST "https://api.infrai.cc/v1/queue/consume" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"billing-jobs-dead","max_messages":10}'

{
  "ok": true,
  "data": {
    "items": [
      {
        "message_id": "qmsg_IsHFbUwrn2UT8EW60cfXinqy",
        "queue": "billing-jobs-dead",
        "payload": { "invoice_id": "inv_204", "action": "charge" },
        "status": "in_flight",
        "delivery_count": 1
      }
    ],
    "next_cursor": null
  }
}

Reading them is the point of the exercise. Half the value of a dead-letter queue is triage: you look at ten payloads and learn whether you’re replaying an outage or deleting a bug.

Step three: replay, one message at a time

Redrive takes the parent queue’s name in the path and the dead message’s id in the body:

curl -sS -X POST "https://api.infrai.cc/v1/queue/dlq/redrive/billing-jobs" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"message_id":"qmsg_IsHFbUwrn2UT8EW60cfXinqy"}'

The response confirms one message moved back, and the replayed message arrives with a fresh delivery budget rather than the exhausted one it died with. The bulk form — posting an empty body to drain everything — currently fails with an internal backend error, so per-message is the path that works. That’s an inconvenience, not a blocker: loop it.

import process from "node:process";

const BASE = "https://api.infrai.cc";
const PARENT = "billing-jobs";
const DEAD = "billing-jobs-dead";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");
const H = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

async function call(path, payload) {
  const res = await fetch(`${BASE}${path}`, { method: "POST", headers: H, body: JSON.stringify(payload) });
  const out = await res.json();
  if (!out.ok) throw new Error(`${path}: ${out.error.code} — ${out.error.message}`);
  return out.data;
}

// Replay everything whose payload passes a sanity check; leave the rest to a human.
export async function redriveAll({ accept = () => true, limit = 100 } = {}) {
  let moved = 0;
  let skipped = 0;
  while (moved + skipped < limit) {
    const { items } = await call("/v1/queue/consume", { queue: DEAD, max_messages: 10 });
    if (!items.length) break;
    for (const msg of items) {
      if (!accept(msg.payload)) {
        skipped++;
        console.warn(`skipping ${msg.message_id}: ${JSON.stringify(msg.payload).slice(0, 120)}`);
        continue;
      }
      await call(`/v1/queue/dlq/redrive/${PARENT}`, { message_id: msg.message_id });
      moved++;
    }
  }
  return { moved, skipped };
}

const result = await redriveAll({ accept: (p) => typeof p?.invoice_id === "string" });
console.log(`redriven ${result.moved}, left alone ${result.skipped}`);

Run that behind a scheduled job once an hour and you have an automatic redrive service; run it by hand after an incident and you have a runbook. The accept predicate is doing real work — replaying a payload that failed because it’s malformed just burns three more deliveries and lands it back where it started.

Then confirm with the same stats call you started from. dlq_count should fall to zero, available_count should rise by what you moved.

What’s actually different across the usual options

If your jobs already run on Sidekiq or BullMQ, their dead sets and dashboards are more mature than what’s described here, and switching queue vendors purely for redrive would be a poor trade-off. SQS remains the answer when you need bulk redrive as a first-class operation today.

Limits worth knowing

The delivery budget is fixed at three attempts. We set max_receive_count to 1 at creation and to 5 through PATCH /v1/queue/update/{queue}, and the queue reported 3 both times — dead-lettering happened on the third delivery regardless. If your policy needs ten attempts before giving up, implement the extra attempts in your own handler rather than expecting the queue to count them.

The API also doesn’t support a per-queue region control: queues serve western and China regions, but you can’t pin one to an EU-only data path, so a hard data-residency requirement isn’t something we’d claim to satisfy today.

Redrive itself is free, as are consume, stats and dead-letter reads. Publish is the metered call, at $0.00002 per message (verified 2026-07-26), with $2 free credit on a new account — and rates on this platform have moved downward over time, so check the live figure:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id | startswith("queue.dlq")) | {id, free: .billing.free}'

References

• Infrai queue API reference
• QUEUE_MESSAGE_NOT_FOUND error
• Amazon SQS dead-letter queues
• Svix: DLQ redrive