Job keeps coming back? Three causes of endless queue retries

Diagnosing a poison message in a Node.js worker: unacked leases, nack with no backoff, and a republish loop of your own making — with the checks that tell them apart.

The same job id shows up in your logs every few minutes, forever, and the dead-letter queue stays empty. Three different bugs produce that symptom, and they need three different fixes — one is a missing ack, one is a misunderstanding of what a nack does, and one is a loop your own code is driving. On Infrai you can tell them apart in about a minute using delivery_count and the queue stats route, both free to call.

Start by naming the thing you’re looking at, because “retry” covers two mechanisms that behave nothing alike.

Cause 1: the message isn’t retrying, it’s expiring

When a worker consumes a message, the message goes invisible for the visibility timeout — 300 seconds by default. It is not deleted. If your handler throws, or the pod restarts, or the process exits before the ack, that timer runs out and the message becomes available again. From the log’s point of view it looks like a retry; from the queue’s point of view nobody ever finished the job.

The tell is delivery_count, which increments on every hand-out:

{
  "ok": true,
  "data": {
    "items": [
      {
        "message_id": "qmsg_v6tmquIUXNYQfbNzy2rzsX8b",
        "queue": "image-thumbnails",
        "payload": { "upload_id": "up_77", "width": 512 },
        "status": "in_flight",
        "delivery_count": 3
      }
    ],
    "next_cursor": null
  }
}

Same id, climbing count, empty dead-letter queue: your handler is dying before it acks.

There’s a nastier variant. Acking an id the queue has never seen returns HTTP 200 with "acked": false — no error, no exception, nothing that a res.ok check would catch. A worker that acks the wrong field (or a stale id) looks completely healthy while every message it touches comes back:

import process from "node:process";

const BASE = "https://api.infrai.cc";
const QUEUE = "image-thumbnails";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");
const H = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

export async function ackOrThrow(messageId) {
  const res = await fetch(`${BASE}/v1/queue/ack`, {
    method: "POST",
    headers: H,
    body: JSON.stringify({ queue: QUEUE, receipt_handle: messageId }),
  });
  const out = await res.json();
  if (!out.ok) throw new Error(`ack: ${out.error.code} ${out.error.message}`);
  if (out.data.acked !== true) throw new Error(`ack silently failed for ${messageId} — wrong id or expired lease`);
  return true;
}

await ackOrThrow(process.argv[2]);

Cause 2: nack is not backoff

A nack hands the message straight back. In our testing the very next consume returns it, with the count incremented and no pause at all. That’s correct for “this worker is going down, someone else take it” and wrong for “the downstream API is throwing 503s”, because a tight worker loop will burn the entire attempt budget in under a second and dead-letter the job before the dependency has finished restarting.

If you’re nacking on every failure inside a fast poll loop, your retries aren’t broken. They’re just too fast to be useful.

The fix is to make the worker wait rather than asking the queue to wait: sleep between polls, or re-publish the job as a new message once you’ve decided it deserves another chance later.

Cause 3: your code is the loop

This one hides in plain sight. On failure, the handler publishes the job back to the queue — no attempt counter in the payload, no ceiling — and every republish is a brand-new message with a brand-new id and a fresh delivery budget. The queue’s own poison protection never triggers, because from its side nothing ever fails three times. Meanwhile message_count stays flat and your publish bill grows.

The check takes one call:

curl -sS "https://api.infrai.cc/v1/queue/stats/image-thumbnails" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

The attempt budget is three, and today you can’t change it

Messages dead-letter on the third delivery. We tried setting max_receive_count to 1 at creation and to 5 through PATCH /v1/queue/update/{queue}; the queue reported 3 in both cases and behaved accordingly. Treat three as fixed and put any additional attempt policy in your own handler — that’s a real limitation compared with BullMQ, where attempts and backoff are per-job options you control.

Here’s the worker shape that stops a poison message cleanly. It decides before doing any work whether the payload can ever succeed, acks the hopeless ones so they leave the queue with a record, and lets genuine failures ride the budget into the dead-letter queue:

import process from "node:process";

const BASE = "https://api.infrai.cc";
const QUEUE = "image-thumbnails";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");
const H = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

const valid = (p) => typeof p?.upload_id === "string" && Number.isInteger(p?.width);

const res = await fetch(`${BASE}/v1/queue/consume`, {
  method: "POST",
  headers: H,
  body: JSON.stringify({ queue: QUEUE, max_messages: 10 }),
});
const out = await res.json();
if (!out.ok) throw new Error(`consume: ${out.error.code} ${out.error.message}`);

for (const msg of out.data.items) {
  if (!valid(msg.payload)) {
    console.error(`poison payload, parking it: ${JSON.stringify(msg.payload).slice(0, 200)}`);
    await ack(msg.message_id);          // never retry something that can't work
    continue;
  }
  try {
    await renderThumbnail(msg.payload);
    await ack(msg.message_id);
  } catch (err) {
    console.warn(`delivery ${msg.delivery_count}/3 failed: ${err.message}`);
    await new Promise((r) => setTimeout(r, 2000 * msg.delivery_count));   // back off in the worker
    await fetch(`${BASE}/v1/queue/nack`, {
      method: "POST",
      headers: H,
      body: JSON.stringify({ queue: QUEUE, message_id: msg.message_id }),
    });
  }
}

async function ack(messageId) {
  await fetch(`${BASE}/v1/queue/ack`, {
    method: "POST",
    headers: H,
    body: JSON.stringify({ queue: QUEUE, receipt_handle: messageId }),
  });
}

async function renderThumbnail(job) {
  console.log(`rendering ${job.upload_id} at ${job.width}px`);
}

Note the asymmetry in field names, since it costs people an afternoon: publish sends body, consume returns payload, ack takes the message id, and nack insists on message_id and rejects the ack spelling.

Getting the queue quiet again

If a bad deploy filled a queue with garbage, drain it rather than debugging it — POST /v1/queue/purge/{queue} empties the queue in one call, and messages are gone for good, so read a few first. To reproduce a poison message on a scratch queue instead, publish one and watch it die:

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"image-thumbnails","body":{"upload_id":null,"width":"big"}}'

Publishing is the only billed step in any of this, at $0.00002 per message (verified 2026-07-26), with $2 free credit for a new account and every diagnostic call free. Rates drift downward here, so read today’s:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id == "queue.publish") | .billing.price_usd'

If per-job retry policy and a dashboard are what you’re missing, BullMQ on your own Redis gives you both and is worth the operational cost. Celery does the same for Python teams. The trade-off we’d defend is different: the queue, the storage the thumbnails land in, the error capture and the tenant-level cost line all sit behind one key here, so debugging this loop doesn’t mean logging into four dashboards.

References

• Infrai queue API reference
• QUEUE_MESSAGE_NOT_IN_FLIGHT error
• BullMQ documentation
• Building webhook retry logic in Node.js