40 report emails failed last night: rerun the job or work the DLQ?

Recovering failed sends from a scheduled report run on Infrai — why a cron rerun mails everyone twice, and how per-message redrive fixes only what broke.

Your 03:00 digest went out to 2,960 of 3,000 tenants. Forty failed — a provider blip, three bad addresses, one tenant whose report generator threw. The question at 09:00 is what to press. Re-triggering the whole schedule is one call and mails 2,960 people a second copy; working the dead-letter queue in Infrai touches exactly the forty that broke. Both are legitimate, and the choice comes down to whether your send path is idempotent.

That’s really the whole article, but the mechanics are worth having in front of you.

Why the rerun is tempting

POST /v1/cron/trigger/{id} fires a scheduled job immediately, out of band, without waiting for the next tick. It’s one request, it needs no new code, and if your report send is genuinely idempotent — you record (run_id, tenant_id) before you hand the message to the provider and skip anything already recorded — then rerunning is fine and you should just do it.

Most send paths aren’t idempotent. They were written as “loop over tenants, call the mailer”, and the second run does exactly what the first one did.

The blast radius is the problem: 2,960 duplicate emails to fix 40 failures is a support ticket per unhappy tenant, and it teaches your recipients to filter you.

What the queue kept for you

If the fan-out went through a queue, the failures are already isolated. A message that fails three deliveries stops being retried and is moved to the dead-letter queue attached to it. Check the count first:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/queue/stats/report-resend" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "queue": "report-resend",
    "message_count": 0,
    "available_count": 0,
    "in_flight_count": 0,
    "delayed_count": 0,
    "dlq_count": 1,
    "oldest_message_age_seconds": 0
  }
}

dlq_count is the number you compare against the failures your logs reported. If they disagree, some sends failed in a way your worker swallowed instead of nacking.

Reading the messages has a wrinkle we hit in testing. GET /v1/queue/dlq/list/report-resend answers with an empty list even while dlq_count says 1, so treat the dead-letter queue as an ordinary queue and consume it by name:

curl -sS -X POST "https://api.infrai.cc/v1/queue/consume" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"report-resend-dlq","max_messages":10}'

{
  "ok": true,
  "data": {
    "items": [
      {
        "message_id": "qmsg_hfElXPP6Z5KaCa0m36NvSSNh",
        "queue": "report-resend-dlq",
        "payload": { "run_id": "2026-07-25", "tenant_id": "t_3081", "email": "ops@example.com" },
        "status": "in_flight",
        "delivery_count": 1
      }
    ],
    "next_cursor": null
  }
}

Now you can actually triage. A bounced address is not the same failure as a 503 from the mailer, and only one of them is worth retrying.

Putting the retryable ones back

Redrive moves a message from the dead-letter queue to the main queue and resets its delivery counter, so it gets a fresh three attempts. It works per message — pass the id you read out of the DLQ:

import process from "node:process";

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set in the environment");
const headers = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

async function readDeadLetters() {
  const res = await fetch("https://api.infrai.cc/v1/queue/consume", {
    method: "POST",
    headers,
    body: JSON.stringify({ queue: "report-resend-dlq", max_messages: 10 }),
  });
  const out = await res.json();
  if (!out.ok) throw new Error(`dlq consume: ${out.error.code}`);
  return out.data.items;
}

function retryable(payload) {
  // Transient provider failures come back; a hard bounce never will.
  return payload.last_error !== "invalid_recipient";
}

let restored = 0;
for (const message of await readDeadLetters()) {
  if (!retryable(message.payload)) {
    console.warn(`skipping ${message.payload.email}: permanent failure, needs a human`);
    continue;
  }
  const res = await fetch("https://api.infrai.cc/v1/queue/dlq/redrive/report-resend", {
    method: "POST",
    headers,
    body: JSON.stringify({ message_id: message.message_id }),
  });
  const out = await res.json();
  if (!out.ok) throw new Error(`redrive ${message.message_id}: ${out.error.message}`);
  restored += out.data.redriven;
}
console.log(`redriven ${restored} messages back onto report-resend`);

Two caveats on that route. Calling it with an empty body to drain the whole dead-letter queue at once currently fails with an internal backend error rather than doing the drain, so the loop above isn’t laziness — it’s the working path. And a redriven message is available immediately, which is wrong if the provider is still down; in that case publish a fresh copy instead and set delay_seconds on it (the field is accepted per message on publish, up to 604800 seconds), then ack the dead-lettered original.

If the failures never reached a queue

Runs that send inline have no messages to inspect. Recovery there means building the failure list yourself during the run and re-enqueueing it afterwards, which is the same work as using a queue in the first place — done later, under pressure, with a partial log.

At minimum, write a row per successful send keyed on (run_id, tenant_id) before you call the provider. It makes a rerun safe, and safe reruns make the whole question easy.

The three recoveries, side by side

Cost and the limits worth knowing

Reading stats, consuming, acking and redriving are all free, rate-limited calls; publishing is the metered one at $0.00002 per message, verified 2026-07-26, so re-publishing forty failures costs $0.0008 — the rounding error on a support call. New accounts carry $2 of credit. Rates fall over time and campaigns happen, so pull the live figure before you plan around it:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.data.capabilities[] | select(.id | startswith("queue.dlq") or .id == "queue.publish") | {id, billing}'

The limitation to plan around: three deliveries is fixed — the update route accepts a different max_receive_count and quietly keeps 3 — and nack requeues immediately with no backoff curve, so a downed provider burns all three attempts in under a minute unless your worker sleeps between them. Retention is 14 days, which is your real deadline for triaging a dead-letter queue.

SQS gives you a redrive policy you can apply to an entire dead-letter queue in one operation, and its documentation on DLQ design is the best reference around regardless of which queue you run. BullMQ’s retry backoff strategies are richer than anything here if you’re already on Redis. The reason to keep the report pipeline on one account is the rest of it — the schedule, the queue, the send and the error record share a credential and a bill, so “which tenant did last night’s failures belong to” is a query rather than a reconciliation.

References

• Infrai Message Queue API reference
• QUEUE_MESSAGE_NOT_IN_FLIGHT error reference
• Amazon SQS: using dead-letter queues
• BullMQ: retrying failing jobs