Triaging a dead-letter backlog after a rate-limited API run

What's parked, why it parked, and which of it is worth replaying. Monitoring the DLQ depth, classifying failures, and a Node drainer that replays only the transient ones.

Parked messages are evidence, not garbage. When a sync against a rate-limited vendor API blows through its retry budget, the dead-letter queue tells you which records failed, how many times each was attempted, and — if you put the right fields in the payload — which tenant is now out of sync. Infrai gives every queue a dead-letter lane named after it, and reading that lane costs nothing: consume, ack, stats and the dead-letter routes are all free, so triage is never the expensive part.

The expensive part is replaying blindly. Half a parked backlog is usually messages that will fail again for the same permanent reason, and pushing them back onto the live queue just buys you the same outage tomorrow.

Three ways to end up parked

A queue’s max_receive_count defaults to 3. Every consume that doesn’t end in an ack bumps delivery_count, and the fourth strike moves the message to <queue>.dlq. That single counter can’t tell you why, so classify before you replay.

The first two rows are the ones a redrive is for. The last two will happily consume another three deliveries and land straight back where they started.

Seed the lane, then look at it

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue":"vendor-sync","body":{"tenant":"acme","record_id":55219,"op":"upsert"}}'

Counters first. This is the monitor we’d run on a five-minute schedule; it exits non-zero so any supervisor can alert on it.

import os
import sys
import requests

BASE = "https://api.infrai.cc"
KEY = os.environ.get("INFRAI_API_KEY")
if not KEY:
    sys.exit("INFRAI_API_KEY is not set")

HEADERS = {"Authorization": f"Bearer {KEY}"}
DLQ_ALERT = 25
AGE_ALERT_SECONDS = 900

try:
    res = requests.get(f"{BASE}/v1/queue/stats/vendor-sync", headers=HEADERS, timeout=15)
    res.raise_for_status()
except requests.RequestException as exc:
    sys.exit(f"stats call failed: {exc}")

body = res.json()
if body.get("ok") is not True:
    sys.exit(f"api error: {body.get('error', {}).get('code')}")

data = body["data"]
print(
    f"available={data['available_count']} in_flight={data['in_flight_count']} "
    f"dlq={data['dlq_count']} oldest={data['oldest_message_age_seconds']}s"
)

problems = []
if data["dlq_count"] >= DLQ_ALERT:
    problems.append(f"dead-letter backlog {data['dlq_count']}")
if data["oldest_message_age_seconds"] >= AGE_ALERT_SECONDS:
    problems.append(f"oldest message {data['oldest_message_age_seconds']}s")
if problems:
    sys.exit("ALERT: " + "; ".join(problems))

{
  "ok": true,
  "data": {
    "queue": "vendor-sync",
    "message_count": 0,
    "available_count": 0,
    "in_flight_count": 0,
    "delayed_count": 0,
    "dlq_count": 1,
    "oldest_message_age_seconds": 0
  }
}

dlq_count is the alert. oldest_message_age_seconds is the one that tells you whether the live queue is also in trouble — a growing age with a healthy dead-letter count usually means the worker died rather than the vendor.

The redrive route, honestly

There’s a one-call recovery route, and you should know its current state before you plan around it.

curl -sS -X POST "https://api.infrai.cc/v1/queue/dlq/redrive/vendor-sync" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{}'

On 2026-07-26 that answered with an internal argument error on every queue we tried, and GET /v1/queue/dlq/list/{queue} returned an empty items array on a queue whose dlq_count was 1:

{
  "ok": false,
  "error": {
    "code": "INVALID_ARGUMENT",
    "http_status": 400,
    "message": "dlq.redrive failed: 'RedisStreamsBackend' object has no attribute '_index'",
    "retryable": false
  }
}

That’s a limitation, not a dead end. The dead-letter queue is an ordinary queue with a predictable name, so everything below works today with the routes that do.

The drainer that classifies before it replays

import process from "node:process";

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const LIVE = "vendor-sync";
const PARKED = "vendor-sync.dlq";
const h = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

async function send(route, payload) {
  const response = await fetch(`https://api.infrai.cc${route}`, { method: "POST", headers: h, body: JSON.stringify(payload) });
  const result = await response.json();
  if (result.ok !== true) throw new Error(`${route}: ${result.error.code} ${result.error.message}`);
  return result.data;
}

// Only replay what a pause would have fixed.
const replayable = (payload) => payload.last_status === 429 || payload.last_status >= 500;

let replayed = 0;
let quarantined = 0;

for (;;) {
  const { items } = await send("/v1/queue/consume", { queue: PARKED, max_messages: 10 });
  if (items.length === 0) break;

  for (const message of items) {
    const payload = message.payload ?? {};
    if (replayable(payload)) {
      await send("/v1/queue/publish", { queue: LIVE, body: { ...payload, replay_of: message.message_id } });
      replayed++;
    } else {
      console.warn(`quarantined ${message.message_id} tenant=${payload.tenant} status=${payload.last_status}`);
      quarantined++;
    }
    await send("/v1/queue/ack", { queue: PARKED, receipt_handle: message.message_id });
  }
}

console.log(`replayed ${replayed}, quarantined ${quarantined}`);

Two details make this safe to run twice. Acking the parked copy only after the republish means a crash mid-loop leaves the message in the dead-letter queue rather than nowhere. And replay_of gives the live worker a key to deduplicate on, which matters because at-least-once delivery is a promise about the queue, not about your vendor’s side effects.

Quarantined messages get logged and dropped here. Send them to a vendor-sync.parked queue instead if someone is going to look at them on Monday.

What a replay costs

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id | startswith("queue.")) | {id, billable: .billing.is_billable, price_usd: .billing.price_usd}'

Republishing is the only metered call in this whole workflow — $0.00002 per message, verified 2026-07-26 — so recovering a 10,000-message backlog is $0.20, and inspecting it first is free. That asymmetry is deliberate and it’s the argument for triaging rather than mass-redriving: reading costs nothing, writing costs a fifth of a cent per thousand. New accounts start with $2 of free credit. Check GET /v1/account/usage after a big replay rather than trusting this figure, which will probably be lower by the time you read it.

Where a different stack fits better

sqs has a first-party redrive with a maximum-messages-per-second control and CloudWatch alarms already wired to ApproximateNumberOfMessagesVisible — if you’re on AWS and this is your only queue, stick with it. celery with a results backend gives Python teams richer per-task failure metadata than a payload field you maintain yourself.

The trade-off you accept here is fewer built-in operational tools in exchange for one credential across the stack: the alert email, the error record and the per-tenant cost of the replay all live on the same account as the queue. On pacing the replay so you don’t immediately re-trigger the rate limit, see the 429 and Retry-After consumer guide.

References

• Infrai queue API reference
• QUEUE_MESSAGE_NOT_IN_FLIGHT error reference
• Amazon SQS dead-letter queues
• DEV: retries, dead-letter queues and the parking lot pattern