429s in an image pipeline: retry the upload, never the render

Backoff and queue design for AI image jobs where the storage write fails after generation — so a rate limit costs a retry, not a second render fee.

The 429 that costs you money isn’t the one from the image model. It’s the one that lands after the render succeeded — the upload to object storage fails, your job handler throws, and the retry re-generates a picture you already paid for. Get the order of operations right and a rate limit costs a few seconds; get it wrong and every transient error bills you twice. Infrai keeps the storage write, the retry queue and the error record behind a single key, which is mostly a convenience, but it does mean the durable-write step is a call you already have credentials for.

So: persist the expensive artifact first, retry only the cheap steps, and make every retry land on the same object key.

Two different 429s

They look identical in a log line and need opposite responses.

A provider quota rejection — Gemini, OpenAI, whoever generates your images — means that account is over its per-minute or per-day allowance, and no amount of client-side cleverness raises that ceiling, so the only real responses are to wait, to spread the load across a longer window, or to move up a tier. Backing off works. Hammering doesn’t. A platform rate limit on the storage or queue side is a different animal. Infrai publishes those as RATE_LIMIT_ACCOUNT and RATE_LIMIT_VENDOR, both HTTP 429, both flagged auto-retry with Retry-After honoured:

{
  "ok": false,
  "error": {
    "code": "RATE_LIMIT_ACCOUNT",
    "http_status": 429,
    "message": "account rate limit",
    "retryable": true
  }
}

retryable is the field to branch on, and it lines up with the status code rather than fighting it: a terminal client mistake — a malformed key, a bucket that doesn’t exist, a body the endpoint won’t take — comes back as a 400 with retryable: false, while 503 and retryable: true are reserved for genuine upstream trouble. That makes the old rule of thumb safe to apply here. Retry 5xx and 429, never 4xx; a 400 is never going to succeed on attempt four, and retrying it just burns your budget on a bug.

Order of operations

Here’s the pipeline that doesn’t lose money, in the order the steps must happen:

1. Reserve a job id and derive the object key from it. Not from a timestamp, not from a random UUID generated inside the retry loop.
2. Generate the image.
3. Write the bytes to storage immediately — before thumbnails, before the database row, before the webhook.
4. Everything else, retried freely, because everything else is cheap.

Step 3 is the whole idea. Until the bytes are somewhere durable, every failure downstream — a thumbnail worker that OOMs, a database that rejects the row, a webhook that times out, a deploy that restarts the pod mid-handler — is a failure that can only be repaired by paying a second generation fee. Durability first. Everything else after.

Backoff that doesn’t stampede

Fixed one-second retries across a pool of workers produce a synchronised wave that arrives exactly when the limit is still exhausted. Exponential delay with full jitter spreads it out; a cap keeps a slow recovery from turning into a ten-minute stall.

export function nextDelayMs(attempt, retryAfterHeader, { baseMs = 500, capMs = 30000 } = {}) {
  const retryAfter = Number(retryAfterHeader);
  if (Number.isFinite(retryAfter) && retryAfter > 0) return Math.min(retryAfter * 1000, capMs);

  const ceiling = Math.min(capMs, baseMs * 2 ** (attempt - 1));
  return Math.floor(Math.random() * ceiling);
}

export function isRetryable(status, errorCode) {
  if (status === 429 || status >= 500) return true;
  return ["RATE_LIMIT_ACCOUNT", "RATE_LIMIT_VENDOR", "VENDOR_DOWN", "VENDOR_TIMEOUT"].includes(errorCode);
}

Full jitter — a uniform pick between zero and the ceiling — beats “exponential plus a little noise” for exactly this shape of contention, and it’s two lines. With baseMs at 500 and a 30 s cap, attempts land somewhere inside 0.5 s, 1 s, 2 s, 4 s, 8 s and so on, but never all at once.

The worker

Three attempts on the storage write, an idempotent key, and a retry job on the queue if it still won’t land. The generate call is deliberately left as your provider’s — the point is what surrounds it.

import { setTimeout as sleep } from "node:timers/promises";
import { nextDelayMs, isRetryable } from "./backoff.mjs";

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const API = "https://api.infrai.cc";
const BUCKET = "render-queue-assets";
const headers = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

async function call(method, path, payload) {
  const res = await fetch(`${API}${path}`, {
    method,
    headers,
    body: payload === undefined ? undefined : JSON.stringify(payload),
    signal: AbortSignal.timeout(30000),
  });
  const json = await res.json().catch(() => ({ ok: false, error: { code: "BAD_JSON" } }));
  return { status: res.status, retryAfter: res.headers.get("retry-after"), json };
}

async function storeRender(jobId, pngBytes) {
  const key = `renders/2026-07/${jobId}.png`;
  const payload = { data_base64: pngBytes.toString("base64"), content_type: "image/png" };

  for (let attempt = 1; attempt <= 3; attempt++) {
    const r = await call("PUT", `/v1/storage/object/put/${BUCKET}/${key}`, payload);
    if (r.json.ok) return { key, stored: r.json.data };

    const code = r.json.error?.code;
    if (!isRetryable(r.status, code)) throw new Error(`${code}: not retryable`);
    if (attempt < 3) await sleep(nextDelayMs(attempt, r.retryAfter));
  }

  const requeue = { queue: "render-retry", payload: { job_id: jobId, key, stage: "store" } };
  const q = await call("POST", "/v1/queue/publish", requeue);
  if (!q.json.ok) throw new Error(`render ${jobId} is unstored and unqueued — investigate now`);

  const incident = {
    message: `storage write failed 3x for ${jobId}`,
    fingerprint: "render-store-failure",
    environment: "production",
  };
  await call("POST", "/v1/errors/capture", incident);
  return { key, stored: null, queued: q.json.data.message_id };
}

The last few lines are the part people skip. A job that failed three times and then vanished silently is worse than one that crashed, because nobody finds out until a customer asks where their image went.

Why the key has to come from the job id

renders/2026-07/job_5521.png is derived, so attempt one and attempt four write to the same place. Object writes replace whatever was there — no versioning — so a duplicated retry produces one object, not four, and your bucket doesn’t quietly fill with orphans nobody can attribute.

Check the result rather than trusting the 200:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS \
  "https://api.infrai.cc/v1/storage/object/head/render-queue-assets/renders/2026-07/job_5521.png" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "found": true,
    "status": "found",
    "key": "renders/2026-07/job_5521.png",
    "size_bytes": 26,
    "etag": "082f4ca1fa3e34f77e9dff1d2c704a89",
    "content_type": "image/png",
    "last_modified": "2026-07-26T00:35:20Z"
  }
}

Knowing when to stop

A retry queue without a stop condition is a slow-motion denial of service against your own provider. Queues created through the API come with a 300-second visibility timeout, a delivery cap of 3 and 14-day retention, so a poison job moves to the dead-letter queue instead of cycling forever:

curl -sS "https://api.infrai.cc/v1/queue/stats/render-retry" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "queue": "render-retry",
    "message_count": 2,
    "available_count": 0,
    "in_flight_count": 2,
    "delayed_count": 0,
    "dlq_count": 0,
    "oldest_message_age_seconds": 0
  }
}

Alert on dlq_count and on oldest_message_age_seconds, not on the raw error rate. Rate limits are normal; a backlog that stops draining is not.

Trade-offs, honestly

If you’re already on AWS, S3 plus SQS plus a Lambda does everything above and integrates with the IAM model you already run — that’s a reasonable reason to stay. Cloudflare R2 is the better pick if egress volume is your dominant cost line. And if what you need is sophisticated retry orchestration — step functions, per-step concurrency limits, replay from a UI — a workflow specialist will beat a plain queue, and you should use one.

The Infrai limitation worth flagging here: base64 uploads through the API aren’t recommended above 1 MB, so a large render should go through a presigned PUT instead, and the retry logic then wraps two requests rather than one — the presign, which is cheap and safely repeatable, and the PUT to the returned URL, which is where the bytes actually go. Bucket CORS is configurable, so that PUT can be issued from a browser tab as well as from a worker; if you take that route, the backoff code above has to live in the browser too, and a user closing the tab mid-upload becomes one more way to lose a render you’ve already paid for. For a generation pipeline, keeping the write on the server is the safer default.

What the retry path costs

Storage writes are $0.0001 per call and queue.publish is $0.00002, with error capture at $0.00005 — verified 2026-07-26, against $2 of free credit on a new account. Set that against a single image generation, which on any commercial model is several cents at least: the entire retry apparatus costs less than one avoided re-render, which is the actual argument for building it.

Published rates drift downward over time, so read them live rather than trusting this paragraph:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '[.capabilities[] | select(.id | test("^(storage.object|queue|errors)")) | {id, price: .billing.price_usd}]'

References

• Infrai Storage API reference
• Infrai Queue API reference
• RATE_LIMIT_ACCOUNT
• Handling 429 resource exhaustion errors (Google Cloud)