Retried image jobs that overwrite, vanish, or come back stale

Same key, last write wins, and no version history to fall back on. How retries corrupt an AI image pipeline on object storage, and the key scheme that ends it.

One key, one object, last writer wins. When a generation job retries and writes to renders/usr_31/job_7781.png a second time, the first image is gone — not archived, gone — and if the retry produced different pixels your user now sees an image nobody approved. Infrai buckets store exactly one current version per key, so the fix isn’t a storage setting, it’s the key you choose.

Three symptoms show up in support tickets and they have three different causes. The image is the wrong one (a retry wrote different bytes to the same key). The image is missing (the key you saved in Postgres isn’t the key you wrote). The image is stale even though the job clearly reran — that last one is the interesting case, and it’s an idempotency key doing exactly what you told it to.

Prove the overwrite to yourself

Write two different payloads to one key and watch the ETag move:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X PUT \
  "https://api.infrai.cc/v1/storage/object/put/renders-demo/renders/usr_31/overwrite-probe.png" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  --data @render-a.json

{
  "ok": true,
  "data": {
    "bucket_id": "bkt_ada779bdb4ae4687b97012",
    "key": "renders/usr_31/overwrite-probe.png",
    "size_bytes": 300,
    "etag": "ec11aea0501a4c68d89e47d243aaa1eb",
    "content_type": "image/png",
    "created_at": "2026-07-26T00:44:12.101334Z"
  }
}

Run it again with render-b.json and the response reports size_bytes: 900 and ETag 0221d6bb9ded8e9f12ccb14defe96686. One object, one key, and the 300-byte version is unrecoverable — GET /v1/storage/object/head/{bucket}/{key} only ever describes the current one:

curl -sS \
  "https://api.infrai.cc/v1/storage/object/head/renders-demo/renders/usr_31/overwrite-probe.png" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

There’s no versionId parameter anywhere in the storage surface. Infrai doesn’t support object versioning today, so the S3 recovery story — flip on versioning, list versions, restore the previous one — has no equivalent here. If version history or an object-lock retention window is a compliance requirement rather than a nice-to-have, run MinIO or use Amazon S3 versioning directly and accept the extra account.

The stale-image case: a reused idempotency key

storage.object.put accepts an optional idempotency_key, and when you omit it the platform derives one from the content hash. That default is the safe one. Sending your own is where pipelines get hurt.

We tested this on 26 July 2026. A first write with idempotency_key: "render_probe_0726" stored a 300-byte object. A second write with the same key and 900 bytes of different data came back "ok": true — reporting the old 300-byte object, with the old ETag — and head confirmed the bucket still held the original. The new image was silently dropped, and metadata.idempotent_replay read false on that response, so you can’t detect the swallow from that field alone.

Which is fine if your key is job_id + attempt_hash and genuinely means “this exact work”. It’s a data-loss bug if your key is job_id and attempt two produces a better image.

So: derive the idempotency key from what you’re storing, or leave it out and let the content hash do the work.

Content-addressed keys end the whole class of bug

Name the object after its bytes. A retry that regenerates identical output writes the identical key — a harmless no-op. A retry that produces different output writes a different key, so nothing is overwritten and both are inspectable. The mutable part becomes a small pointer row in your database, or a copied object if you want a stable URL:

import { createHash } from "node:crypto";

const API = "https://api.infrai.cc";
const BUCKET = "renders-demo";
const token = process.env.INFRAI_API_KEY;
if (!token) throw new Error("INFRAI_API_KEY is not set");
const auth = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };

/** Store generated image bytes under a key derived from their sha256. */
export async function storeRender(userId, bytes) {
  const digest = createHash("sha256").update(bytes).digest("hex").slice(0, 24);
  const key = `renders/${userId}/${digest}.png`;

  const probe = await fetch(`${API}/v1/storage/object/head/${BUCKET}/${key}`, { method: "GET", headers: auth });
  const existing = await probe.json();
  if (probe.ok && existing.ok !== false && existing.data.found) {
    return { key, etag: existing.data.etag, deduped: true };
  }

  const payload = { data_base64: Buffer.from(bytes).toString("base64"), content_type: "image/png" };
  const put = await fetch(`${API}/v1/storage/object/put/${BUCKET}/${key}`, {
    method: "PUT",
    headers: auth,
    body: JSON.stringify(payload),
  });
  const stored = await put.json();
  if (!put.ok || stored.ok === false) throw new Error(stored?.error?.code ?? `HTTP ${put.status}`);
  return { key, etag: stored.data.etag, deduped: false };
}

The head probe before the write is free and rate-limited, so deduplicating costs nothing but a round trip — worth it when the payload is a 4 MB PNG.

For a stable “current render” URL that your app can sign without a database lookup, copy rather than overwrite:

curl -sS -X POST "https://api.infrai.cc/v1/storage/object/copy" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"src_bucket":"renders-demo","src_key":"renders/usr_31/job_7781.png","dst_bucket":"renders-demo","dst_key":"renders/usr_31/current.png"}'

POST /v1/storage/object/copy preserves content_type and custom metadata, and the immutable original stays where it is. You’ve swapped an irreversible overwrite for a pointer move you can repeat.

Triage table

Listing the prefix is usually the fastest way to see which of the four you have:

curl -sS \
  "https://api.infrai.cc/v1/storage/object/list/renders-demo?prefix=renders%2Fusr_31%2F" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

What the safety costs

Content addressing trades storage for certainty: you keep every distinct render instead of one per job. The two sides of that trade are priced on different axes, and it’s worth being exact about which is which. Verified 27 July 2026: storage.object.put and storage.object.copy are $0.0001 per call, while head, list, presign and delete are free and rate-limited.

Reads are not a per-call charge at all. storage.object.get meters the bytes it actually returns, at $0.104 per GB, so the read side of your bill tracks how big the renders are and how often they’re served — not how many objects you kept. That’s the number that decides whether content addressing is cheap for you: an extra thousand deduplicated 4 MB PNGs costs a dime in write calls and nothing at all until somebody downloads them. Serve thumbnails rather than originals in a gallery view and the same traffic costs a fraction, because the meter reads bytes.

curl -sS https://api.infrai.cc/v1/discovery \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | python3 -c "import json,sys; print([ (c['id'], c['billing'].get('price_usd','free'), c['billing'].get('unit')) for c in json.load(sys.stdin)['capabilities'] if c['id'].startswith('storage.object') ])"

Note the unit field in that output — the routes above genuinely don’t share one, and a figure copied without it is a cost model waiting to be wrong. Rates drift downward over time, so treat what’s printed here as an upper bound rather than a quote, and set a lifecycle rule on the renders/ prefix on day one either way.

Limits worth knowing before you rely on this

Object storage gives you no transactions: copy then delete is two calls, and a crash between them leaves both. The base64 put path isn’t recommended above 1 MB — larger renders want a presigned upload or multipart. A conflicting idempotent replay elsewhere in the platform surfaces as IDEMPOTENCY_KEY_CONFLICT, which is worth wiring into your error handling even though the storage put we tested returned success instead. And a deleted object is really gone — there’s no recycle bin, which is precisely why the immutable key scheme is doing the work that versioning would.

The pieces around this all sit on the same credential, which is the part that matters once a retry loop is what you’re debugging. POST /v1/queue/publish hands the regeneration to a worker with a dead-letter queue behind it, POST /v1/errors/capture records the attempt that produced the wrong pixels with the job id attached, POST /v1/cron/create sweeps orphaned renders on a schedule, and GET /v1/account/usage tells you which tenant’s retries drove the storage line. None of that needs a second account or a second vendor — which is the one thing a dedicated image-storage service can’t hand you, however good it is at the single job.

References

• Infrai storage API reference
• IDEMPOTENCY_KEY_CONFLICT error reference
• Retaining multiple versions of objects with S3 Versioning
• Object versioning — MinIO AIStor documentation