Retried image jobs that overwrite, vanish, or come back stale

Same key, last write wins, and no version history to fall back on. How retries corrupt an AI image pipeline on object storage, and the key scheme that ends it.

One key, one object, last writer wins. When a generation job retries and writes to renders/usr_31/job_7781.png a second time, the first image is gone — not archived, gone — and if the retry produced different pixels your user now sees an image nobody approved. Infrai buckets store exactly one current version per key, so the fix isn’t a storage setting, it’s the key you choose.

Three symptoms show up in support tickets and they have three different causes. The image is the wrong one (a retry wrote different bytes to the same key). The image is missing (the key you saved in Postgres isn’t the key you wrote). The image is stale even though the job clearly reran — that last one is the interesting case, and it’s an idempotency key doing exactly what you told it to.

Prove the overwrite to yourself

Write two different payloads to one key and watch the ETag move:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X PUT \
  "https://api.infrai.cc/v1/storage/object/put/renders-demo/renders/usr_31/overwrite-probe.png" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  --data @render-a.json

{
  "ok": true,
  "data": {
    "bucket_id": "bkt_ada779bdb4ae4687b97012",
    "key": "renders/usr_31/overwrite-probe.png",
    "size_bytes": 300,
    "etag": "ec11aea0501a4c68d89e47d243aaa1eb",
    "content_type": "image/png",
    "created_at": "2026-07-26T00:44:12.101334Z"
  }
}

Run it again with render-b.json and the response reports size_bytes: 900 and ETag 0221d6bb9ded8e9f12ccb14defe96686. One object, one key, and the 300-byte version is unrecoverable — GET /v1/storage/object/head/{bucket}/{key} only ever describes the current one:

curl -sS \
  "https://api.infrai.cc/v1/storage/object/head/renders-demo/renders/usr_31/overwrite-probe.png" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

There’s no versionId parameter anywhere in the storage surface. Infrai doesn’t support object versioning today, so the S3 recovery story — flip on versioning, list versions, restore the previous one — has no equivalent here. If version history or an object-lock retention window is a compliance requirement rather than a nice-to-have, run MinIO or use Amazon S3 versioning directly and accept the extra account.

The stale-image case: a reused idempotency key

storage.object.put accepts an optional idempotency_key, and when you omit it the platform derives one from the content hash. That default is the safe one. Sending your own is where pipelines get hurt.

We tested this on 26 July 2026. A first write with idempotency_key: "render_probe_0726" stored a 300-byte object. A second write with the same key and 900 bytes of different data came back "ok": true — reporting the old 300-byte object, with the old ETag — and head confirmed the bucket still held the original. The new image was silently dropped, and metadata.idempotent_replay read false on that response, so you can’t detect the swallow from that field alone.

Which is fine if your key is job_id + attempt_hash and genuinely means “this exact work”. It’s a data-loss bug if your key is job_id and attempt two produces a better image.

So: derive the idempotency key from what you’re storing, or leave it out and let the content hash do the work.

Content-addressed keys end the whole class of bug

Name the object after its bytes. A retry that regenerates identical output writes the identical key — a harmless no-op. A retry that produces different output writes a different key, so nothing is overwritten and both are inspectable. The mutable part becomes a small pointer row in your database, or a copied object if you want a stable URL:

import { createHash } from "node:crypto";

const API = "https://api.infrai.cc";
const BUCKET = "renders-demo";
const token = process.env.INFRAI_API_KEY;
if (!token) throw new Error("INFRAI_API_KEY is not set");
const auth = { Authorization: `Bearer ${token}`, "Content-Type": "application/json" };

/** Store generated image bytes under a key derived from their sha256. */
export async function storeRender(userId, bytes) {
  const digest = createHash("sha256").update(bytes).digest("hex").slice(0, 24);
  const key = `renders/${userId}/${digest}.png`;

  const probe = await fetch(`${API}/v1/storage/object/head/${BUCKET}/${key}`, { method: "GET", headers: auth });
  const existing = await probe.json();
  if (probe.ok && existing.ok !== false && existing.data.found) {
    return { key, etag: existing.data.etag, deduped: true };
  }

  const payload = { data_base64: Buffer.from(bytes).toString("base64"), content_type: "image/png" };
  const put = await fetch(`${API}/v1/storage/object/put/${BUCKET}/${key}`, {
    method: "PUT",
    headers: auth,
    body: JSON.stringify(payload),
  });
  const stored = await put.json();
  if (!put.ok || stored.ok === false) throw new Error(stored?.error?.code ?? `HTTP ${put.status}`);
  return { key, etag: stored.data.etag, deduped: false };
}

The head probe before the write is free and rate-limited, so deduplicating costs nothing but a round trip — worth it when the payload is a 4 MB PNG.

For a stable “current render” URL that your app can sign without a database lookup, copy rather than overwrite:

curl -sS -X POST "https://api.infrai.cc/v1/storage/object/copy" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"src_bucket":"renders-demo","src_key":"renders/usr_31/job_7781.png","dst_bucket":"renders-demo","dst_key":"renders/usr_31/current.png"}'

POST /v1/storage/object/copy preserves content_type and custom metadata, and the immutable original stays where it is. You’ve swapped an irreversible overwrite for a pointer move you can repeat.

Triage table

Listing the prefix is usually the fastest way to see which of the four you have:

curl -sS \
  "https://api.infrai.cc/v1/storage/object/list/renders-demo?prefix=renders%2Fusr_31%2F" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

What the safety costs

Content addressing trades storage for certainty: you keep every distinct render instead of one per job. Verified 26 July 2026, storage.object.put and storage.object.copy are $0.0001 per call, storage.object.get is $0.0002, and head, list, presign and delete are free and rate-limited, with stored bytes and egress metered on top. A thousand extra renders is a dime in call fees; the bytes are the real line item, which is why a lifecycle rule on the renders/ prefix is worth setting on day one. Read the current numbers:

curl -sS https://api.infrai.cc/v1/discovery \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | python3 -c "import json,sys; print([ (c['id'], c['billing'].get('price_usd','free')) for c in json.load(sys.stdin)['capabilities'] if c['id'].startswith('storage.object') ])"

Rates drift downward over time, so treat those figures as an upper bound rather than a quote.

Limits worth knowing before you rely on this

Object storage gives you no transactions: copy then delete is two calls, and a crash between them leaves both. The base64 put path isn’t recommended above 1 MB — larger renders want a presigned upload or multipart. A conflicting idempotent replay elsewhere in the platform surfaces as IDEMPOTENCY_KEY_CONFLICT, which is worth wiring into your error handling even though the storage put we tested returned success instead. And a deleted object is really gone — there’s no recycle bin, which is precisely why the immutable key scheme is doing the work that versioning would.

The pieces around this all sit on the same credential: the queue that retries the generation, the cron that expires old renders, the error tracker that catches the corrupt PNG, and one usage view that tells you which tenant caused the storage bill.

References

• Infrai storage API reference
• IDEMPOTENCY_KEY_CONFLICT error reference
• Retaining multiple versions of objects with S3 Versioning
• Object versioning — MinIO AIStor documentation