OpenAI's Batch API vs a gateway's batch queue: the discount and the ceremony

The 50% batch discount is real and vendor-specific. The developer-experience gap is bigger, and it's mostly about file handling. A measured comparison.

Yes, there’s a real difference, but it isn’t the one the comparison posts fixate on. OpenAI’s Batch API halves the token rate on its own models in exchange for a 24-hour window; a platform batch like Infrai’s passes vendor cost through instead, and wins on the part nobody advertises — no file upload, no JSONL round trip, and a different model per row if you want one.

Which matters more depends on whether your batch is locked to one vendor’s models. If it is, take the discount. If half your rows would run fine on a model that already costs a tenth of the discounted one, the discount is a rounding error — that arbitrage is the whole reason Infrai exists as a layer.

The two shapes

OpenAI’s flow is file-oriented, and deliberately so: you write a .jsonl file where each line is a request with a custom_id, upload it through the Files API, create a batch that references the file id, poll the batch object, then download an output file and join it back onto your rows by custom_id. It’s built for very large offline jobs and it scales beautifully — but it’s four API surfaces and two file formats before you see a single result.

A gateway batch is request-oriented. You POST an array.

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/submit" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"model": "glm-4-flash", "max_tokens": 20, "messages": [{"role": "user", "content": "Summarise in five words: the checkout page timed out twice."}]},
      {"model": "glm-4-flash", "max_tokens": 20, "messages": [{"role": "user", "content": "Summarise in five words: invoice PDF fails to download."}]}
    ],
    "batch_timeout": 3600,
    "metadata": {"job": "ticket-summaries"},
    "store": true
  }'

{
  "ok": true,
  "data": { "batch_id": "batch_c34e73a9f5402964f955eb82", "state": "completed", "total_count": 3 }
}

Note what came back — three fields. The reference describes estimated_eta and estimated_cost_usd alongside them; we didn’t get either on any submit we ran on 2026-07-26, so don’t build a progress UI that depends on them.

Axis by axis

Size the job before you send it

Token counting is free on this surface, so there’s no reason to guess what a few thousand rows will cost.

curl -sS -X POST "https://api.infrai.cc/v1/ai/tokens/count" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5-mini",
    "messages": [
      {"role": "system", "content": "Summarise the ticket in five words."},
      {"role": "user", "content": "The checkout page timed out twice and the order never appeared in my history."}
    ]
  }'

{ "ok": true, "data": { "prompt_tokens": 41, "model": "gpt-5-mini" } }

Forty-one tokens per row times 5,000 rows is 0.2 Mtok of input. On a model at $0.25 per Mtok that’s about five cents of input before output; halving it saves you two and a half cents. On a job of that size the discount argument evaporates, which is why we’d rather talk about which model you run than what percentage you shave off it.

curl -sS "https://api.infrai.cc/v1/ai/models?capability=chat&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Verified 2026-07-26, that catalogue held 22 servable chat models spanning $0.00 to $15.00 per Mtok of input, and the submit call itself is listed in GET /v1/discovery at $0.001 per call on top of pass-through token cost. Rates fall over time and vendors run promotions, so read the catalogue rather than trusting this paragraph in six months.

Polling and reading, with no files involved

curl -sS "https://api.infrai.cc/v1/ai/batch/status/batch_c34e73a9f5402964f955eb82" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "batch_id": "batch_c34e73a9f5402964f955eb82",
    "state": "completed",
    "progress": 1.0,
    "total_count": 3,
    "completed_count": 3,
    "failed_count": 0,
    "created_at": "2026-07-26T01:10:23.692377Z"
  }
}

There’s no total_cost_usd on that object, whatever the flow summary implies — per-item cost_usd lives on the results, so the job total is a sum you do yourself:

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const batchId = "batch_c34e73a9f5402964f955eb82";
let cursor = null;
let spent = 0;
let rows = 0;

do {
  const url = new URL(`https://api.infrai.cc/v1/ai/batch/results/${batchId}`);
  url.searchParams.set("limit", "100");
  if (cursor) url.searchParams.set("cursor", cursor);
  const res = await fetch(url, { headers: { Authorization: `Bearer ${KEY}` } });
  if (!res.ok) throw new Error(`results page failed: ${res.status}`);
  const { data } = await res.json();
  for (const item of data.items) {
    rows++;
    spent += item.cost_usd ?? 0;
    if (!item.ok) console.error(`row ${item.request_index} failed: ${JSON.stringify(item.error)}`);
  }
  cursor = data.next_cursor;
} while (cursor);

console.log(`${rows} rows, $${spent.toFixed(6)} total`);

If you do want the OpenAI-shaped artefact after all, export gives you one line of JSON per row without ever having uploaded one:

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/export/batch_c34e73a9f5402964f955eb82" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{}'

{
  "ok": true,
  "data": {
    "batch_id": "batch_c34e73a9f5402964f955eb82",
    "format": "jsonl",
    "content": "{\"request_index\": 0, \"ok\": true, \"result\": {\"content\": \"Checkout page timeout retries\"}}\n{\"request_index\": 1, \"ok\": true, \"result\": {\"content\": \"PDF download error\"}}",
    "total_count": 3
  }
}

Status, results, export and cancel are all free reads on this account class, so watching a long job costs nothing — which is not true of every metered gateway.

The honest ledger

Three things bit us, and you should plan around them.

Every row is dispatched as a chat completion. Put an image-generation request in the array and it fails with a vendor error rather than routing to the image surface, so keep batches single-modality. GET /v1/ai/batch/list returned an empty array immediately after a successful store: true submit, so treat the batch_id from the submit response as the only reliable handle and persist it. And the per-row error object is the vendor’s, not a normalised one — worth flagging if you plan to alert on specific failure codes.

Which one to pick

Stick with OpenAI’s Batch API if your workload is pinned to OpenAI models, runs into the millions of rows, and genuinely tolerates 24 hours — the discount is real money at that scale and the file pipeline is the right abstraction for it. Together’s batch endpoint makes the same trade on its own hosted models. Choose a gateway batch when the rows are heterogeneous, when latency in minutes beats latency in hours, or when the job is one step in a product flow rather than an offline chore.

That last case is where one credential stops being a slogan: the same key that ran the batch also holds the queue that scheduled it, the storage the outputs land in, the email that tells someone it finished, and the usage view that attributes all of it to a tenant. You’re not comparing two batch APIs. You’re comparing one batch API against a batch API plus four accounts you’d otherwise open.

References

• Infrai AI runtime API reference
• BATCH_REJECTED error reference
• OpenAI Batch API guide
• Together AI batch inference