A few thousand LLM requests as one job: submit, watch, retry, ship the file

The full operational path for a batch inference run on Infrai — chunk sizing, the state that will hang your poll loop, per-row retries and a JSONL export.

A complete run on Infrai is four calls: submit an array of requests, poll status until the job reaches a terminal state, page the results, and optionally export the whole thing as JSONL. No file upload, no custom_id bookkeeping — rows come back keyed by request_index in the order you sent them.

The parts that aren’t obvious are chunk sizing and one state value that will hang a naive poll loop forever. We ran a 50-row job on 2026-07-26 to pin both down, and the numbers below are from that run rather than from the reference.

Submit, and expect it to block

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/submit" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"model": "glm-4-flash", "max_tokens": 12, "messages": [{"role": "user", "content": "One word sentiment: the app crashed again"}]},
      {"model": "glm-4-flash", "max_tokens": 12, "messages": [{"role": "user", "content": "One word sentiment: support fixed it in ten minutes"}]}
    ],
    "batch_timeout": 1800,
    "metadata": {"job": "sentiment-backfill"},
    "store": true
  }'

{
  "ok": true,
  "data": { "batch_id": "batch_79b97c64cb22cdd5f90b646d", "state": "partial", "total_count": 50 }
}

Fifty rows took 24.4 seconds before that response came back — roughly half a second per row, spent inside the HTTP call. So POST /v1/ai/batch/submit is asynchronous in shape but not in practice at small sizes, and your client timeout, not the API, is what limits chunk size.

Chunks of 50 to 100 rows with a 120-second timeout is the sweet spot we’d start from. There’s no published ceiling on array length; an oversized array comes back as a BATCH_REJECTED error rather than a truncated job, so probe upward rather than guessing.

Persist the batch_id the moment you get it. GET /v1/ai/batch/list returned an empty array right after a successful store: true submit, so it isn’t a recovery mechanism — the id in your own database is.

The state that hangs your loop

Poll status the way you’d poll anything else:

curl -sS "https://api.infrai.cc/v1/ai/batch/status/batch_79b97c64cb22cdd5f90b646d" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "batch_id": "batch_79b97c64cb22cdd5f90b646d",
    "state": "partial",
    "progress": 0.98,
    "total_count": 50,
    "completed_count": 49,
    "failed_count": 1,
    "created_at": "2026-07-26T01:19:26.792844Z"
  }
}

One row of the fifty referenced a model that doesn’t exist. The job didn’t fail and it didn’t complete — it settled on partial, at progress: 0.98, and stayed there. We polled it ten more times and nothing moved.

That’s the trap. If your loop waits for completed you’ll spin until your own timeout fires on a job that finished half a minute ago. Treat partial as terminal alongside completed, failed, expired and cancelled, and let the per-row inspection decide what to do about the failures.

The runner

import json
import os
import time

import requests

BASE = "https://api.infrai.cc"
KEY = os.environ.get("INFRAI_API_KEY")
if not KEY:
    raise SystemExit("INFRAI_API_KEY is not set")

HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
TERMINAL = {"completed", "partial", "failed", "expired", "cancelled"}
CHUNK = 50


def chunks(rows, size):
    for i in range(0, len(rows), size):
        yield rows[i:i + size]


def submit(rows):
    payload = {
        "requests": rows,
        "batch_timeout": 1800,
        "metadata": {"job": "sentiment-backfill"},
        "store": True,
    }
    r = requests.post(f"{BASE}/v1/ai/batch/submit", headers=HEADERS, json=payload, timeout=180)
    r.raise_for_status()
    return r.json()["data"]["batch_id"]


def wait(batch_id, poll_seconds=5, limit_seconds=3600):
    deadline = time.time() + limit_seconds
    while time.time() < deadline:
        r = requests.get(f"{BASE}/v1/ai/batch/status/{batch_id}", headers=HEADERS, timeout=30)
        r.raise_for_status()
        data = r.json()["data"]
        print(f"{batch_id} {data['state']} {data['progress']:.0%} "
              f"ok={data['completed_count']} bad={data['failed_count']}")
        if data["state"] in TERMINAL:
            return data
        time.sleep(poll_seconds)
    raise TimeoutError(f"{batch_id} never reached a terminal state")


def collect(batch_id):
    items, cursor = [], None
    while True:
        params = {"limit": 100}
        if cursor:
            params["cursor"] = cursor
        r = requests.get(f"{BASE}/v1/ai/batch/results/{batch_id}", headers=HEADERS, params=params, timeout=60)
        r.raise_for_status()
        data = r.json()["data"]
        items.extend(data["items"])
        cursor = data.get("next_cursor")
        if not cursor:
            return items


def run(rows):
    done, spent = [], 0.0
    for part in chunks(rows, CHUNK):
        batch_id = submit(part)
        wait(batch_id)
        for item in collect(batch_id):
            spent += item.get("cost_usd") or 0.0
            done.append(item)
    failures = [i for i in done if not i.get("ok")]
    print(f"{len(done)} rows, {len(failures)} failed, ${spent:.6f}")
    return done, failures


if __name__ == "__main__":
    tickets = [f"ticket {n}: the checkout page timed out" for n in range(120)]
    requests_payload = [
        {"model": "glm-4-flash", "max_tokens": 12,
         "messages": [{"role": "user", "content": f"One word sentiment: {t}"}]}
        for t in tickets
    ]
    results, bad = run(requests_payload)
    with open("batch-output.jsonl", "w", encoding="utf-8") as fh:
        for item in results:
            fh.write(json.dumps(item, ensure_ascii=False) + "\n")
    print(json.dumps(bad[:3], indent=2)[:400])

Note the cost accounting: there’s no job total on the status object, so you sum cost_usd off the items yourself. Free reads make that cheap — status, results and export are all unbilled, so a long poll loop costs nothing but time.

Reading rows, including the broken ones

curl -sS "https://api.infrai.cc/v1/ai/batch/results/batch_79b97c64cb22cdd5f90b646d?limit=10" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{
  "ok": true,
  "data": {
    "items": [
      { "ok": true, "request_index": 0, "cost_usd": 0.0, "vendor": "zhipu", "region": "china", "result": { "content": "Negative", "finish_reason": "stop", "model": "glm-4-flash", "usage": { "prompt_tokens": 19, "completion_tokens": 3, "total_tokens": 22 } } },
      { "ok": false, "request_index": 7, "cost_usd": 0.0, "error": { "code": "VENDOR_DOWN", "http_status": 503, "message": "The model or service ID not-a-real-model does not exist." } }
    ],
    "total_count": 50,
    "next_cursor": "10"
  }
}

next_cursor is an offset, so paging is exactly as boring as it looks. The failed row carries the vendor’s own message rather than a normalised code — worth flagging if you plan to branch on error types, because the string shape belongs to whichever vendor served the row.

Retrying is a fresh submit containing only the rows whose request_index failed. Keep your source rows in a list and index straight into it; that’s the whole retry mechanism, and it’s why request_index ordering matters more than it first appears.

Getting the file out

If a colleague wants the artefact rather than the API, export writes the whole job as JSONL in one response:

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/export/batch_79b97c64cb22cdd5f90b646d" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{}' \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['content'])" > job.jsonl

wc -l job.jsonl

And if a job is doing the wrong thing, POST /v1/ai/batch/cancel/{id} stops it — free, like the other control calls. Cancel is worth wiring into your runner from day one rather than after the first accidental 5,000-row run.

What it costs to run

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

That returns 30 days of spend broken down per capability, which is the number to reconcile your summed item costs against. Discovery lists the submit call itself at $0.001 with the model tokens passed through on top; our 50-row job on a zero-rated model came back at $0.00 total, and a vision-model row in a separate job cost $0.0001364 on its own. Read GET /v1/discovery for today’s figure — these rates drift downward, and new-account credit of $2 covers a lot of experimenting before any of it matters.

Limits worth knowing before you scale it

Every row is dispatched as a chat completion, so a batch is single-modality by construction — vision rows work because they’re still chat, but an image-generation request in the array won’t route. There’s no webhook on completion, so something in your infrastructure has to do the polling; a cron job on the same key is the obvious home for it. And if your workload is pinned to OpenAI models and can genuinely wait a day, OpenAI’s own Batch API discount will beat this on raw token price, while Together’s batch endpoint does the same for its hosted models — we’d rather you know that than discover it on an invoice.

The reason to run it here anyway is that the queue that feeds the job, the storage the JSONL lands in, the cron that polls, the error tracker that catches a bad row and the usage view that prices the whole thing are the same account and the same key. One job, one bill, no reconciliation.

References

• Infrai AI runtime API reference
• BATCH_REJECTED error reference
• Together AI batch inference
• OpenAI Batch API guide