Cheaper than GPT-4 for summarise, classify and extract: five levers

Which cost levers actually move the bill on high-volume text work, measured on one 500k-document workload — model tier, output caps, batching, prompt size and cache.

Rank the levers by leverage and the list is short: change the model tier, cap the output, push the non-interactive half into a batch, shrink the prompt, then let the cache pick up repeats. On Infrai all five ride the same OpenAI-shaped call, so trying them is a config edit rather than four integrations — and the measuring calls that tell you whether a lever worked are free.

Only the first lever is worth an afternoon. The rest are worth an hour each, and they compound.

Take one workload for the whole piece: 500,000 support documents a month, each about 1,200 prompt tokens, each producing a one-sentence summary plus a JSON tag block of roughly 200 tokens.

Lever one: the tier, and nothing else is close

Published per-million-token rates span three orders of magnitude, and the gap between a flagship and a flash-tier model dwarfs every clever optimisation you could apply to either. Pull the live card first:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/ai/models?capability=chat&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" | head -c 2000

Verified 2026-07-26, that catalogue put gpt-5 at $1.25 in / $10 out per million tokens, gpt-5-mini at $0.25 / $2, and glm-4-flashx at $0.014 / $0.014. Our 500k-document month therefore costs $1,750 on gpt-5, $350 on gpt-5-mini and $9.80 on glm-4-flashx. Read those as ratios rather than as facts with a long shelf life — model prices have been falling steadily, and the cheap end falls fastest.

The catch is that a flash-tier model is not a flagship at every task. Summarisation and closed-set tagging survive the downgrade well in our testing; multi-step reasoning over contradictory documents does not. So gate the swap on your own eval set, not on the rate card.

Lever two: the output cap you probably forgot

Output tokens cost 4× to 10× input on most Western models, which makes an uncapped completion the single most expensive thing in a summarisation loop. max_tokens is a hard ceiling, and on this workload dropping 200 to 80 takes gpt-5-mini from $350 to $230 a month — a 34% cut for one line of JSON.

Pair it with a prompt that asks for the shorter form, or the model will simply get truncated mid-sentence.

Lever three: batch the half that nobody is waiting for

Nightly tagging, backfills and archive summarisation have no user staring at a spinner. Price that before you build it:

curl -sS -X POST "https://api.infrai.cc/v1/ai/cost/estimate" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Summarise this ticket in one sentence and pick a tag."}],
    "expected_output_tokens": 120,
    "batch_mode": true
  }'

The batch_discount line came back at roughly half the interactive figure. The submission side is a single POST that takes an array of ordinary chat requests — no JSONL file upload, no file id to babysit:

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");
const BASE = "https://api.infrai.cc";
const headers = { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" };

const tickets = [
  "My card was charged twice for the same invoice last Tuesday.",
  "The export button does nothing in Firefox 141.",
];

const submitted = await fetch(`${BASE}/v1/ai/batch/submit`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    requests: tickets.map((t) => ({
      model: "glm-4-flash",
      messages: [{ role: "user", content: `Label this ticket with one word: ${t}` }],
      max_tokens: 8,
    })),
    batch_timeout: 3600,
    metadata: { job: "nightly-tagging" },
    store: true,
  }),
});
if (!submitted.ok) throw new Error(`submit failed: ${submitted.status}`);
const { data: job } = await submitted.json();
console.log(`${job.batch_id} ${job.state} ${job.total_count} rows`);

let state = job.state;
while (state !== "completed" && state !== "failed" && state !== "expired") {
  await new Promise((r) => setTimeout(r, 2000));
  const s = await fetch(`${BASE}/v1/ai/batch/status/${job.batch_id}`, { method: "GET", headers });
  if (!s.ok) throw new Error(`status failed: ${s.status}`);
  ({ state } = (await s.json()).data);
}

const r = await fetch(`${BASE}/v1/ai/batch/results/${job.batch_id}`, { method: "GET", headers });
if (!r.ok) throw new Error(`results failed: ${r.status}`);
const { data: page } = await r.json();
for (const item of page.items) {
  if (!item.ok) { console.error(`row ${item.request_index} failed`); continue; }
  console.log(`${item.request_index}: ${item.result.content} ($${item.cost_usd})`);
}

Two rows came back completed inside a second and a half in our testing, which tells you the queue runs synchronously at small sizes — don’t design around that for 50,000 rows.

{
  "ok": true,
  "data": {
    "items": [
      { "ok": true, "request_index": 0, "result": { "content": "Duplicate Charge", "usage": { "prompt_tokens": 18, "completion_tokens": 4 } }, "cost_usd": 0.0, "vendor": "zhipu" },
      { "ok": true, "request_index": 1, "result": { "content": "Bug", "usage": { "prompt_tokens": 18, "completion_tokens": 3 } }, "cost_usd": 0.0, "vendor": "zhipu" }
    ],
    "total_count": 2,
    "next_cursor": null
  }
}

Two limitations, both worth knowing before you commit. The submit response carries only batch_id, state and total_count — no estimated cost, whatever the flow documentation implies — and GET /v1/ai/batch/list returned an empty array immediately after a successful submit, so keep your own record of the id.

Lever four: the prompt you send 500,000 times

A system prompt with four few-shot examples is charged on every call, forever. Measure it in isolation before deciding whether the examples earn their place:

curl -sS -X POST "https://api.infrai.cc/v1/ai/tokens/count" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5-mini",
    "messages": [{"role": "system", "content": "You are a support triage assistant. Return JSON with keys summary and tag. Tags: billing, bug, account, other."}]
  }'

Trimming 1,200 prompt tokens to 700 takes the gpt-5-mini month from $350 to $287 — about 18%. Smaller than the tier swap, permanent, and free to attempt.

Lever five: repeats, and what the cache is really worth

Send a byte-identical request body twice and the second one is served from an account-scoped exact-match cache. It isn’t free: our repeat billed at roughly 70% of the original, and the response flips infrai.cache from false to true so you can measure your own hit rate instead of assuming one. A paraphrase misses. Deduplicate identical documents upstream and you’ll get more out of it than any prompt trick.

Does the cheap model still return valid JSON?

For extraction this is the question that decides everything, so test it rather than trusting a feature matrix.

curl -sS -X POST "https://api.infrai.cc/v1/chat/completions" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-4-flashx",
    "max_tokens": 60,
    "response_format": {"type": "json_object"},
    "messages": [{"role": "user", "content": "Extract invoice_no and amount_usd as JSON: Invoice INV-4471 for $128.50 was paid on 3 March."}]
  }'

That returned {"invoice_no":"INV-4471","amount_usd":128.50} cleanly. json_object mode holds on the flash tier; a full json_schema response format is accepted but not enforced on those models, so validate against your own schema — Zod, Pydantic, whatever you already have — and retry once on a parse failure. Budget for that retry: a 2% invalid-JSON rate is a 2% cost increase.

The ledger

Stacking the first four on the offline portion turns $1,750 a month into something closer to a rounding error. Confirm it against GET /v1/account/usage, which breaks 30 days of real spend down per capability — the only number that settles the argument.

Where somebody else is the better buy

If your traffic is already all-OpenAI and you’re happy there, their Batch API gives a documented 50% discount with prompt caching on top, and you’d be better off using it than adding a hop. If the documents can’t leave your network, run a small model under Ollama where the marginal token cost is zero and the constraint is GPU time. If you need sub-second first-token latency on a cheap model, a specialist inference host like Groq will beat a general gateway on that axis alone.

The argument for consolidating is not that any single rate is lowest. It’s that the tagging call, the queue that schedules it, the object store holding the source documents and the usage view that attributes all of it to a tenant sit behind one key — and moving to whichever model wins next quarter stays a one-string change.

References

• Infrai AI runtime API reference
• Infrai account API reference
• OpenAI Batch API guide
• Ollama: embedding and small models