A low-cost chatbot backend: what one support turn actually costs

Per-turn token math for a SaaS support bot, where caching and batching really pay, the live price lookup, and the EU/US routing caveat nobody prints.

For a support bot inside a small SaaS, the cheapest backend is rarely the one with the lowest headline rate. It’s the one where moving to a cheaper model is a config change instead of a sprint. Infrai serves an OpenAI-compatible endpoint across models from OpenAI, Zhipu, Alibaba, Moonshot and Tencent on a single key, so swapping gpt-5.1 for glm-4-air is one string — and the queue, object storage and error tracking the bot needs around it are already on that same key.

The token arithmetic below is evidence for that argument, not the argument itself. Infrai bills the final price per call and reports it back on every response, so you never have to trust a rate card that someone typed six months ago.

What a turn costs, and how to read today’s number

A support turn is small: a system prompt, three or four turns of history, one question, a short answer. Call it 700 input tokens and 150 output tokens. Ask the API what those tokens cost right now:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/ai/models?capability=chat&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq -r '.data[] | select(.price_input_per_mtok != null)
      | [.id, .owned_by, .price_input_per_mtok, .price_output_per_mtok] | @tsv'

Reading from that endpoint on 2026-07-25, and doing the 700/150 multiplication by hand:

At 50,000 turns a month that’s $3 on glm-4-air against $119 on gpt-5.1. Same key, same call shape, forty-fold difference — which is why the ability to switch matters more than any one row.

Rates move, and they tend to move down; discount campaigns on China-origin models run often. So treat the table as a reading with a date on it, re-run the lookup on the day you decide, and expect to find something cheaper rather than dearer.

Caching and batching, priced before you ship

POST /v1/ai/cost/estimate prices a request shape without running it, and it models the two levers this query asks about — cache reuse and batch mode:

curl -sS -X POST "https://api.infrai.cc/v1/ai/cost/estimate" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role":"user","content":"Why was I charged twice for invoice 4411?"}],
    "model": "openai/gpt-4o-mini",
    "expected_output_tokens": 120,
    "cache_strategy": "infrai",
    "batch_mode": true
  }'

{
  "ok": true,
  "data": {
    "model": "openai/gpt-4o-mini",
    "vendor": "openai",
    "vendor_region": "western",
    "prompt_tokens": 22,
    "expected_output_tokens": 120,
    "breakdown": {
      "currency": "USD",
      "input_cost": 3.3e-06,
      "output_cost": 7.2e-05,
      "cache_discount": 2.259e-05,
      "batch_discount": 3.765e-05,
      "final": 1.883e-05
    }
  }
}

Cache reuse takes about 30% off the modelled base and batch mode about half, and the two stack. Both are estimates rather than promises — actual cache behaviour depends on whether the prompt prefix really repeats.

One caveat is easy to trip over: the estimator resolves vendor-prefixed model ids from its own price catalogue (openai/gpt-4o-mini, deepseek/deepseek-chat, qwen/qwen-max), and a bare id straight out of GET /v1/ai/models comes back as MODEL_NOT_FOUND rather than a wrong price. Annoying, but the right failure — a silent fallback would quietly misprice your whole plan.

The turn itself, in Node 22

import OpenAI from "openai";

const infrai = new OpenAI({
  baseURL: "https://api.infrai.cc/v1",
  apiKey: process.env.INFRAI_API_KEY,
});

const SYSTEM = "You are the support assistant for Acme Analytics. Answer in under 80 words. If you're unsure, say so and offer to open a ticket.";

export async function answerTurn({ tenantId, history, question }) {
  try {
    const turn = await infrai.chat.completions.create({
      model: "glm-4-air",
      max_tokens: 220,
      messages: [{ role: "system", content: SYSTEM }, ...history, { role: "user", content: question }],
    });
    const meta = turn.infrai ?? {};
    return {
      tenantId,
      reply: turn.choices[0].message.content,
      costUsd: meta.cost_usd ?? 0,
      vendor: meta.vendor,
      region: meta.region,
      servedFromCache: meta.cache === true,
      usage: turn.usage,
    };
  } catch (err) {
    if (err.status === 429 || err.status === 503) return { tenantId, reply: null, retryable: true };
    throw err;
  }
}

That infrai object is the part worth wiring into your own metrics on day one:

{
  "id": "chatcmpl-d88a16c15c324203b6110de5",
  "object": "chat.completion",
  "model": "glm-4-flash",
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "ok" }, "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 12, "completion_tokens": 3, "total_tokens": 15 },
  "infrai": { "cost_usd": 0.0, "vendor": "zhipu", "region": "china", "model": "glm-4-flash", "cache": false, "request_id": "req_d88a16c15c324203b6110de5" }
}

Store cost_usd against tenantId and per-tenant attribution stops being a reconciliation project.

Push the non-live work into a batch

Nightly conversation summaries, backlog re-tagging, evaluation runs against yesterday’s transcripts — none of that needs a human waiting, so it belongs in POST /v1/ai/batch/submit:

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/submit" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"model":"glm-4-flash","messages":[{"role":"user","content":"Summarize in one sentence: the export button does nothing on Safari."}]},
      {"model":"glm-4-flash","messages":[{"role":"user","content":"Summarize in one sentence: invoice 4411 was charged twice."}]}
    ],
    "batch_timeout": "6h",
    "metadata": {"job":"nightly-summaries","tenant":"tenant_42"},
    "store": true
  }'

One thing to get right: "model": "auto" is fine on a live chat call but not inside a batch item, where the vendor sees the literal string and rejects it. Pin a served id.

The body keys are exactly requests, batch_timeout, metadata and store. Submit answers with batch_id, state and total_count; then you poll:

const key = process.env.INFRAI_API_KEY;
const headers = { Authorization: `Bearer ${key}` };

export async function drainBatch(batchId) {
  for (let i = 0; i < 60; i++) {
    const st = await fetch(`https://api.infrai.cc/v1/ai/batch/status/${batchId}`, { headers });
    const { data } = await st.json();
    if (["completed", "failed", "expired", "cancelled"].includes(data.state)) break;
    await new Promise((r) => setTimeout(r, 5000));
  }
  const res = await fetch(`https://api.infrai.cc/v1/ai/batch/results/${batchId}`, { headers });
  const { data } = await res.json();
  return data.items.filter((item) => item.ok).map((item) => item.result.content);
}

Submitting a batch is billable at $0.001 per call while status, results and list are free, and new accounts start with $2 of credit. Confirm both from the manifest rather than from me:

curl -sS "https://api.infrai.cc/v1/discovery" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '.capabilities[] | select(.id | startswith("ai.batch")) | {id, billing: .billing.unit, free: .billing.free}'

Where the request actually runs

Every response tells you the vendor and its region. The cheap end of the catalogue is China-origin, which means a European request takes an extra transcontinental hop and first-token latency is worse than a same-region vendor would give you. If your buyers ask for contractual EU-only processing, you’d be better off pinning Azure OpenAI or Bedrock in-region and paying the difference; Infrai doesn’t sell a residency guarantee, and pretending otherwise would be a bad trade for both of us.

For everything else, the honest split looks like this.

What it doesn’t do

Batch runs chat requests only — image or speech work has to fan out across the live endpoints yourself. There’s no server-side conversation store either, so history is your database’s problem, which is fine because your database is where it belongs.

And watch the spend rather than assuming it:

curl -sS "https://api.infrai.cc/v1/account/usage" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" | jq '.data.breakdown'

If a support bot is the only AI you’ll ever ship, buy the cheapest chat API you can find and skip the rest of this. If it’s the first of six features, the second one is already paid for.

References

• Infrai AI runtime API reference
• Infrai OpenAI-compatible surface
• MODEL_NOT_FOUND error reference
• OpenAI prompt caching guide
• OpenRouter documentation