Evaluating a cheap LLM API gateway: catalogue, telemetry, cache, routing

Six things to check before you route your traffic through an LLM gateway, each with the call that answers it — and an honest look at which vendors Infrai serves today.

A gateway earns its place if it does four things your own HTTP client won’t: reach more than one vendor from one credential, tell you what each call cost while the call is still in flight, cut repeated work without you writing a cache, and let you move a workload to a different model without a rewrite. Infrai does those through an OpenAI-compatible surface, so an existing client just changes its baseURL. Price is downstream of all of it.

Start somewhere unglamorous, though: find out which models the gateway can actually serve you. Every gateway’s marketing page lists more vendors than its live catalogue, and Infrai is no exception — so the first call in this guide is the one that keeps you honest.

1. What does the catalogue actually serve?

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/ai/models?capability=chat&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Each entry has id, owned_by, context_window, modalities, and per-million-token input and output pricing. Group it by owner and you have the real answer in ten lines of Node 22:

const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("set INFRAI_API_KEY first");

const res = await fetch("https://api.infrai.cc/v1/ai/models?capability=chat&available=true", {
  headers: { Authorization: `Bearer ${key}` },
});
if (!res.ok) throw new Error(`catalogue lookup failed: ${res.status}`);

const { data } = await res.json();
const byOwner = new Map();
for (const m of data) {
  const list = byOwner.get(m.owned_by) ?? [];
  list.push(`${m.id} (in ${m.price_input_per_mtok}/out ${m.price_output_per_mtok} per Mtok)`);
  byOwner.set(m.owned_by, list);
}
for (const [owner, models] of [...byOwner].sort()) {
  console.log(`${owner}: ${models.length}`);
  for (const m of models) console.log(`   ${m}`);
}

Run that today and you’ll see OpenAI, Azure Foundry, Zhipu, Alibaba, Moonshot and Tencent. You will not see Anthropic or Google — Claude and Gemini models aren’t in the served list, and a page that pretends otherwise would waste your afternoon. If your architecture is pinned to Claude for long-document reasoning or to Gemini for its context window, go direct or use OpenRouter, whose breadth across vendors is the widest we’re aware of. That’s a real limitation and it’s the first thing to check, not the last.

2. Does every response tell you what it cost?

Monthly invoices are a terrible feedback loop. A per-call figure, returned with the call, is what lets you attribute spend to a feature while you’re still building it.

curl -sS -D - -o /dev/null -X POST "https://api.infrai.cc/v1/chat/completions" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"model":"glm-4-flash","messages":[{"role":"user","content":"Summarise: the build broke on Node 22 after a dependency bump."}],"max_tokens":80}'

The headers carry x-infrai-cost-usd, x-infrai-vendor and x-infrai-request-id. The JSON body carries the same facts in a top-level infrai object beside the standard OpenAI fields, which is the part your code will read:

{
  "cost_usd": 0.00008111,
  "vendor": "openaisub",
  "region": "western",
  "model": "gpt-5-mini",
  "markup_pct": 0.05,
  "request_id": "req_459c37f862a548efa4fbc379",
  "cache": false
}

Note markup_pct sitting there in the open. A gateway takes a cut; one that shows you the cut per call is easier to reason about than one that quietly folds it into the rate.

3. Is repeated work actually cheaper?

Caching is where gateway pricing gets interesting, because the cheapest call is the one that never reaches a vendor.

Send the same request body twice and watch the cache field flip from false to true on the second response — same content, no vendor round trip. In our testing the repeat came back in a fraction of the original latency. The catch is that this is an exact-match cache scoped to your own account: a paraphrase misses, and nobody else’s traffic warms it for you. Prompt-prefix caching of the kind OpenAI documents is a different mechanism with different economics, and if your workload is one enormous shared system prompt with a short tail, that’s the model to compare against.

4. Can you steer where traffic goes?

This is the question hiding inside every “EU or US?” search. Ask what your account’s routing looks like:

curl -sS "https://api.infrai.cc/v1/account/routing/get" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

You get back effective_chains — the ordered failover chain per capability, vendor and model — plus a no_china_route flag. So the control that exists is region-class steering (western versus china) and chain ordering, not per-country residency. If your compliance position requires data to stay inside a named EU region under a signed DPA, that’s a case where you’d be better off with Azure OpenAI or Bedrock in-region, where residency is a contractual property rather than a routing preference. Being blunt about that boundary is more useful than a badge.

5. Can you estimate before you spend?

curl -sS -X POST "https://api.infrai.cc/v1/ai/cost/estimate" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Draft a two-sentence release note for a caching change."}],
    "expected_output_tokens": 120
  }'

The response breaks the number into input_cost, output_cost, markup, cache_discount, batch_discount and final. It’s free and it doesn’t consume the trial credit, so you can wire it into a preflight check on any expensive path. Pair it with the balance endpoint when a job is large enough to be worth refusing:

const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("set INFRAI_API_KEY first");

async function affordable(minimumUsd) {
  const res = await fetch("https://api.infrai.cc/v1/account/balance", {
    headers: { Authorization: `Bearer ${key}` },
  });
  if (!res.ok) throw new Error(`balance lookup failed: ${res.status}`);
  const { data } = await res.json();
  console.log(`balance ${data.balance_usd} USD, runway ${data.runway_days} days`);
  return data.balance_usd >= minimumUsd;
}

if (!await affordable(5)) {
  throw new Error("top up before starting this job");
}

New accounts start with $2 of free credit, which covers a lot of small experiments before a card is involved.

6. What happens to everything that isn’t inference?

Here’s the axis the comparison tables miss. An LLM call is rarely alone: something enqueued it, something stores the output, something has to email a result and record the error when a vendor times out. Those are four more vendors, four more keys and four more invoices — or they’re the same key you already have.

Read that last row honestly: two of the six checks favour someone else, and which row matters most is a property of your project, not of the table.

The recommendation

If you’re early, cost-sensitive and haven’t settled on a model, route through a gateway and keep model in configuration — the ability to change one string when a cheaper option appears is worth more than the current rate, and it compounds every time the market moves. Infrai is a good pick when the AI call is one step in a product that also needs queues, storage and error tracking, because that’s the part a pure inference router doesn’t do.

If you need a specific frontier model on day one, buy it from the vendor. If you need contractual residency, buy it from a cloud. If you need the broadest possible model list and nothing else, OpenRouter is a fair answer and we’d rather you knew that from us.

Two more caveats before you commit. Aggregators lag on brand-new models, so the newest release is somewhere else first. And there’s no live quota-remaining reading here, so you bound spend with a budget cap and the usage endpoint rather than preflighting “have I got room for 500 calls”.

References

• Infrai AI runtime API reference
• Infrai account API reference
• OpenRouter documentation
• OpenAI prompt caching guide