OpenAI-compatible gateway: what a baseURL swap buys you, and what it can't

Compatibility is a request shape, not a model list. Coverage, cache behaviour, batch pass-through and per-token cost telemetry, each with the call that proves it.

There’s a trap hiding in the phrase “OpenAI, Claude and Gemini compatible”. Compatibility describes the wire format — one request shape, one response shape, one SDK. Coverage describes which models the thing will actually route to. A gateway can be perfectly OpenAI-compatible and serve no Anthropic or Google models at all, and if you don’t check the second thing before the first, you’ll port your client and then discover the model you came for isn’t there.

Infrai is in exactly that position, so we’ll open with the awkward part rather than bury it: the compat surface is real, the catalogue does not include Claude or Gemini today, and there’s a call below that will tell you whether that’s still true when you read this.

Coverage: one call, before anything else

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/ai/models?capability=chat&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  | jq '{count, owners: ([.data[].owned_by] | group_by(.) | map({owner: .[0], models: length}))}'

{
  "count": 22,
  "owners": [
    { "owner": "alibaba_intl", "models": 6 },
    { "owner": "azure_foundry", "models": 1 },
    { "owner": "moonshot", "models": 3 },
    { "owner": "openai", "models": 5 },
    { "owner": "tencent", "models": 1 },
    { "owner": "zhipu", "models": 6 }
  ]
}

Twenty-two chat models, six owners, read on 2026-07-26. No anthropic, no google. So if your architecture depends on Claude for long-document reasoning or Gemini for its context window, this is the wrong gateway for that leg of the work and OpenRouter — whose catalogue breadth is the widest we’re aware of — is the honest recommendation. Cloudflare’s AI Gateway is the other shape of answer: it proxies to providers you already have accounts with, so coverage is whatever you’ve already bought.

Run that same call once a week in CI. Catalogues change, and a gateway’s marketing page is always ahead of its live routing table.

The swap itself really is one line

If the models you need are in that list, migrating an existing OpenAI client is a base URL and a key.

import OpenAI from "openai";

const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is not set");

const client = new OpenAI({
  apiKey: key,
  baseURL: "https://api.infrai.cc/v1",
});

const completion = await client.chat.completions.create({
  model: "glm-4-air",
  max_tokens: 200,
  messages: [
    { role: "system", content: "You write terse changelog entries." },
    { role: "user", content: "We moved response caching from the edge to the account layer." },
  ],
});

console.log(completion.choices[0].message.content);
console.log(completion.infrai);

Note what didn’t change: no new SDK, no new response parser, no rewrite of your streaming handler. That’s the durable value, and it cuts both ways — leaving is the same one-line change, which is the point.

What compatibility does not give you is provider-native surface. Anthropic’s content-block tool format and Gemini’s safety settings have no home in an OpenAI-shaped request, so a client written against those SDKs isn’t a base-URL swap away from anything. Compat surfaces converge on the intersection of what all providers do, and structured output is where you feel it most: json_object is honoured across the models we tried, json_schema is not enforced on the cheaper ones, so validate on your side regardless of what you asked for.

Cost per token, before you send anything

Two free routes price a call in advance. POST /v1/ai/cost/compare takes several model ids at once, which is the one you want when you’re choosing:

curl -sS -X POST "https://api.infrai.cc/v1/ai/cost/compare" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["openai/gpt-4o", "openai/gpt-4o-mini", "deepseek/deepseek-chat"],
    "messages": [{"role": "user", "content": "Summarise this support thread in three bullets."}],
    "expected_output_tokens": 150
  }'

{
  "ok": true,
  "data": [
    { "model": "deepseek/deepseek-chat", "vendor_region": "china", "prompt_tokens": 20,
      "breakdown": { "input_cost": 0.0000028, "output_cost": 0.000042, "markup": 0.0, "final": 0.0000448 } },
    { "model": "openai/gpt-4o-mini", "vendor_region": "western", "prompt_tokens": 20,
      "breakdown": { "input_cost": 0.000003, "output_cost": 0.00009, "markup": 0.00000465, "final": 0.00009765 } },
    { "model": "openai/gpt-4o", "vendor_region": "western", "prompt_tokens": 20,
      "breakdown": { "input_cost": 0.0001, "output_cost": 0.00225, "markup": 0.0001175, "final": 0.0024675 } }
  ]
}

Read the shape rather than the digits. Markup is itemised instead of folded into the rate — a gateway takes a cut and you should be able to see it. The china-region model carried no markup at all on that reading, and the spread between the cheapest and the dearest row is a factor of fifty-five for the same prompt, which is the entire argument for keeping model in configuration.

Now the drawback, because this endpoint has a sharp one:

curl -sS -X POST "https://api.infrai.cc/v1/ai/cost/estimate" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}],"expected_output_tokens":100}'

{
  "ok": false,
  "error": {
    "code": "MODEL_NOT_FOUND",
    "http_status": 400,
    "message": "model string not recognized.",
    "code_detail": "gpt-5-mini"
  }
}

gpt-5-mini is in the served catalogue and runs fine through chat completions — the estimator just doesn’t know it. In our testing the pricing routes resolved a short fixed list (openai/gpt-4o, openai/gpt-4o-mini, deepseek/deepseek-chat, deepseek/deepseek-reasoner and a couple more) and returned MODEL_NOT_FOUND for everything else, auto included. Treat the estimator as a relative-cost tool for planning, and take your real per-call figure from the infrai object on the response instead. That’s a limitation and we’d rather you hit it here than in a budgeting sprint.

Caching: exact match, your account only

The cheapest call is the one that never reaches a vendor. Send the same body twice and watch the flag flip:

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const body = JSON.stringify({
  model: "glm-4-flash",
  max_tokens: 16,
  messages: [{ role: "user", content: "Reply with the single word: cachetest" }],
});

for (const attempt of [1, 2]) {
  const res = await fetch("https://api.infrai.cc/v1/chat/completions", {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body,
  });
  if (!res.ok) throw new Error(`attempt ${attempt} failed: ${res.status} ${await res.text()}`);
  const json = await res.json();
  console.log(`attempt ${attempt}: cache=${json.infrai.cache} vendor=${json.infrai.vendor} cost=${json.infrai.cost_usd}`);
  console.log(`  header x-infrai-cost-usd: ${res.headers.get("x-infrai-cost-usd")}`);
}

First call cache=false, second cache=true. The catch is what “same” means: it’s an exact-match cache scoped to your own account, so a reworded prompt misses and nobody else’s traffic warms yours. That’s a different mechanism from prompt-prefix caching, and if your workload is one huge shared system prompt with a short user tail, prefix caching at the provider is the thing to compare against — Cloudflare document their own caching layer in detail and it’s worth reading before you assume all “caching” means the same thing.

Batch, and the three things to know about it

Bulk work goes through POST /v1/ai/batch/submit at cost pass-through. It’s genuinely cheap. It also behaves differently from OpenAI’s file-based batch API in ways that will bite a naive port:

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/submit" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"model":"glm-4-flash","max_tokens":24,"messages":[{"role":"user","content":"Say OK"}]},
      {"model":"glm-4-flash","max_tokens":24,"messages":[{"role":"user","content":"Say FINE"}]}
    ],
    "metadata": {"job": "gateway-demo"},
    "store": true
  }'

{
  "ok": true,
  "data": { "batch_id": "batch_8393d98606fd3e4f47090d7f", "state": "completed", "total_count": 2 }
}

One: the submit blocks while rows run — two rows took about 2.5 seconds — so it’s asynchronous in shape but not in behaviour for small jobs. Two: every row is routed as a chat request, so embedding or image rows fail. Three: the response carries no cost estimate, so budget from POST /v1/ai/cost/compare beforehand and from GET /v1/ai/batch/results/{id} afterwards, where each item has its own cost_usd and vendor.

Choosing

Three of those seven rows go the other way, and which row decides it is a property of your project rather than of the table.

The recommendation, plainly: if you need Claude or Gemini specifically, don’t route them through here. If you’re picking a cheap default for summarisation, classification and extraction — the work that makes up most production LLM traffic — a compat gateway with itemised markup and a cache is a sound choice, and the fact that the same credential also runs your queues, object storage and error tracking is worth more over a year than the difference between two rows in a rate card.

References

• Infrai AI runtime API reference
• MODEL_NOT_FOUND error reference
• OpenRouter FAQ
• Cloudflare AI Gateway caching