tiktoken or a token-count endpoint? Three numbers for one prompt

We counted the same two messages three ways and got 26, 33 and 24. Which number to trust, when to call Infrai's counter, and the silent zero that catches people.

Trust neither as a billing oracle. Run one prompt through a local tokenizer, through Infrai’s POST /v1/ai/tokens/count, and then through an actual completion, and you get three different integers — we measured 26, 33 and 24 for the same two messages on 26 July 2026. Only the third one, the usage block the vendor returns after the work is done, is the number you’re charged on.

That doesn’t make the other two useless. It makes them tools for different jobs, and picking the wrong one is how a “cost guard” ends up rejecting valid requests or waving through prompts that blow a context window. Here’s the measurement and the rule we’d apply.

The experiment

Two messages, nothing exotic:

[
  { "role": "system", "content": "You are a terse assistant." },
  { "role": "user", "content": "Summarise the quarterly report in three bullet points." }
]

A local o200k_base tokenizer — the encoding the GPT-4o family uses — puts the raw content at 17 tokens. Add OpenAI’s documented chat overhead of 3 tokens per message plus 3 for priming and you land at 26. That’s the number tiktoken gives you, and for an OpenAI model it’s close enough to plan with.

Now ask the platform:

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS https://api.infrai.cc/v1/ai/tokens/count \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [
          {"role": "system", "content": "You are a terse assistant."},
          {"role": "user", "content": "Summarise the quarterly report in three bullet points."}
        ]
      }'

{
  "ok": true,
  "data": { "prompt_tokens": 33, "model": "gpt-4o-mini" },
  "metadata": { "latency_ms": 4, "cost_usd": 0.0 }
}

Thirty-three. Then send the same messages to a real model and read what the vendor actually counted:

{
  "model": "glm-4-flash",
  "usage": { "prompt_tokens": 24, "completion_tokens": 16, "total_tokens": 40 },
  "infrai": { "cost_usd": 0, "vendor": "zhipu", "model": "glm-4-flash" }
}

Twenty-four. The server-side estimate was 37% high against the model that served the request.

Why the endpoint runs high, and why that’s fine

Here’s the finding that decides the whole question: the count doesn’t change with the model.

We asked for gpt-4o-mini, openai/gpt-4o, deepseek/deepseek-chat, glm-4-flash, qwen3.7-plus and auto. Every one came back 33, with the model field echoing whatever string we sent. It’s one estimator with a conservative margin, not a rack of per-vendor tokenizers — which is a sensible design for a gateway that routes across six vendors, and a trap if you read the field name literally.

So the endpoint is a guardrail, not a meter. Use it to answer “is this prompt roughly within budget before I spend a second on it”, and let it run high, because a guard that errs generous is the wrong kind of wrong.

There’s a sharper trap in the same route. It only reads messages:

curl -sS https://api.infrai.cc/v1/ai/tokens/count \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini", "text": "Summarise the quarterly report in three bullet points."}'

That returns {"prompt_tokens": 0, "model": "gpt-4o-mini"} with ok: true. Not a 400, not a warning — a zero. An empty body does the same thing and defaults the model to openai/gpt-4o. If you’re wrapping this route, assert that prompt_tokens > 0 before you believe it.

The cost estimator inherits the estimate

POST /v1/ai/cost/estimate is free and it does what its name says, but it’s built on the same counter and the same silent-defaults behaviour:

curl -sS https://api.infrai.cc/v1/ai/cost/estimate \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "openai/gpt-4o-mini",
        "messages": [{"role": "user", "content": "Summarise the quarterly report in three bullet points."}],
        "expected_output_tokens": 120
      }'

Two things to know before you wire it into a dashboard. Pass prompt_tokens: 12000 instead of messages and you still get ok: true — scored as prompt_tokens: 0 with expected_output_tokens defaulted to 500. The answer is confidently wrong. And its model vocabulary is much narrower than the router’s: openai/gpt-4o-mini and deepseek/deepseek-chat price fine, while glm-4-flash — a model the chat router serves happily — returns MODEL_NOT_FOUND. The catch is that a server-side estimate can’t price the model your traffic will actually land on when you route with auto.

What we’d actually build

Local tokenizer for the hot path, platform counter for the pre-flight check on unfamiliar text, usage for the ledger.

// npm i gpt-tokenizer
import { encode } from "gpt-tokenizer/model/gpt-4o";

const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("set INFRAI_API_KEY before running this");

const messages = [
  { role: "system", content: "You are a terse assistant." },
  { role: "user", content: "Summarise the quarterly report in three bullet points." },
];

// 1. Local: free, ~microseconds, OpenAI-family encoding only.
const local = messages.reduce((n, m) => n + encode(m.content).length + 3, 3);

// 2. Platform guardrail: one network hop, deliberately conservative.
const counted = await fetch("https://api.infrai.cc/v1/ai/tokens/count", {
  method: "POST",
  headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
  body: JSON.stringify({ model: "auto", messages }),
}).then((r) => r.json());

if (!counted.ok || counted.data.prompt_tokens === 0) {
  throw new Error("token count came back empty — check you sent `messages`, not `text`");
}

// 3. Truth: what the vendor billed.
const completion = await fetch("https://api.infrai.cc/v1/chat/completions", {
  method: "POST",
  headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
  body: JSON.stringify({ model: "auto", messages, max_tokens: 16 }),
}).then((r) => r.json());

console.log({
  local,
  guardrail: counted.data.prompt_tokens,
  billed: completion.usage?.prompt_tokens,
  served_by: completion.infrai?.vendor,
});

The counter is free and rate-limited, and it doesn’t draw down a new account’s trial credit, so calling it on every request is affordable. It’s still a network hop, and at roughly 4 ms of server time plus your round trip, a local tokenizer wins on any path where latency matters.

Choosing between them

If every model you touch is OpenAI’s, use tiktoken locally and read usage afterwards — the round trip buys you nothing and OpenAI publishes the encoding. Anthropic’s dedicated token-counting endpoint is the better reference if Claude is your primary model, because it is genuinely model-aware in a way a cross-vendor estimator can’t be. The reason to call Infrai’s counter is the case in between: you route across vendors, you don’t want a tokenizer per vendor in your dependency tree, and you need a fast conservative bound. Then reconcile against GET /v1/account/usage, which reports real spend per capability on the same key that made the calls.

References

• Infrai AI Runtime API reference
• Infrai OpenAI-compatible surface
• How to count tokens with tiktoken — OpenAI Cookbook