Moderating a million comments a month: cheap first pass, human second

A tiered moderation pipeline costed end to end — batch classification, confidence-band escalation and a review queue, with the calls and the monthly arithmetic.

The cheapest workable design for high-volume moderation is tiered: a small model decides the easy 95%, a stronger model sees only what the first one was unsure about, and a human queue receives what neither could settle. Run tier one as an async batch and the per-item cost falls to a rounding error; the expensive resource is human attention, so the whole design is about protecting it. On Infrai every layer of that — classification, escalation, the queue and the usage ledger — is one credential.

One thing to get out of the way first, because it changes the shape of the code. Infrai’s OpenAI-compatible surface includes POST /v1/moderations, but there’s no moderation vendor keyed on the shared pool at present, so that route returns a vendor-not-configured error rather than category scores. Classification through a chat model is the path that works today.

Check before you build on it

export INFRAI_API_KEY="your_infrai_api_key"

curl -sS "https://api.infrai.cc/v1/ai/models?capability=moderation&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

{ "object": "list", "capability": "moderation", "available_only": true, "count": 0, "data": [] }

An empty list is a straight answer. Make that call part of a startup check, not a one-off — the day it returns entries, a purpose-built moderation model will be cheaper and better calibrated than a chat prompt, and you’ll want to move.

Tier one: classify in bulk

Ask for a label and a confidence in JSON. The confidence is what drives everything downstream, so ask for it explicitly rather than inferring it.

curl -sS -X POST "https://api.infrai.cc/v1/ai/batch/submit" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"model":"glm-4-air","response_format":{"type":"json_object"},"max_tokens":80,"messages":[
        {"role":"system","content":"Classify the comment. Reply JSON: {\"label\":\"ok|spam|harassment|sexual|violence\",\"confidence\":0..1}"},
        {"role":"user","content":"Great write-up, saved me an afternoon."}]},
      {"model":"glm-4-air","response_format":{"type":"json_object"},"max_tokens":80,"messages":[
        {"role":"system","content":"Classify the comment. Reply JSON: {\"label\":\"ok|spam|harassment|sexual|violence\",\"confidence\":0..1}"},
        {"role":"user","content":"buy cheap followers now click here"}]}
    ],
    "metadata": {"job": "moderation-tier-1", "window": "2026-07-26T00:00Z"},
    "store": true
  }'

The submit returns batch_id, state and total_count; poll GET /v1/ai/batch/status/{id} until the state settles, then page GET /v1/ai/batch/results/{id} with next_cursor. Every row carries ok, result, cost_usd and request_index — so an item that failed is one index to retry, and the per-row cost tells you which content types are expensive to judge.

Chunk your submits. Several hundred comments per call is comfortable; one submit per million is not a plan.

Tier two: escalate the uncertain, not the flagged

Here’s the piece that decides your bill. A confident ok and a confident spam are both finished — no second opinion needed. Only the middle band moves up.

const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const LOW = 0.55;
const HIGH = 0.9;

async function judge(comment, model) {
  const payload = {
    model,
    response_format: { type: "json_object" },
    max_tokens: 120,
    messages: [
      { role: "system", content: 'Reply JSON: {"label":"ok|spam|harassment|sexual|violence","confidence":0..1,"reason":string}' },
      { role: "user", content: comment },
    ],
  };
  const res = await fetch("https://api.infrai.cc/v1/chat/completions", {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(payload),
  });
  if (!res.ok) throw new Error(`judge failed: ${res.status} ${await res.text()}`);
  const json = await res.json();
  return { ...JSON.parse(json.choices[0].message.content), spend: json.infrai?.cost_usd ?? 0 };
}

export async function route(comment, tierOne) {
  if (tierOne.confidence >= HIGH) return { decision: tierOne.label, by: "tier-1" };
  if (tierOne.confidence < LOW) return { decision: "review", by: "tier-1-unsure" };
  const second = await judge(comment, "gpt-5-mini");
  if (second.confidence >= HIGH) return { decision: second.label, by: "tier-2", reason: second.reason };
  return { decision: "review", by: "tier-2-unsure", reason: second.reason };
}

Tune LOW and HIGH against a labelled sample of your own content, not against intuition. Wide bands are safer and dearer; narrow bands are cheaper and let more mistakes through. Measure the escalation rate for a week before you trust either number — in our testing the band placement mattered far more to total cost than the choice of model did.

The review queue is the same key

Whatever neither tier settles becomes human work, and human work needs a durable queue rather than a database table you poll.

curl -sS -X POST "https://api.infrai.cc/v1/queue/publish" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "queue": "moderation_review",
    "body": {"content_id": "post_8812", "label": "uncertain", "score": 0.51}
  }'

{
  "ok": true,
  "data": {
    "message_id": "qmsg_A48ontMsZp4isVukvKjw56nY",
    "queue": "moderation_review",
    "payload": { "content_id": "post_8812", "label": "uncertain", "score": 0.51 },
    "status": "available",
    "delivery_count": 0
  }
}

A reviewer tool pulls a small page at a time, and the message stays invisible to other reviewers until it’s acknowledged or its visibility window lapses:

curl -sS -X POST "https://api.infrai.cc/v1/queue/consume" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"queue": "moderation_review", "max_messages": 2}'

Acknowledge with POST /v1/queue/ack when a decision is recorded, and watch the backlog with a stats call — depth and oldest-message age are the two numbers a moderation lead actually cares about:

curl -sS "https://api.infrai.cc/v1/queue/stats/moderation_review" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

What a million items a month costs

Assume a 60-token comment against a 120-token system prompt, a 20-token answer, batches of 500, a 4% escalation rate and 0.5% reaching a human. Rates read 2026-07-26 from the catalogue:

curl -sS "https://api.infrai.cc/v1/ai/models?capability=chat&available=true" \
  -H "Authorization: Bearer ${INFRAI_API_KEY}"

Roughly $21 of machine spend against a thousand-odd of human time — which is the entire argument for tiering, and the reason optimising the model choice below tier one is usually wasted effort. The figures move, generally downward, so rerun the catalogue call before you present a budget. Confirm actual spend afterwards with GET /v1/account/usage, which breaks the total out per capability.

Honest limits, and who does this better

This pipeline handles text. It doesn’t do images, audio or video, so a platform with user-uploaded media needs another tool for that surface. There’s no policy-tuned taxonomy either — you’re writing category definitions in a prompt, which is flexible and also means calibration is your job, forever.

If your volume is modest and you’re already an OpenAI customer, their moderation endpoint is purpose-built, free to call and better calibrated than a prompt; use it and skip all of this. Anthropic publishes a solid content-moderation guide for Claude if you want the prompt-engineering version done properly. For regulated categories at very large scale, a dedicated trust-and-safety vendor with human-in-the-loop tooling will beat a DIY pipeline on both quality and audit trail — that’s a genuine trade-off, not a hedge.

What tips it back this way is everything around the classifier. The batch runner, the queue that feeds reviewers, the storage for evidence and the error tracking when a vendor times out are the same account, the same key and one invoice — and per-tenant attribution is a metadata field rather than a reconciliation project.

References

• Infrai AI runtime API reference
• Infrai queue API reference
• OpenAI moderation guide
• Anthropic content moderation guide