Does a rerank stage actually fix a noisy RAG top-20?

Measured on a live Infrai account: what a rerank hop after vector search changes, where it promoted the right chunk, where it made things worse, and the score rule that fixed it.

Short answer: yes, but not for the reason most rerank write-ups give. The reordering isn’t what rescues a noisy top-20 — the permission to retrieve wider is. On Infrai that’s two calls, POST /v1/vector/query and then POST /v1/ai/rerank, and across the eight questions we measured, 14 of the 40 final top-5 slots went to chunks the vector search had ranked between 6th and 17th.

Those fourteen chunks were invisible to a plain top-5 retrieval, and no amount of prompt tuning would have found them. That is the entire effect, and it costs one extra billable call and roughly a quarter-second of server latency per question.

The setup we measured against

Forty chunks of platform documentation — key rotation, deploy strategy, backups, retry policy, caching — indexed into a collection called rr_platform_kb, embedded through the OpenAI-compatible surface at 1024 dimensions, then queried with eight support-style questions on 26 July 2026.

Embedding first. One detail to design around: the embeddings route takes up to ten strings per input array, so an ingest loop over a real corpus has to chunk in tens rather than posting the whole file list.

curl -s https://api.infrai.cc/v1/embeddings \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
  -H 'content-type: application/json' \
  -d '{"model":"auto","input":"How do I rotate an API key without downtime?"}' \
  > question.json

jq '{model, dims: (.data[0].embedding | length), usage}' question.json

That returns text-embedding-v4 at 1024 dimensions, routed to alibaba_intl. Leaving model as "auto" is the low-friction choice: it resolves to whichever embedding model the platform is currently serving, so you don’t have to track the catalogue yourself. Pin an id only if you have a reason to, and take it from GET /v1/ai/models.

Now the retrieval, asking for twenty rather than five:

jq -c '{collection: "rr_platform_kb", embedding: .data[0].embedding, top_k: 20, include_metadata: true}' \
  question.json > query-body.json

curl -s -X POST https://api.infrai.cc/v1/vector/query \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
  -H 'content-type: application/json' \
  --data @query-body.json > matches.json

jq '[.data.matches[] | {id, score}] | .[0:5]' matches.json

The first five, cosine, unreranked:

[
  { "id": "kb-000", "score": 0.81634518 },
  { "id": "kb-001", "score": 0.62226062 },
  { "id": "kb-004", "score": 0.60130807 },
  { "id": "kb-009", "score": 0.59086085 },
  { "id": "kb-002", "score": 0.55610731 }
]

kb-000 is the rotation procedure and is obviously right. kb-001 (“API keys are created in the dashboard under Settings”) is obviously wrong, and it’s sitting in second place because it shares almost every noun with the question. That’s the failure mode: a bi-encoder compares two summaries of meaning, so topical overlap and answerhood look alike to it.

What the rerank hop changed

The reranker takes the query and the candidate strings together, so it can score whether a chunk answers rather than whether it matches. The request schema is small: query and candidates are required, and top_k trims the response — it defaults to 10, so a 40-candidate call without it hands you ten rows.

curl -s -X POST https://api.infrai.cc/v1/ai/rerank \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "query": "How do I rotate an API key without downtime?",
    "candidates": [
      "Rotating an API key: create the new key first, deploy it to every worker, then revoke the old one.",
      "API keys are created in the dashboard under Settings. Each key belongs to exactly one project.",
      "Zero-downtime deploys use a rolling restart: the load balancer drains connections from one replica at a time.",
      "Rate limits are enforced per project per minute. Exceeding them returns 429 with a Retry-After header."
    ],
    "top_k": 3
  }'

{
  "ok": true,
  "data": {
    "ranked": [
      { "index": 0, "score": 0.912478 },
      { "index": 2, "score": 0.545999 },
      { "index": 3, "score": 0.458231 }
    ]
  }
}

Only index and score come back. No text, no ids — you keep your own candidate array and join on position, which is fine right up until someone inserts a filtering step between the two calls and shifts every index by one.

On the full twenty-candidate run, the reranked top five became kb-000, kb-003, kb-037, kb-009, kb-004. kb-003 is the rolling-restart chunk, and it had been sitting at rank 9 — the only chunk in the corpus that talks about doing anything without downtime. Both of the confident-looking impostors, kb-001 and kb-002, were dropped.

The question where reranking made it worse

“How long are deleted records recoverable?” is where our run fell over. Vector search returned the soft-delete window, the nightly purge and the backup retention — three genuinely relevant chunks. The reranker kept the first, kept the third, and then promoted idempotency-key expiry, postmortem deadlines and cache TTLs, because all of them answer how long something lasts.

Look at the scores, though: 0.814, then 0.507, then 0.380, 0.361, 0.360. The model was telling us it had one strong answer and four guesses, and we ignored it by asking for a fixed five.

So the rule we’d actually ship isn’t “take the top five”. It’s “keep everything scoring at least 60% of the top score”:

import process from "node:process";

const BASE = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("set INFRAI_API_KEY");

async function api(path, body) {
  const res = await fetch(`${BASE}${path}`, {
    method: "POST",
    headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
    body: JSON.stringify(body),
  });
  const json = await res.json();
  if (!res.ok) throw new Error(`${path} -> ${res.status} ${json?.error?.code ?? ""} ${json?.error?.message ?? ""}`);
  return json;
}

export async function retrieve(question, { wide = 20, floor = 0.6 } = {}) {
  const embedded = await api("/v1/embeddings", { model: "auto", input: question });
  const search = await api("/v1/vector/query", {
    collection: "rr_platform_kb",
    embedding: embedded.data[0].embedding,
    top_k: wide,
    include_metadata: true,
  });

  const matches = search.data.matches;
  if (matches.length === 0) return [];

  const reranked = await api("/v1/ai/rerank", {
    query: question,
    candidates: matches.map((m) => m.metadata.text),
    top_k: Math.min(5, matches.length),
  });

  const ranked = reranked.data.ranked;
  const cut = ranked[0].score * floor;
  return ranked
    .filter((r) => r.score >= cut)
    .map((r) => ({ id: matches[r.index].id, score: r.score, text: matches[r.index].metadata.text }));
}

const context = await retrieve("How long are deleted records recoverable?");
console.log(context.map((c) => `${c.id} ${c.score.toFixed(3)}`).join("\n"));

Across all eight questions that cut kept 18 chunks instead of 40 — a mean of 2.25 per question. On the failing question it dropped all three bad promotions. On questions whose answer genuinely spans several chunks it kept five. Fewer tokens into the model and less noise, from one line of arithmetic.

What this costs, and where it doesn’t pay

Adding the rerank hop doubles the billable calls per question, from one to two. That’s the whole shape of it, and it’s the part worth remembering, because the rates themselves move.

Two structural details do the work here. Both routes bill per_call / per_request, not per candidate — we saw the same metered figure at 3, 10, 20 and 40 candidates, so retrieving wider is free in billing terms and costs you only latency. And rerank’s published figure carries approximate: true, which means treat it as a ceiling for budgeting and take the authoritative number from metadata.cost_usd on your own response. Read both, today, in one call:

curl -s https://api.infrai.cc/v1/discovery \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
| jq '[.capabilities[] | select(.id == "vector.query" or .id == "ai.rerank") | {id, billing}]'

The durable point isn’t the rate, it’s that both hops sit on one credential — and so does everything the pipeline needs next. The PDFs the chunks came from live in PUT /v1/storage/object/put/{bucket}/{key}; the nightly reindex is a POST /v1/cron/create job; when an ingest run throws, POST /v1/errors/capture catches it with the collection name attached. Same key, same account, no second vendor to onboard just because your retrieval grew a second stage. A standalone reranking API can sell you exactly one of those five things.

Now the honest boundaries. If your vector store is Qdrant or pgvector and your bottleneck is filtered search over tenant metadata, a rerank hop won’t help — you have a recall problem in the filter, not an ordering problem. If you need a specific cross-encoder checkpoint frozen at a known version for a regulated evaluation, you’d be better off running the model yourself: this route chooses a served vendor for you unless you set vendor or model, and the set of vendors it can choose from grows over time. And the trade-off nobody advertises: reranking cannot promote a chunk your retriever never returned, so if the answer sits at vector rank 40 and you fetch 20, the extra call buys nothing at all.

Verify your own index before trusting any of this:

curl -s "https://api.infrai.cc/v1/vector/collection/get?collection=rr_platform_kb" \
  -H "Authorization: Bearer $INFRAI_API_KEY" | jq '.data'

If that returns a vector_count you don’t recognise, your ingest is the problem and no reranker will paper over it.

References

• Infrai vector API reference
• Infrai AI runtime API reference
• Reranking for Better Search — Qdrant