Does a rerank stage actually fix a noisy RAG top-20?

Measured on a live Infrai account: what a rerank hop after vector search changes, where it promoted the right chunk, where it made things worse, and the score rule that fixed it.

Short answer: yes, but not for the reason most rerank write-ups give. The reordering isn’t what rescues a noisy top-20 — the permission to retrieve wider is. On Infrai that’s two calls, POST /v1/vector/query and then POST /v1/ai/rerank, and across the eight questions we measured, 14 of the 40 final top-5 slots went to chunks the vector search had ranked between 6th and 17th.

Those fourteen chunks were invisible to a plain top-5 retrieval, and no amount of prompt tuning would have found them. That is the entire effect, and it’s worth about 600 milliseconds and a hundredth of a cent per question.

The setup we measured against

Forty chunks of platform documentation — key rotation, deploy strategy, backups, retry policy, caching — indexed into a collection called rr_platform_kb, embedded through the OpenAI-compatible surface at 1024 dimensions, then queried with eight support-style questions on 26 July 2026.

Embedding first. One detail to design around: the embeddings route takes up to ten strings per input array, so an ingest loop over a real corpus has to chunk in tens rather than posting the whole file list.

curl -s https://api.infrai.cc/v1/embeddings \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
  -H 'content-type: application/json' \
  -d '{"model":"auto","input":"How do I rotate an API key without downtime?"}' \
  > question.json

jq '{model, dims: (.data[0].embedding | length), usage}' question.json

That returns text-embedding-v4 at 1024 dimensions, routed to alibaba_intl. Leaving model as "auto" is the low-friction choice: it resolves to whichever embedding model the platform is currently serving, so you don’t have to track the catalogue yourself. Pin an id only if you have a reason to, and take it from GET /v1/ai/models.

Now the retrieval, asking for twenty rather than five:

jq -c '{collection: "rr_platform_kb", embedding: .data[0].embedding, top_k: 20, include_metadata: true}' \
  question.json > query-body.json

curl -s -X POST https://api.infrai.cc/v1/vector/query \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
  -H 'content-type: application/json' \
  --data @query-body.json > matches.json

jq '[.data.matches[] | {id, score}] | .[0:5]' matches.json

The first five, cosine, unreranked:

[
  { "id": "kb-000", "score": 0.81634518 },
  { "id": "kb-001", "score": 0.62226062 },
  { "id": "kb-004", "score": 0.60130807 },
  { "id": "kb-009", "score": 0.59086085 },
  { "id": "kb-002", "score": 0.55610731 }
]

kb-000 is the rotation procedure and is obviously right. kb-001 (“API keys are created in the dashboard under Settings”) is obviously wrong, and it’s sitting in second place because it shares almost every noun with the question. That’s the failure mode: a bi-encoder compares two summaries of meaning, so topical overlap and answerhood look alike to it.

What the rerank hop changed

The reranker takes the query and the candidate strings together, so it can score whether a chunk answers rather than whether it matches. The request schema is small: query and candidates are required, and top_k trims the response — it defaults to 10, so a 40-candidate call without it hands you ten rows.

curl -s -X POST https://api.infrai.cc/v1/ai/rerank \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "query": "How do I rotate an API key without downtime?",
    "candidates": [
      "Rotating an API key: create the new key first, deploy it to every worker, then revoke the old one.",
      "API keys are created in the dashboard under Settings. Each key belongs to exactly one project.",
      "Zero-downtime deploys use a rolling restart: the load balancer drains connections from one replica at a time.",
      "Rate limits are enforced per project per minute. Exceeding them returns 429 with a Retry-After header."
    ],
    "top_k": 3
  }'

{
  "ok": true,
  "data": {
    "ranked": [
      { "index": 0, "score": 0.912478 },
      { "index": 2, "score": 0.545999 },
      { "index": 3, "score": 0.458231 }
    ]
  }
}

Only index and score come back. No text, no ids — you keep your own candidate array and join on position, which is fine right up until someone inserts a filtering step between the two calls and shifts every index by one.

On the full twenty-candidate run, the reranked top five became kb-000, kb-003, kb-037, kb-009, kb-004. kb-003 is the rolling-restart chunk, and it had been sitting at rank 9 — the only chunk in the corpus that talks about doing anything without downtime. Both of the confident-looking impostors, kb-001 and kb-002, were dropped.

The question where reranking made it worse

“How long are deleted records recoverable?” is where our run fell over. Vector search returned the soft-delete window, the nightly purge and the backup retention — three genuinely relevant chunks. The reranker kept the first, kept the third, and then promoted idempotency-key expiry, postmortem deadlines and cache TTLs, because all of them answer how long something lasts.

Look at the scores, though: 0.814, then 0.507, then 0.380, 0.361, 0.360. The model was telling us it had one strong answer and four guesses, and we ignored it by asking for a fixed five.

So the rule we’d actually ship isn’t “take the top five”. It’s “keep everything scoring at least 60% of the top score”:

import process from "node:process";

const BASE = "https://api.infrai.cc";
const KEY = process.env.INFRAI_API_KEY;
if (!KEY) throw new Error("set INFRAI_API_KEY");

async function api(path, body) {
  const res = await fetch(`${BASE}${path}`, {
    method: "POST",
    headers: { authorization: `Bearer ${KEY}`, "content-type": "application/json" },
    body: JSON.stringify(body),
  });
  const json = await res.json();
  if (!res.ok) throw new Error(`${path} -> ${res.status} ${json?.error?.code ?? ""} ${json?.error?.message ?? ""}`);
  return json;
}

export async function retrieve(question, { wide = 20, floor = 0.6 } = {}) {
  const embedded = await api("/v1/embeddings", { model: "auto", input: question });
  const search = await api("/v1/vector/query", {
    collection: "rr_platform_kb",
    embedding: embedded.data[0].embedding,
    top_k: wide,
    include_metadata: true,
  });

  const matches = search.data.matches;
  if (matches.length === 0) return [];

  const reranked = await api("/v1/ai/rerank", {
    query: question,
    candidates: matches.map((m) => m.metadata.text),
    top_k: Math.min(5, matches.length),
  });

  const ranked = reranked.data.ranked;
  const cut = ranked[0].score * floor;
  return ranked
    .filter((r) => r.score >= cut)
    .map((r) => ({ id: matches[r.index].id, score: r.score, text: matches[r.index].metadata.text }));
}

const context = await retrieve("How long are deleted records recoverable?");
console.log(context.map((c) => `${c.id} ${c.score.toFixed(3)}`).join("\n"));

Across all eight questions that cut kept 18 chunks instead of 40 — a mean of 2.25 per question. On the failing question it dropped all three bad promotions. On questions whose answer genuinely spans several chunks it kept five. Fewer tokens into the model and less noise, from one line of arithmetic.

What this costs, and where it doesn’t pay

POST /v1/vector/query bills $0.0002 per call and POST /v1/ai/rerank metered at $0.0001 per request in the response metadata.cost_usd, flat at 3, 10, 20 and 40 candidates — the catalogue’s own estimate for rerank is higher, at roughly $0.00105, and marked approximate, so read the live figure rather than either number here. New accounts get $2 in credit. Rates on this platform move down over time and discount campaigns run, so what you find today may well be lower:

curl -s https://api.infrai.cc/v1/discovery \
  -H "Authorization: Bearer $INFRAI_API_KEY" \
| jq '[.capabilities[] | select(.id == "vector.query" or .id == "ai.rerank") | {id, billing}]'

The durable point isn’t the rate, it’s that both hops sit on one credential. The same key already reaches storage for the source documents, cron for the reindex job and error tracking for the failures — so adding a rerank stage is one more call on an account you already have, not a fourth vendor with its own SDK and invoice.

Now the honest boundaries. If your vector store is Qdrant or pgvector and your bottleneck is filtered search over tenant metadata, a rerank hop won’t help — you have a recall problem in the filter, not an ordering problem. If you need a specific cross-encoder checkpoint frozen at a known version for a regulated evaluation, you’d be better off running the model yourself: this route chooses a served vendor for you unless you set vendor or model, and the set of vendors it can choose from grows over time. And the trade-off nobody advertises: reranking cannot promote a chunk your retriever never returned, so if the answer sits at vector rank 40 and you fetch 20, the extra call buys nothing at all.

Verify your own index before trusting any of this:

curl -s "https://api.infrai.cc/v1/vector/collection/get?collection=rr_platform_kb" \
  -H "Authorization: Bearer $INFRAI_API_KEY" | jq '.data'

If that returns a vector_count you don’t recognise, your ingest is the problem and no reranker will paper over it.

References

• Infrai vector API reference
• Infrai AI runtime API reference
• Reranking for Better Search — Qdrant