Tuning Top-N: How Many Candidates Should Go Into the Re-Ranker?

The knob between retrieval and ranking

A two-stage search pipeline retrieves the top-N nearest candidates, then a re-ranker scores each query-product pair jointly and returns the final top-K — usually 10. Re-ranker top-N tuning is choosing that N, and it's a real trade-off with a right answer you can measure.

The two failure modes

N too small. Say N is 20. Fast and cheap, but if a relevant product was ranked #150 by the vector search, the re-ranker never sees it — it can only reorder what it's given. If your retrieval isn't precise, a small N locks in a weak set.

N too large. Say N is 500. The cross-encoder's latency and cost scale roughly linearly with N, and most of those 500 candidates are noise that will never make the top 10. You're paying to score results that don't matter.

How to find the right N

Start around 100 to 200. Then measure recall@N against your evaluation set — the share of "should be in the top 10" results that appear anywhere in the retrieved top-N. Increase N until recall@N stops improving, and stop there. Past that point you're adding latency and cost for candidates the re-ranker will discard anyway.

The latency budget

Each re-rank round-trip is tens to a couple of hundred milliseconds, and N drives how much data goes into that call. On our semantic search project the re-rank step runs over a filtered top-N before serving — and even with that extra hop, total query latency dropped, because the previous search was an unindexed full-table scan. What you're replacing matters as much as what N costs.

The interaction with hybrid search

If you fuse lexical and vector candidates before re-ranking, N is the size of the fused set going into the re-ranker. Same tuning logic — grow N until recall@N plateaus.

recall@N in practice

Build an evaluation set of around 50 queries with a known-good result for each. For every query, run the vector search and note the rank it gives the known-good product. Recall@100 is the share of those known-good products that land at rank 100 or better. If recall@100 is 98%, then N of 100 to 150 is plenty — the re-ranker gets everything it needs. If recall@100 is 80%, you have a choice: push N higher, or fix the retrieval — a bigger N papering over a weak embedding or a thin composed field is an expensive way to avoid the real problem.

The cost side of a large N

If your re-ranker is priced per document scored, cost is roughly N times query volume. At N of 200 and a million searches a month, that's 200 million documents scored monthly. Know that number before you turn N up — the latency you can see in testing, but the bill arrives later.

Where this stops being right

• A small, clean catalogue where the vector search is already precise may hit recall@50 near 100% — no need for a large N.
• A re-ranker priced per document makes large N expensive fast — check the pricing model before you tune upward.
• No evaluation set means you're tuning N blind — build one first, or you're guessing.

Where teams get it wrong

Two anti-patterns. Setting N to 10 — the re-ranker then has nothing to work with, it can only reorder the exact ten the vector search already picked, so the re-ranker adds latency for almost no gain. And setting N to 1,000 "to be safe" — you're paying the cross-encoder cost on 990 candidates that are noise, for a top 10 that was decided by the first 150. The right instinct is: as small as recall@N allows, and no smaller.

FAQ

What's a good starting N? 100 to 200 candidates into the re-ranker, then tune against recall@N on your evaluation set.

Does a bigger N always mean better results? Only up to the point where recall@N plateaus. Past that you're paying latency and cost for noise.

How does N relate to the final result count? N is the input to the re-ranker; K — usually 10 — is what it returns. N is much larger than K.

ISTRALLEN tunes the retrieval-to-re-rank hand-off against recall and the latency budget, not a default; see AI for Retail.