How Vector Search and Re-Ranking Work Together in Product Search

Retrieval is stage one of two

"Vector search" often gets described as the whole answer for modern product search. In a catalogue that has to convert, it's the first of two stages. Vector search and re-ranking do different jobs: retrieval casts a wide net quickly, and re-ranking decides which of the catch actually belongs on the first screen. Skip the second stage and the results are topically plausible but not quite right.

Stage 1: vector search

At index time, each product's text — title, key attributes, category, a short description — is turned into an embedding and stored in an approximate-nearest-neighbour index. At search time, the query is embedded the same way, and the index returns the closest N products by vector distance, typically in single-digit to low-double-digit milliseconds at catalogue scale. On our semantic search project this stage runs on a vector index held in the same database as the catalogue itself, so results can respect live price and stock.

What this stage produces is an approximation of relevance. Vector proximity means "these mean roughly similar things," not "this is the best product for this query."

Why stage 1 isn't enough

A bi-encoder — the model behind vector retrieval — embeds the query and each product separately, then compares the results. It never looks at the query and a specific product together. So "close enough" matches float to the top: search "red running shoe" and a red dress shoe sits near the query because "red" and "shoe" dominate the vector, even though it's wrong for the intent. On the first screen of results, that's a lost sale — shoppers don't scroll past a bad top row, they leave.

Stage 2: re-ranking

A re-ranker is a cross-encoder: it takes the top candidates from stage 1 — usually the top 50 to 200 — and scores each query-and-product pair jointly, then reorders them and returns the final top 10. Because it evaluates the pair together, it catches the distinctions the bi-encoder missed.

Independent benchmarks generally put the improvement from cross-encoder re-ranking in the range of several nDCG points over retrieval alone; vendor materials claim more. Treat the conservative range as the expectation. The cost is a second network round-trip — tens to a couple of hundred milliseconds.

The latency and cost budget

The design principle is: retrieve broad but cheap, re-rank narrow but precise.

• Don't re-rank thousands of candidates. The cross-encoder cost is per pair. Feed it a filtered top-N, not the whole retrieval set.
• Don't retrieve only ten. If stage 1 returns a tight set, re-ranking can reorder but can't recover a relevant product that retrieval missed.
• Top-N is the tuning knob. Too small and a bad recall set can't be fixed downstream; too large and you pay latency for candidates that will never rank. Start around 100-200 and tune against conversion.

Counterintuitively, adding re-ranking can still leave total search latency lower than before — in our case it did, because the pipeline replaced an unindexed pattern scan across a large table. Properly indexed vector search plus re-ranking on a filtered candidate set beat a full-table scan even with the extra hop.

Where a second stage isn't worth it

• Low-stakes internal search. A tool where "good enough" results are fine doesn't need to spend the latency and cost on precision.
• Tiny candidate sets. If retrieval naturally returns a dozen items, reorder them with something cheap or not at all.
• A hard latency floor. Type-ahead that must respond under ~150ms may have to ship vector-only and accept the precision hit.
• No relevance signal to tune against. Without click or conversion data, top-N and the threshold are guesswork, and the second stage's value is hard to confirm.

FAQ

Can I re-rank without vector search first? You need a candidate generator, but it doesn't have to be vector search — lexical retrieval works too. Re-ranking is always the second stage on top of whatever produced the candidates.

How big should top-N be? Start at roughly 100-200 candidates into the re-ranker and adjust against conversion. Larger isn't free — it's linear extra cost and latency.

Is a cross-encoder just a bigger embedding model? No. An embedding model encodes each side independently; a cross-encoder scores the query and a candidate together in one pass. That joint view is where both the accuracy gain and the extra cost come from.

ISTRALLEN builds two-stage retrieval-and-re-ranking search tuned to a real catalogue's latency budget; see AI for Retail.