Semantic Search at Marketplace Scale: Millions of SKUs, Thousands of Sellers

The scale where the architecture changes

A retrieval design that is comfortable at a hundred thousand SKUs is not automatically comfortable at ten million. Semantic search at millions of SKUs crosses several thresholds at once — index size, re-embedding throughput, query concurrency, and catalogue churn — and the components that were the right call at mid scale become the bottleneck.

Embeddings in the same Postgres instance as the catalogue is an excellent mid-scale choice, and our semantic search project documents exactly why. Into the multi-million-vector range, a dedicated vector database becomes worth the added operational surface: purpose-built approximate-nearest-neighbour indexes, horizontal sharding, and memory management tuned for vector workloads.

ANN index tuning is now a real job

At small scale, exact or near-exact search is fine. At this scale, you are running approximate nearest neighbour, and the index parameters — graph connectivity, search breadth, build time versus recall — are a tuning problem with real trade-offs. Recall that was a rounding error at 100k SKUs becomes a measurable miss rate at ten million.

Re-embedding needs change data capture

A single trigger-and-queue worker keeps embeddings fresh when the catalogue changes a few thousand times a day. At marketplace churn — new listings, price and stock updates, sellers editing content constantly — that worker falls behind. The pattern shifts to change-data-capture tooling streaming updates into an embedding pipeline that can scale independently.

Sharding and the long tail

Millions of SKUs include a huge long tail that is rarely queried but still has to be searchable. Sharding strategy, tiered storage for cold vectors, and deciding what stays in the hot index are scale-specific decisions that do not exist at mid scale.

Latency under concurrency

Sub-second latency for one query is easy. Sub-second latency at thousands of concurrent queries against a ten-million-vector index, with a reranking pass on top, requires capacity planning, caching of common queries, and sometimes a tiered retrieval approach — a fast coarse pass, then a precise one.

A worked example

A marketplace crosses eight million active listings and search latency starts spiking at peak. The investigation finds three compounding causes: the vector index no longer fits comfortably in memory on one node, the re-embedding worker is hours behind because sellers are editing listings faster than it can process, and popular queries are each doing a full index scan with no cache. The fix is structural — shard the index, move re-embedding onto a change-data-capture stream, and cache hot queries at the edge — not a parameter tweak.

The migration is incremental

None of this has to happen at once. A common path is to add query caching first, then move re-embedding to CDC, then introduce a dedicated vector store for the hot set while the long tail stays in the existing database. Each step buys headroom and can be validated before the next, which matters when the system is already serving live traffic.

Where this stops being right

• If your catalogue is stable and mid-sized, none of this applies — a dedicated vector database here is cost and complexity with no return.
• Marketplace catalogue matching — deciding which of millions of listings are the same product — is a prerequisite data problem that has to be solved alongside retrieval.
• Per-shopper personalization at this scale is a separate recommendation and feature-store system, not a tuning parameter on search.

FAQ

When does semantic search need a dedicated vector database? Around the multi-million-vector range, when index size, sharding needs, and ANN tuning outgrow what a general-purpose database does well. Below that, keeping vectors with the catalogue is usually better.

Why does re-embedding need change data capture at scale? Because a single queue worker cannot keep up with marketplace-rate catalogue changes. Streaming updates through CDC lets the embedding pipeline scale on its own.

What makes latency hard at this scale? Thousands of concurrent queries against a huge approximate index, plus reranking. It needs capacity planning, query caching, and often a coarse-then-precise retrieval approach.

ISTRALLEN builds semantic search that scales from a single database to a sharded vector store and a CDC pipeline as the catalogue grows — see AI for Retail.