Multi-Tenant RAG for B2B Products

The question every B2B buyer asks first

When a B2B product adds an AI assistant over customer documents, the first serious question from a buyer's security team is not about answer quality. It is: can one customer's data ever show up in another customer's answer? Multi-tenant RAG has to make the honest answer "no, and here is the mechanism", not "the prompt tells the model not to".

Retrieval-augmented generation pulls passages from your data into the model's context. If the retrieval step can see passages from the wrong tenant, nothing after it reliably stops them from reaching the answer. Isolation is a retrieval and data-model problem, not a prompting problem.

Filter by tenant where the data lives

The core rule: every retrieval query is filtered by tenant in the database, using an identifier that comes from the authenticated session, never from the user's message or the model's output.

With Postgres and pgvector this means a tenant_id column on the chunks table and a WHERE tenant_id = %s on every similarity query, applied by the service layer, not left to each caller to remember. Putting vectors in the same database as the rest of the product data is what makes this simple: the tenant filter is the same kind of filter the application already applies everywhere else.

Two refinements matter in practice:

• Defence in depth. Postgres row-level security can enforce the tenant filter as a second layer behind the application code. It has caveats: table owners bypass it unless FORCE ROW LEVEL SECURITY is set, superusers and roles with BYPASSRLS always bypass it, and the current tenant has to be set per request, which needs care with connection pooling. Treat it as a backstop, not a replacement for filtering in the service layer.
• Filtered vector search returns fewer results than you asked for. With an approximate index such as HNSW, a selective filter can be applied after the index scan, so a top-10 query for a small tenant may come back with three results. Recent pgvector versions add iterative index scans to address this; check your version, and for very large tenants consider partial indexes or partitioning by tenant.

Permissions inside a tenant

Tenant isolation is the floor, not the ceiling. Inside one customer, not every user can see every document. An assistant that answers from the whole tenant's corpus will happily quote the board pack to an intern.

The fix is to carry the source system's permissions into the index: each chunk stores which groups or users may read the document it came from, and retrieval filters on the current user's access the same way it filters on tenant. Two hard parts:

• Keeping permissions fresh. When a document's sharing changes in the source system, the index must change too. Sync permissions on the same schedule as content, or better, on change events from the source.
• Derived content. Summaries or cached answers built from several documents inherit the strictest permissions of their sources. If that is too complex to track, do not cache across documents.

Everything around retrieval leaks too

Teams that get retrieval right sometimes leak through the side doors:

• Caches. A semantic cache of "similar question, reuse the answer" must be keyed by tenant and, where permissions differ, by user scope. A shared cache is a cross-tenant leak with extra steps.
• Logs and traces. Prompts and retrieved passages in observability tools contain customer data. Store them with the same tenant tagging and retention rules as the source data, and restrict who can read them.
• Evaluation sets. Test questions built from one customer's documents stay with that customer. Shared evaluation data should be synthetic or come from your own content.
• Agent tools. If the assistant can call APIs, every tool call carries the tenant and user from the session, and the API enforces it. The model never chooses which tenant to act on.

Deletion has to reach the index

When a customer deletes a document or leaves the product, their content must disappear from the vector index, the caches, and the logs on the same timeline as from the primary database. Store chunks with a reference to the source document so deletion is a query, not a re-index of everything, and include the index in whatever process you already use to prove deletion to customers.

Where this stops being right

• Single-tenant deployments. If each large customer gets its own database or its own deployment, tenant filtering in queries matters less and operational cost matters more. That trade is common for enterprise customers with strict contractual isolation.
• Very large corpora per tenant. Once one tenant's vectors dominate the index, a shared table with filters stops being the efficient layout. Separate indexes or a dedicated vector database per large tenant can make sense; the isolation rules above stay the same.
• Public or shared knowledge. Product documentation that every customer may see belongs in a shared, clearly marked corpus that retrieval adds alongside the tenant's own data, never mixed into a tenant's rows.

FAQ

Can the prompt enforce tenant isolation? No. The prompt can shape answers, but only retrieval filters in the data layer guarantee the model never sees another tenant's content.

Does each tenant need its own embedding model? Usually not. One embedding model with tenant-filtered retrieval is the common setup. Separate models make sense only when tenants work in different languages or domains.

Where should permissions be checked: at indexing or at query time? Both. Store permissions on each chunk when indexing, and filter by the current user's access at query time, because access changes after a document was indexed.

Our semantic catalog search runs retrieval on Postgres with pgvector; tenant filtering builds on the same layout. More on how we add retrieval and assistants to a product under AI integration services.