Enterprise RAG: Hybrid Search with BM25, Vector Retrieval and Reranking

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding Large Language Models in private data. However, the standard tutorial approach—chunking text, embedding it, and using k-Nearest Neighbors (k-NN) for vector retrieval—often falls apart when faced with complex enterprise data.

In this article, we explore why pure vector retrieval is insufficient and how to implement a robust hybrid search pipeline combining BM25, vector search, and cross-encoder reranking.

The Limits of Pure Vector Retrieval

Dense vector embeddings (like OpenAI's text-embedding-3-small or open-source models) are excellent at capturing semantic meaning. If a user searches for "canine companion," a vector database will correctly retrieve documents containing "dog."

However, vector embeddings struggle with:

1. Exact keyword matches: Searching for an exact error code like ERR_509_AUTH often fails because the semantic representation gets blurred.
2. Proper nouns and acronyms: Niche industry acronyms or specific project names are not well-represented in the embedding model's latent space.
3. Out-of-vocabulary terms: Words the model has never seen cannot be embedded accurately.

Lexical Retrieval and BM25

To solve this, we bring back traditional lexical search. BM25 (Best Matching 25) is a sparse retrieval algorithm based on TF-IDF (Term Frequency-Inverse Document Frequency).

BM25 excels exactly where dense vectors fail: exact keyword matches and rare terminology. It doesn't understand semantics, but it guarantees that if you search for ERR_509_AUTH, you will find documents containing exactly that string.

The Hybrid Search Architecture

A modern enterprise retrieval pipeline uses both.

1. Semantic Chunking

Before indexing, documents must be chunked. Instead of arbitrary fixed-size chunking (e.g., 500 tokens), semantic chunking attempts to keep related thoughts together by splitting at paragraph boundaries or using a small LLM to determine logical breakpoints.

2. Dual Indexing

Each chunk is indexed twice:

• Passed through an embedding model and stored in a vector database (e.g., Pinecone, Qdrant, pgvector).
• Indexed in a sparse search engine (e.g., Elasticsearch, OpenSearch) for BM25 retrieval.

3. Parallel Retrieval and Reciprocal Rank Fusion (RRF)

At query time, the user's query is sent to both the vector database and the BM25 index.

We retrieve the top K results from both (e.g., top 20 from each).

Because BM25 scores (often 10-30) and cosine similarity scores (usually 0.7-1.0) are on completely different scales, we cannot simply add them. Instead, we use Reciprocal Rank Fusion (RRF), which ranks documents based on their position in both lists rather than their absolute scores.

4. Cross-Encoder Reranking

The combined pool of documents (e.g., ~40 chunks) now contains a mix of semantically relevant and keyword-relevant results. However, dual-encoder embedding models are fast but slightly inaccurate, because the query and document are embedded separately.

We pass these top results to a Cross-Encoder reranking model (like Cohere Rerank or BGE-Reranker). A cross-encoder analyzes the query and the document together, providing a highly accurate relevance score.

Because cross-encoders are computationally expensive, we only run them on the top 40 retrieved chunks, not the entire database. The reranker outputs the final top 5 chunks that will be fed into the LLM context window.

Security Boundaries and Metadata

In enterprise environments, data access is rarely uniform. Role-Based Access Control (RBAC) must be respected at the retrieval level.

Both the vector database and the BM25 index must support metadata filtering. When a query is executed, it must be accompanied by the user's access tokens or group IDs. The database pre-filters the permissible documents before calculating similarity, ensuring no unauthorized data leaks into the LLM's context.

Conclusion

The shift from simple vector search to hybrid retrieval with reranking represents the difference between a prototype and a production-grade RAG system. While it introduces more infrastructure—requiring both dense and sparse indexes—the resulting improvement in retrieval accuracy is essential for enterprise deployments.