Why my RAG pipeline needed BM25

Ask My Papers answers questions about four research papers on floor plans, using only what the papers say. Because all four share a topic, their chunks sit almost on top of each other in embedding space.

• Vector search alone: nomic-embed-text embeddings in Chroma, top chunks passed to llama3.1:8b.

• The chunk listing one paper’s three contributions ranked #250 by vector search. What tells these papers apart is mostly proper nouns, and embeddings compress exactly those away.
• Once I added keyword search, Reciprocal Rank Fusion buried the same chunk again: it rewards agreement, and a chunk ranked #1 by one method and #250 by the other loses to chunks both rate as mediocre.

• Added BM25 keyword search alongside vector search.
• Gave each method reserved slots before fusion fills the rest.
• Prefixed each chunk with its paper’s title before embedding, so mid-paper chunks that only say “our dataset” still point at their source.

• The key chunk: #250 by vector search, #1 by BM25.
• Adding BM25 turned two hard failures into passes.
• Title prefixes took document-level recall to 100% at every k.
• With reranking on top, retrieval reached 100% hit@6 on 16 labelled questions.

• When documents share a topic, the words that separate them matter more than overall meaning, and keyword search keeps those words.
• Fusion methods reward agreement. A method that is uniquely right needs protecting.