Ask My Papers
A learning project: a local RAG over floor-plan research papers that cites its sources and refuses to answer when the evidence is weak.
- Role
- Solo passion project
- Status
- SHIPPED
- Stack
- Python, LangChain, Ollama, Chroma, BM25, FlashRank, FastAPI, Streamlit, pytest
The problem
Can a model running on my laptop answer questions about research papers using only what the papers say, point to the passage it used, and say so when the papers don’t cover the question?
I built this to learn how retrieval-augmented generation actually works: how chunking, embeddings and context decide what a model gets to read. I used floor-plan research papers as the material, and measured every change instead of trusting the answers.
Constraints
- 4 floor-plan research papers: 60 pages, 303 chunks.
- Everything runs locally: Ollama for embeddings (nomic-embed-text) and answers (llama3.1:8b), Chroma in-process, and a 21 MB ONNX reranker. No API keys, no costs.
- 16 labelled questions to score retrieval, plus answer checks and refusal tests.
The system
- 01Ingested 4 papers (60 pages, 303 chunks) with pypdf, stripping hidden LaTeX and repairing 568 words hyphenated across lines.
- 02Hybrid retrieval: BM25 and vector search (nomic-embed-text in Chroma) merged with Reciprocal Rank Fusion. One key chunk ranked #250 by vectors and #1 by BM25.
- 03A FlashRank cross-encoder reranks 20 candidates to the top 6, then a relevance threshold decides whether to answer at all.
- 04llama3.1:8b answers through a grounding prompt. Exposed as a CLI, a FastAPI endpoint and a Streamlit app.
- 01 Problem
Vector search ranked the passage that answered the question #250. All four papers are about floor plans, so their chunks sit almost on top of each other.
- 02 Decision
Add BM25 keyword search next to it. The same passage ranked #1 by keywords.
- 03 System
Merge both lists with Reciprocal Rank Fusion, then let a cross-encoder rerank 20 candidates down to 6.
- 04 Failure
Fusion buried the BM25 #1 result. It rewards agreement, and a chunk ranked #1 by one method and #250 by the other lost to chunks both rated mediocre.
- 05 Iteration
Reserve slots for each method before fusion fills the rest, and put a relevance gate in front of the model: if nothing clears it, the answer is a refusal.
- 06 Result
hit@6 went from 94% to 100% and MRR from 0.677 to 0.783 on 16 labelled questions. Out-of-scope questions are refused in about a second.
Engineering decisions
- Keyword search next to vector search
- All four papers are about floor plans, so their embeddings sit close together and the proper nouns that tell them apart get lost. One key chunk ranked #250 by vector search and #1 by BM25.
- A cross-encoder reranker
- hit@10 was already 100% while hit@6 was 94%: the right chunk was being found but not ranked high enough to reach the prompt. That gap is exactly what reranking fixes.
- Paper titles added to each chunk before embedding
- A chunk from the middle of a paper says “our dataset” and never names it. Prefixing the title took document-level recall to 100% at every k.
- A relevance cut-off before the model is called
- Reranker scores separate cleanly: relevant chunks scored 0.85 to 1.00, irrelevant ones 0.0000. If nothing clears 0.05, the model is never called, so an out-of-scope question gets a refusal in about 1 second instead of 6.
- Tests written against real bugs
- Most tests are regressions for the bugs below. I checked them by putting the bugs back: 7 tests failed, as they should.
Evaluation
- 16 labelled questions, scored with hit@k and MRR.
- An answer set (14 pass, 1 partial) and two refusal tests.
- 38 automated tests, all passing when last run (October 2026), mostly regressions for real bugs.
What failed
- Hidden LaTeX: one paper embeds the source of every equation as invisible text. It was 44% of that paper’s characters and 57% of its chunks, spotted because 15 pages produced 165 chunks.
- 568 words hyphenated across line breaks (“FloorPlan-CAD”) could never match a keyword search until they were repaired at load time.
- Rank fusion buried the one chunk ranked #1 by keywords and #250 by vectors. Each method now gets reserved slots before fusion fills the rest.
- The model multiplied two unrelated figures from different pages and presented the result as fact. The prompt now forbids arithmetic on numbers the text doesn’t state.
What changed
- Source-aware chunks: each chunk prefixed with its paper’s title.
- Hybrid retrieval: BM25 alongside vector search.
- Reserved slots for each method before rank fusion.
- A cross-encoder reranks 20 candidates down to 6.
- A relevance cut-off before generation, with a sanctioned refusal.
Result
- Before reranking, hit@6
- 94%
- Before reranking, MRR
- 0.677
- After reranking, hit@6
- 100%
- After reranking, MRR
- 0.783
- Retrieval on 16 labelled questions: hit@6 100%, MRR 0.783.
- Reranker lifted MRR from 0.677 to 0.783 and hit@6 from 94% to 100%.
- Answer eval: 14 pass, 1 partial on the main set; both refusal tests pass.
- 38 automated tests, all passing (October 2026). Answer eval: 14 pass, 1 partial; both refusal tests pass.
Limitations
- “Why” questions still go wrong: the right passage now reaches the prompt, but the 8B model answers from the results section instead of the motivation.
- When a paper states two different counts, the model sometimes leads with the wrong one.
- Text from diagrams and tables still ends up inside chunks.
- Web pages that build their content with JavaScript return no text.
- hit@k is slightly optimistic, because the right answers were labelled per page and a page holds several chunks.
What I'd try next
- Query expansion for “why” questions: search for the problem a mechanism solves, not only its name.
- A larger model. Retrieval already reaches 100% hit@6, so the remaining failures are the model misreading text it was correctly given.
What I learned
- Every design change here was made after measuring something, not after reading answers.
- A test that has never failed may be asserting nothing, so I put each bug back to check.
Links
Want to talk through how I built this?
I'm happy to walk through the decisions, what failed and what I'd change.