A hands-on test of BGE-M3 + Qwen3 (RAG vs. direct-context answering) on a real research paper and a full-length book including a retrieval bug hiding in a footnote, and one surprisingly good model behavior.
I wanted to answer a simple question: when you feed a document to an AI model, is it actually reading it or just pattern-matching to whatever text happens to look similar to your question?
So I built a small open-source pipeline to test this directly. For any document and question, it generates two separate answers:
RAG answer: BGE-M3 finds the most relevant chunks of the document, and Qwen3 answers using only those chunks.
Direct answer: Qwen3 reads the raw document text directly, no retrieval involved.
Both run on a free Google Colab GPU. I kept the retrieval side deliberately “vanilla” fixed-size chunking, plain cosine similarity, no reranking, no fancy tricks so I could see exactly where the basic version breaks before adding any fixes.
Before running my first real test, I already knew one thing to guard against: reference lists. Early experimentation (not covered here) showed that a paper’s bibliography, once chunked like any other text, can get retrieved as if it were real content a citation for a paper about “text embeddings” can look deceptively similar to a generic question about a document’s topic. So going in, my pipeline already strips everything after a References/Bibliography heading before chunking.
With that fix in place, I ran two real tests.
Test 1: A research paper on Nepali legal machine translation
First document: a SIGUL 2024 workshop paper on a bidirectional English-Nepali machine translation system for the legal domain. Question: “What is this paper about?”
RAG answer:
This paper presents the first transformer-based bidirectional machine translation system for the English-Nepali legal domain, using a custom-built parallel corpus of 125,000 sentences. It achieves encouraging BLEU scores and addresses the scarcity of domain-specific legal translation resources.
Direct answer:
This paper presents a bidirectional English-Nepali Machine Translation (MT) system specifically designed for the legal domain using a Neural Machine Translation (NMT) model with a transformer architecture. It introduces a custom-built legal corpus of 125,000 parallel sentences and reports BLEU scores of 7.98 (Nepali→English) and 6.63 (English→Nepali).
Both agree. The direct answer adds slightly more numeric precision (the exact BLEU scores), which makes sense it read the whole document, while RAG only had 5 retrieved chunks to work with.
Worth noting: the top retrieved chunk (similarity 0.515) was actually the paper’s Ethics Statement, not its abstract retrieval still wasn’t hitting the “best” chunk first. But this time it didn’t break the answer, because enough of the other retrieved chunks were genuinely relevant to the paper’s actual content. A clean result, and proof my references fix generalizes beyond the document I originally built it for.
Test 2: A full-length book where things fell apart
Second document: the book Hands-On Large Language Models by Jay Alammar and Maarten Grootendorst. Same question: “What is this paper about?”
RAG answer:
This paper is about machine learning research communication through illustrated and interactive web articles, aiming to rethink how scientific understanding in machine learning is shared. It emphasizes visual and interactive methods to make complex ML concepts more accessible to researchers and engineers.
That is not what this book is about. Not even close.
Direct answer, for comparison:
The document is not a paper but a book review and promotional content for Hands-On Large Language Models... It highlights the book's comprehensive, visually engaging, and practical coverage of large language models (LLMs), including topics like language understanding, generation, tokenization, semantic search, and RAG.
The direct answer correctly identifies what the document actually is even catching that it’s a book, not a paper, which I hadn’t specified in the question.
So why did RAG go so wrong? I checked the retrieved chunks. The top-scoring one (similarity 0.520) turned out to be a footnote buried in the book’s dedication, citing a real Jay Alammar blog post literally titled “Machine learning research communication via illustrated and interactive web articles.” My model didn’t describe the book at all, it described a footnote’s citation title, almost word-for-word.
This is the same disease as the references bug, just in a different part of the document. My fix only strips content after the References heading it does nothing about a book’s front matter: dedications, acknowledgments, “about the author” sections, and footnotes scattered throughout the early pages. Books front-load several pages of non-content material before the real subject starts, and none of that is currently being filtered out.
Test 3: A genuinely good surprise, on the same book
I asked a second, more specific question about the same document: “What is the main contribution of this paper?”
RAG answer:
The context provided does not include a specific paper with a main contribution described. Therefore, based solely on the given information, the main contribution of any paper cannot be identified.
Instead of confidently answering from irrelevant chunks the way it just did one question earlier the model refused to answer, because the retrieved context genuinely didn’t contain a relevant answer.
This is worth pausing on. My RAG prompt has always included the instruction “answer using ONLY the context below.” Here, for the first time, I actually watched that instruction do real work, when retrieval handed the model garbage, it recognized the garbage instead of inventing a plausible-sounding wrong answer anyway. That’s the difference between a RAG system that fails loudly (useful you can tell something went wrong) and one that fails silently (dangerous you’d never know). I didn’t engineer this specifically, it just showed up.
I’ll also flag the honest caveat: this is one data point. The model hallucinated on one question and correctly declined on the very next one, about the same document. I can’t yet say this refusal behavior is reliable just that it’s possible, and worth testing for deliberately.
What I’m taking from these first two tests
- My references fix works, but it’s narrow: It solves one specific shape of junk (bibliographies at the end of academic papers) and does nothing for a completely different shape of junk that shows up in books.
- The real failure category is “structurally dense but non-substantive text: Footnotes, dedications, acknowledgments, and tables of contents are major retrieval risks.
- Good instruction-following can act as a safety net, not a guarantee: It helps when retrieval fails, shouldn’t be relied on it without testing further.
Next steps:
- Build a more general “strip non-content sections” step likely needs different logic for books vs. papers, since their front/back matter is structured differently
- Test whether adding a reranking step (re-scoring retrieved chunks for actual relevance, not just embedding similarity) would have caught the footnote issue on its own, without document-type-specific cleanup rules
- Run the “does the model refuse or hallucinate on bad context” test deliberately across more documents, since right now I only have one example of each
If you’re building anything with RAG, my early takeaway a fix that solves one document’s retrieval problem doesn’t necessarily solve the next document’s version of the same problem. Papers and books look structurally similar long text, sections, references but the actual shape of their noise is different enough to break a narrow fix.
Full pipeline (BGE-M3 + Qwen3, Colab notebook) is open on GitHub.
