--- title: "Retrieval is rarely the bottleneck. Assembly is." date: "2024-02-01T06:00:00.000Z" categories: "Agents" canonical: "https://xraph.com/writing/retrieval-is-not-the-bottleneck" --- # Abstract Most retrieval failures I have debugged had the right passage in the result set and lost it to a naive concatenation with no token budget. ## Where the debugging usually ends up Someone reports that the system answered a question badly. The first assumption is that retrieval missed the relevant document. Checking the retrieval results shows it did not: the right passage was ranked third. What happened is that the prompt builder concatenated the top ten results, hit a length limit, and truncated. Or included all ten, and the model attended to the first and last. The retriever did its job and the assembly step threw the answer away. ## Assembly deserves to be its own stage The retrieval-augmented generation formulation separates a retriever from a generator [1], and most implementations treat what sits between them as string handling. Surveys of the area now consistently identify that middle step as a distinct concern with its own design space [2]. In the pipeline I built it is a first-class stage with three responsibilities. **Budget.** Every passage has a token cost. The assembler knows the budget and decides what fits, rather than discovering a limit by being truncated. **Ordering.** Position matters measurably [3], so the most relevant passage does not go in the middle. Putting the strongest results at the beginning and the end of the context is a cheap change with a real effect. **Attribution.** Each passage carries an identifier through into the prompt so the model can cite it and the answer can be checked. Without this, verifying a claim means re-running retrieval and guessing. ## Chunking is a retrieval decision and an assembly decision Small chunks retrieve precisely and lose context. Large chunks carry context and dilute the embedding, so they match less precisely. What has worked is retrieving small and assembling large: embed and rank at paragraph level, then expand the winners to their surrounding section before assembly. That costs a second lookup and it gets both properties. ## Duplication Three passages saying the same thing waste two thirds of the budget they occupy, and the effect is worse than waste. A model given the same claim three times treats it as better supported than a claim stated once, which is not what the corpus meant. Deduplicating by similarity before assembly is a small amount of work and it is the change that most improved answer quality in my own measurements. ## What retrieval cannot fix If the answer is not in the corpus, no amount of retrieval produces it, and the model will often produce something anyway. Hallucination under insufficient evidence is well documented [4], and the mitigation is not better retrieval. It is letting the pipeline return nothing. A system that can say it does not know is more useful than one that always answers, and building that requires a confidence signal from the retrieval stage that the assembly stage is willing to act on. # References 1. Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", Advances in Neural Information Processing Systems (NeurIPS), 2020 2. Yunfan Gao et al., "Retrieval-Augmented Generation for Large Language Models: A Survey", arXiv:2312.10997, 2023 3. Nelson F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", Transactions of the Association for Computational Linguistics, vol. 12, pp. 157-173, 2024 4. Ziwei Ji et al., "Survey of Hallucination in Natural Language Generation", ACM Computing Surveys, vol. 55, no. 12, 2023 --- *Unpublished draft. Not peer reviewed and not submitted to a venue.*