Tweet by madiator

September 23, 2025

Let me try to explain why I don't really like RAG. But before I start, I want to say, RAG is a pretty useful tool and can address 95+% of the use cases, especially now that contexts are quite long and with newer techniques like agentic RAG. So the following text is mostly for understanding sake and to develop a different perspective. ## Point 1: Compositional generalization So, during pretraining, LLMs see all sorts of text. They learn about Shakespeare and they learn about programming, independently. And then they are able to write a python program with comments the style of Shakespeare, though they have not seen these two together. This is compositional generalization. LLMs are able to learn many different skills/facts during pretraining and then combine them together during inference, and this is not something that can happen easily with RAG especially due to the retrieval part. One ~visual analogy for thinking about RAG is as follows: you have a question about a book, and you ask a retrieval model to tear a page or two, which you use to read and answer. Tough luck if the answer is not there, or if the answer is spread across many many pages (e.g. "how did the character evolve over the different chapters"). ## Point 2: Compression <-> Understanding Feynman had famously said "What I can't create I don't understand". That's why we invented "generative" models, which compress all the knowledge in the world during pretraining and can then generate (after having understood the world). Search stands in the way. Long context is better, but it has it's own challenges. Agentic RAG and all the other techniques are mostly hacks to improve upon the situation. Techniques like building a knowledge graph etc., are very much heuristics and don't really generally work well across the spectrum. What has worked well for all of the Internet is to pretrain and then do SFT. So I hope we can do something similar with a small corpus, and that's what Mr. mcconaughey wants.

Author
madiator
Date
September 23, 2025
Open on X