Ask an LLM about its own training data and it does fine. Ask it about your internal pricing policy from last Tuesday and it invents something confident and wrong. The reason is simple and often misunderstood: a model isn't Google, and it has no memory after training. Everything it "knows" for a given answer has to be handed to it inside the request.
The standard fix is RAG — Retrieval-Augmented Generation. This post explains it from first principles, without jargon, and shows where to feel it working in a real tool.
The one sentence
A model can't remember your documents, so instead of making it remember, you retrieve the relevant part of your documents at answer-time and paste it into the prompt. The model then writes an answer grounded in those exact words.
That's it. The whole field of "AI reading your company's knowledge base" is built on that trick.
Why the model needs help at all
Two properties force the design:
- No post-training memory. A pretrained LLM is a snapshot. It doesn't update itself, it doesn't browse, it doesn't recall. To use new or private information, that information must enter the context window of the request.
- The context window is finite. You can't dump a thousand pages into the prompt. So you need a way to pick, fast, the passages actually relevant to the question.
RAG solves pick + paste.
The pipeline in four steps
| Step | What happens | Why |
|---|---|---|
| 1. Ingest (offline) | Split each document into small chunks and turn each chunk into an embedding (a vector of its meaning) | So we can search by meaning, not keywords |
| 2. Query | Embed the user's question into the same vector space | So we can compare it to the chunks |
| 3. Retrieve | Find the chunks whose vectors are "closest" to the question's vector | Picks the genuinely relevant passages |
| 4. Generate | Put those chunks into the prompt as grounding context | Lets the model answer from the sources |
Two pieces deserve a closer look, because they're the ones people hand-wave.
Embedded meaning beats keyword search
An embedding is a list of numbers representing the meaning of text, arranged so similar meanings sit at nearby points in space. Search "how do I refund this invoice" and a storeholder of "return policy" documents surfaces even if no exact keyword appears — because the meaning is close. Keyword search fails on paraphrase; embedding search doesn't care about wording, only about topic.
Good chunking is quietly important
Before embedding, the document is split into chunks — small, self-contained pieces respecting paragraph and heading boundaries. A whole chapter embeds poorly and can't be retrieved as a snippet; a well-chunked piece is one coherent idea that fits in the prompt alongside everything else. Chunking quality often matters more than people expect.
RAG vs. fine-tuning vs. prompting
| Approach | Changes | Cost | When |
|---|---|---|---|
| Prompting | Nothing — you write better instructions | Free | Most everyday tasks |
| RAG | The context you feed | Cheap, reversible, auditable | Private/up-to-date facts, sourcing, retrieval-heavy answers |
| Fine-tuning | The model's weights | Expensive, irreversible, needs data | Specializing a model on a narrow style/domain |
RAG's superpower is that you always know exactly which documents grounded an answer — full auditability. It's why knowledge platforms that care about correctness lean on it rather than on fine-tuning.
Feel it working
The cleanest way to experience the pipeline is to drop your own documents into a real RAG store and ask it questions your model never trained on. The Knowledge Base in SynthHires (/space/knowledge) is exactly that: upload PDFs, DOCX, Markdown, a URL, or even a YouTube video; it chunks and embeds them; then you can reuse the matching chunks as grounding context in any chat.
- See the full mechanics (embeddings, chunking, RAG) in AI concepts from scratch.
- See the product surface (ingestion, semantic search, statuses) in Memory & knowledge.
The boundary to remember
RAG grounds the model in what you retrieved, but it doesn't make the model omniscient about your whole corpus — only about the chunks it pulled this turn. The art of a good RAG setup is: chunk well, retrieve the right slices, and keep the window focused. Get those right and your AI stops hallucinating your last Tuesday and starts actually knowing it.