RAG shows up in nearly every serious LLM application that needs to answer questions about specific, current, or private data. Here's what it actually does under the hood.
The problem it solves
A language model's knowledge is frozen at training time and limited to what was in its training data. Ask it about your company's internal documentation, a document uploaded five minutes ago, or anything published after its cutoff, and it either says it doesn't know — or worse, confidently makes something up.
RAG's fix: instead of relying purely on the model's trained-in knowledge, retrieve relevant information at query time and hand it to the model as context, then ask the model to answer using that context.
The pipeline, step by step
- Ingest: your documents (docs, PDFs, support tickets, whatever) are split into chunks and converted into vector embeddings — numerical representations that capture semantic meaning — then stored in a vector database.
- Query: when a user asks a question, that question is also converted into an embedding.
- Retrieve: the system finds the stored chunks whose embeddings are most similar (closest in vector space) to the question's embedding — typically the top 3-10 most relevant chunks.
- Augment: those chunks get inserted into the prompt sent to the LLM, usually with instructions like "answer using only the following context."
- Generate: the model produces an answer grounded in the retrieved text, ideally citing which chunk it came from.
Why not just fine-tune the model on your data instead?
Fine-tuning bakes information into the model's weights — it's slower to update (retrain every time your data changes), more expensive, and doesn't give you a way to show which source an answer came from. RAG keeps your data external and swappable: update a document, and the next query immediately has access to the new version, no retraining required. It also lets you cite sources, which matters a lot for trust in any answer-facing product.
Where RAG actually breaks down
- Bad chunking. Splitting documents at arbitrary character counts instead of natural boundaries (paragraphs, sections) can cut a relevant sentence in half across two chunks, and neither chunk alone contains the full answer.
- Retrieval that misses the right chunk. If the top-k search doesn't surface the chunk that actually answers the question — common with vague queries or documents using different terminology than the question — the model never sees the right information and either says "I don't know" or guesses.
- Context window overload. Stuffing in too many retrieved chunks "just in case" dilutes the signal and can push out genuinely relevant information, or exceed the model's effective context length where it reliably pays attention.
The practical takeaway
RAG isn't a single technique so much as a pattern — the quality of a RAG system is determined far more by the retrieval step (chunking strategy, embedding model, search relevance) than by which LLM sits at the end of the pipeline. Most production RAG problems are debugging what got retrieved, not the model's reasoning.
RAG and prompt caching solve adjacent but different problems — RAG gets fresh, specific context into a prompt; caching makes repeatedly sending large stable context (like retrieved chunks reused across a conversation) cheap. Real production RAG systems usually need both.
Hybrid search: combining vector and keyword search
Pure vector (semantic) search sometimes underperforms on queries containing exact identifiers — a product SKU, an error code, a specific proper noun — because embedding similarity captures meaning, not exact string matches. Hybrid search runs a traditional keyword search (BM25 or similar) alongside the vector search and combines both result sets, typically via a weighted score or reciprocal rank fusion:
Query: "error E4021"
Vector search alone: may return semantically similar errors,
missing the exact E4021 match entirely
Hybrid search: keyword component guarantees an exact
"E4021" match surfaces, vector component
still catches semantically related context
Most managed vector databases (Pinecone, Weaviate, and others) support hybrid search natively now — it's usually a configuration choice on the query, not a separate system to build, and it's worth defaulting to for any RAG system fielding real user queries rather than only clean natural-language questions.
Evaluating whether a RAG system actually works
"It seems to answer things correctly" isn't a real evaluation strategy once a RAG system is handling meaningful traffic. A minimal, honest evaluation setup needs:
- A test set of real questions with known-correct answers, ideally sourced from actual user queries rather than ones the team invents (invented test questions tend to be easier than real ones, and can overstate quality).
- Retrieval metrics — did the system retrieve the chunk that actually contains the answer, independent of whether the final generated answer was correct? This isolates whether a wrong answer came from bad retrieval or bad generation.
- Answer correctness, judged against the known-correct answer — either by a human reviewer or an LLM-as-judge setup, with the honest caveat that LLM-as-judge has its own failure modes and needs periodic human spot-checking, not blind trust.
Separating retrieval quality from generation quality in the evaluation is what makes the results actionable — a system scoring poorly on answer correctness but well on retrieval points at a prompt or model problem; poor retrieval scores point at chunking or embedding problems instead, an entirely different fix.
Common mistakes
- Debugging a wrong RAG answer by tweaking the prompt first, instead of checking what was actually retrieved. If the wrong (or no) chunks were retrieved, no amount of prompt engineering fixes that — inspect the retrieved context before touching the prompt.
- Using a fixed chunk size regardless of document structure. Splitting purely by character or token count, ignoring paragraph and section boundaries, is the most common cause of a relevant answer getting cut across two chunks where neither alone is sufficient.
- Retrieving a fixed top-k count regardless of query complexity. A narrow factual question and a broad comparison question need different amounts of context — a fixed k is a reasonable starting default, not something to leave unexamined once you have real usage data.
- Assuming a bigger context window means chunking strategy matters less. Even with a large context window, dumping in more marginally-relevant chunks can dilute the signal and increase cost without improving answer quality — relevance still matters more than volume.
Related reading
- Prompt Caching for LLM Apps: How It Works and Why It Cuts Costs — shares tags: ai, programming (same category).
- Async Python with asyncio: A Practical Introduction — shares tags: programming.
- Big O Notation Without the Math Panic — shares tags: programming.
- Clean Code Principles That Actually Hold Up in Practice — shares tags: programming.
- Core Web Vitals Explained: What Actually Affects Your Score — shares tags: programming.
