Skip to main content
BitsNotes
—
Natural Language Processing

Retrieval Augmented Generation

Published: 2026-08-09
Level: postgraduate
Audience: Postgraduate students in Natural Language Processing

14.1 Limitations of Large Language Models

Hook: Imagine hiring the most brilliant scholar in the world — someone who has read every book, Wikipedia page, and article published up to 2023. They can summarize complex theories, translate languages flawlessly, and write beautiful poetry. But there is a catch: they have been locked in a windowless room since early 2023, they know nothing about your company's internal documents, and when they do not know the answer, they confidently make one up. This scholar is a metaphor for every large language model in production today.

Large language models (LLMs) are the backbone of nearly every AI application today — question answering, conversational chatbots, text summarization, machine translation, content generation, and more. At their core, language models are trained to predict or generate text given some input, and they excel at producing fluent, human-like language across all these tasks.

Yet three fundamental limitations hold them back in production settings.

The Three Limitations of LLMs

  1. Knowledge cutoff. Every pre-trained LLM is frozen at the point its training data ends. A model trained on data up to 2023, for instance, cannot answer questions about events or information that emerged after that date. The model's internal knowledge is static — it does not update itself as the world changes.
  2. Hallucinations. LLMs sometimes fabricate answers that sound plausible but are factually wrong. This is unacceptable in business, legal, or medical contexts where decisions carry real consequences. A model confidently stating an incorrect drug dosage or a fabricated legal precedent can cause serious harm.
  3. Lack of specificity. Proprietary, organization-specific, or federated knowledge is simply not in the training data. A generic LLM has no access to your company's internal HR policies, product manuals, or customer support history. If you ask it about your international travel reimbursement policy, it might return the policy of some other company — or worse, a plausible-sounding fabrication.

Intuition: Think of an LLM as a brilliant student taking a closed-book exam. They studied everything up to a certain date, but they cannot look anything up during the test. If they do not remember the answer, they guess — and they guess confidently. The three limitations map directly to this analogy: knowledge cutoff means the exam covers material after their study period, hallucination means they guess when unsure, and lack of specificity means the exam asks about notes they never had access to.

These three problems — knowledge cutoff, hallucination, and lack of specificity — share a common root: the model's knowledge is confined to its training data. The solution is to give the model access to live, external, up-to-date information at query time. That is the purpose of retrieval augmented generation.

Pitfall — Confusing fluency with accuracy: The most dangerous property of an LLM is that its wrong answers sound just as confident as its correct ones. A human expert who does not know something will say "I'm not sure." An LLM will produce a grammatically perfect, authoritative-sounding paragraph that is entirely fabricated. Never trust an LLM's output in a high-stakes domain without grounding it in verified source material — which is exactly what RAG provides.

Pitfall — Assuming the model "knows" your data: A common beginner mistake is asking an LLM about company-specific information and treating the response as if it came from internal systems. The model has never seen your HR policy, your product specs, or your customer database. Any answer it gives about your proprietary data is either a hallucination or a lucky coincidence from training on similar public data.

14.1.1 Why Not Just Fine-Tune?

Hook: If the problem is that the model does not know your data, why not just train it on your data? This seems like the obvious fix — but for large models, it is like rebuilding a skyscraper to change a light bulb.

Fine-tuning is one way to adapt a pre-trained model to domain-specific data. The idea is straightforward: take a pre-trained model and continue training it on your proprietary dataset so its weights encode your organization's knowledge.

Why fine-tuning fails at scale:

  • A model like GPT-4 has approximately 175 billion parameters. Fine-tuning even a fraction of these requires multiple high-end GPU servers — each costing tens of lakhs of rupees — and training takes significant time (days to weeks).
  • Fine-tuning must be repeated every time the underlying data changes. If your company updates its HR policy quarterly, you must re-fine-tune quarterly. The cost and time multiply with every update.
  • Fine-tuning risks catastrophic forgetting — the model may lose some of its general capabilities as it over-specializes on your domain data.
  • The data must be carefully curated and formatted. Noisy or inconsistent fine-tuning data can degrade model quality.

The key insight: instead of changing the model's weights, give it access to a live knowledge base at inference time. RAG does exactly that — it retrieves relevant external information and feeds it to the model as context, without any retraining.

Analogy — The open-book exam: If standard LLM usage is a closed-book exam where the model relies entirely on its memory, RAG transforms it into an open-book exam. When you ask a question, the system first searches the library for the exact pages containing the answer, places those pages on the desk, and says: "Read these specific pages and answer the question." The student (LLM) does not need to have memorized the book — they just need to be good at reading and understanding what is in front of them.

Real-world: Students shared that in their companies, RAG is used for engineering document review, paper-journal analysis, and proprietary data querying — uploading documents to a RAG system and asking questions to aid understanding without reading entire files.

Worked example — Cost comparison:

Suppose your company has a 500-page internal policy manual that changes every quarter.

  • Fine-tuning approach: Curate Q&A pairs from the manual → fine-tune a 7B-parameter model → deploy. Cost: ₹5–10 lakhs per cycle in compute alone, plus weeks of engineering time. Repeat every quarter.
  • RAG approach: Extract text from the PDF → chunk it → embed it → store in Qdrant. Deploy a small local model (Qwen, Llama) or use an API. Cost: ₹10,000–50,000 for the initial setup; updates require only re-chunking the changed pages. The model itself is never retrained.

The RAG approach is orders of magnitude cheaper and faster to update. This is why, in practice, almost every production AI system today uses RAG rather than fine-tuning for knowledge-grounded tasks.

Recap: LLMs have three inherent limitations — knowledge cutoff, hallucination, and lack of specificity — all stemming from the fact that their knowledge is frozen in their training data. Fine-tuning is too expensive and slow for most organizations. RAG solves all three by giving the model access to live, external knowledge at inference time, without changing a single model weight. This sets the stage for understanding the RAG pipeline itself.

Q: Can we use open-source models hosted locally instead of proprietary API-based models for RAG? A: Yes — RAG works with any LLM, proprietary or open source. The data being provided through RAG is proprietary; the model itself can be anything. You can host a local copy of a small language model like Llama or Qwen using Ollama to keep all data within your organization. The appended context goes to whichever model you choose.

14.2 RAG Pipeline — Retrieval, Augmentation, Generation

Hook: You have a question. Somewhere in a mountain of documents — PDFs, databases, policy manuals — lies the answer. How do you build a system that finds the right needle in that haystack and hands it to a language model so it can give you a grounded, accurate answer? That is the RAG pipeline.

RAG stands for Retrieval Augmented Generation. Each word in the name describes one stage of the pipeline:

  1. Retrieval — find the most relevant chunks from an external knowledge base, given the user's query.
  2. Augmentation — append those relevant chunks to the user's prompt as context.
  3. Generation — send the augmented prompt to the LLM, which generates an answer grounded in the provided context.

Intuition — The librarian analogy: Think of the RAG system as a three-person team. The retriever is a librarian who instantly finds the most relevant pages from a vast library. The augmenter is an assistant who clips those pages and tapes them to the student's question sheet. The generator is the student (the LLM) who reads the question, reads the attached pages, and writes an answer based on what is in front of them. The student does not need to have memorized the library — they just need to be good at reading comprehension.

The three stages happen in sequence for every user query. The critical property is that the LLM never needs to "know" the answer from its training data — it only needs to be good at understanding and using the context it is given.

14.2.1 The Knowledge Base

The external knowledge base can be anything: PDF documents, company ERP data, PowerPoint presentations, CSV files, internal web pages, product manuals, customer tickets, historical records, or live databases. The key is that this knowledge is proprietary, up-to-date, and specific to the organization.

What counts as a knowledge base: Any collection of information that the organization owns and wants the LLM to be able to reference. Common examples:

  • HR policy documents (PDF)
  • Product manuals and technical specifications
  • Customer support ticket history
  • Financial reports and quarterly filings
  • Internal wiki pages and engineering documentation
  • Meeting transcripts and recordings (via multimodal RAG)

The knowledge base is external to the model — it lives in a database, not in the model's weights. This is what makes RAG different from fine-tuning.

14.2.2 Indexing Phase (Offline)

Before any queries can be answered, the knowledge base must be pre-processed. This is the offline or indexing phase — it happens once (and again whenever documents are updated), not at query time.

Indexing pipeline — four steps:

  1. Document extraction — convert source documents (PDFs, web pages, etc.) into plain text. Tools like PyPDF can extract text from PDF files.
  2. Chunking — divide the text into manageable pieces (chunks), typically 400–600 tokens each. (Chunking strategies are covered in detail in §14.4.)
  3. Embedding — convert each chunk into a vector representation using a word embedding model such as Hugging Face Sentence Transformers. Each chunk becomes a point in a high-dimensional space.
  4. Storage — store the chunk embeddings in a vector database (also called a vector data store) such as Qdrant, Pinecone, FAISS, or Weaviate.

Pitfall — Using different embedding models for indexing and querying: A critical implementation detail that is easy to miss: the same embedding model must be used when indexing chunks and when embedding the user query. If the knowledge base uses a 768-dimensional embedding and the query uses a 300-dimensional embedding, the vectors are incompatible and cosine similarity produces meaningless results. This is one of the most common bugs in RAG systems.

14.2.3 Query Phase (Online)

When a user submits a query, the online phase executes in real time:

Query pipeline — four steps:

  1. Query embedding — the same embedding model used during indexing converts the user's question into a vector.
  2. Similarity search — compute cosine similarity (§14.3.3) between the query vector and all stored chunk vectors. Select chunks above a similarity threshold (e.g., 0.80) or the top-K most similar chunks.
  3. Augmentation — concatenate the selected chunks with the user's original prompt. The augmented prompt typically looks like: "Using the following context: [retrieved chunks]. Answer this: [user query]."
  4. Generation — send the augmented prompt to the LLM, which generates an answer grounded in the retrieved context.

The LLM as an English tutor: The professor describes the LLM's role in RAG as being like an English tutor. The LLM does not need to "know" the facts — it needs to be good at reading the provided context and producing a well-formed, grammatically correct, coherent answer from it. Its primary job is language generation quality, not knowledge retrieval. The retrieval step handles the knowledge; the LLM handles the language.

14.2.4 Worked Example — Company Travel Policy Chatbot

Full trace — From question to grounded answer:

An employee asks: "What is the international travel policy for employees?"

Step 1 — Query embedding: The system converts the question into a 768-dimensional vector using the same Sentence Transformer model used during indexing.

Step 2 — Similarity search: The system computes cosine similarity between this query vector and all 3,847 stored chunk vectors (from 200 company documents). Suppose the scores are:

Chunk Source Similarity
Chunk #142 HR Policy §7.3 — "Overseas Business Trips" 0.94
Chunk #143 HR Policy §7.4 — "Expense Approval" 0.91
Chunk #87 Travel FAQ — "Visa Requirements" 0.82
Chunk #2001 Benefits Guide — "Domestic Travel" 0.71
... ... ...

With a threshold of 0.80, chunks #142, #143, and #87 qualify.

Step 3 — Augmentation: The system builds the augmented prompt:

"Using the following context: [Chunk #142: Overseas Business Trips — Employees traveling internationally must obtain VP-level approval at least 14 days before departure...] [Chunk #143: Expense Approval — International travel expenses are reimbursed up to \$250/day for Tier-1 cities...] [Chunk #87: Visa Requirements — The company sponsors business visas for all full-time employees...] Answer this: What is the international travel policy for employees?"

Step 4 — Generation: The LLM reads the augmented prompt and generates:

"The international travel policy requires VP-level approval at least 14 days before departure. Expenses are reimbursed up to \$250/day for Tier-1 cities. The company sponsors business visas for full-time employees."

Sense-check: Every sentence in the answer is traceable to a retrieved chunk. The model did not invent any facts — it synthesized information from the provided context. Without RAG, a generic LLM might have returned a vague answer about "typical corporate travel policies" drawn from training data about other companies.

Recap: The RAG pipeline has two phases — offline indexing (extract, chunk, embed, store) and online querying (embed query, search, augment, generate). The LLM's role is to read and synthesize, not to recall. This architecture ensures answers are grounded in the organization's actual documents, not in the model's training data.

Real-world: This architecture ensures the answer comes from the organization's proprietary data, prevents hallucination by grounding the response in retrieved facts, and always uses the latest version of the documents.

Q: Is RAG only for proprietary LLM models, not open-source models hosted locally? A: No — RAG works with any LLM, proprietary or open source. The data being provided through RAG is proprietary; the model itself can be anything. You can use API-based models if you are comfortable sending data externally, or you can host a local copy of a small language model like Llama or Qwen to keep all data within your organization. The appended context goes to whichever model you choose.

Q: If we are querying only against our own proprietary data, what is the benefit of using a proprietary paid model over a cheaper open-source model? A: If you use a local model like Qwen, your data stays completely within your organization — nothing is sent to external APIs. This is strongly advised when the queries are primarily about your proprietary data. The LLM's role is mainly to understand the context and generate a well-formed answer from it. A local model can do that well for domain-specific queries. The disadvantage is that for open-set or general-knowledge queries outside your data, the smaller local model may not perform as well as a larger proprietary one.

14.4 Chunking Strategies

Hook: You have a 500-page company policy manual. You cannot feed the entire document to an LLM — it would blow past the context window and cost a fortune. So you cut it into pieces. But how you cut matters enormously. Cut in the wrong place, and a sentence like "The policy applies only to employees who have completed" loses its ending — "their probation period." The art and science of cutting documents into retrievable pieces is called chunking, and it is one of the most consequential design decisions in a RAG system.

Chunking — how the knowledge base text is divided into retrievable pieces — determines how much context each retrieved piece carries, how much information is lost at boundaries, and how many chunks the system must store and search.

14.4.1 Chunk Size

Chunk size is the number of tokens per chunk. Typical values range from 400 to 600 tokens. The optimal size depends on several factors:

  • Vector data store capabilities — different stores (Qdrant, Milvus, etc.) may have different maximum chunk sizes they can index efficiently.
  • Application requirements — more tokens per chunk means richer context per retrieved piece, but also higher cost per chunk and fewer chunks fitting in the context window.
  • Quality-cost-time trade-off — more and larger chunks mean better context, but also higher cost and higher latency. There is no free lunch.

Intuition — Chunk size as a zoom level: Think of chunk size like the zoom level on a map. Zoom in too much (small chunks, e.g., 100 tokens) and you see individual sentences but lose the paragraph context. Zoom out too much (large chunks, e.g., 2000 tokens) and you get rich context but the retrieval becomes less precise — a large chunk may contain a mix of relevant and irrelevant information. The sweet spot (400–600 tokens) balances precision and context richness.

14.4.2 Chunk Overlap

The problem: When a document is chunked sequentially, information that spans the boundary between two chunks can be lost. A sentence split across two chunks may lose its meaning in both. For example:

Chunk 1 ends with: "The approval process requires the employee to submit a travel request form and obtain..." Chunk 2 begins with: "...written approval from their direct supervisor at least 14 days before departure."

Neither chunk alone contains the complete instruction. Chunk overlap solves this by making consecutive chunks share some tokens.

Chunk overlap and stride:

  • Chunk size (\(S\)) = total tokens per chunk (e.g., 600 tokens)
  • Overlap (\(O\)) = number of shared tokens between consecutive chunks (e.g., 120 tokens)
  • Stride (\(S - O\)) = number of unique tokens advanced per chunk (e.g., 600 − 120 = 480 tokens)

The stride is the "step size" — how far the chunking window moves forward for each new chunk. The overlap ensures that when two related chunks are extracted, the bridging information is not lost.

Intuition — Overlapping windows: Imagine reading a book through a sliding window. The window is 600 words wide. After reading each window, you slide it forward by 480 words. The last 120 words of the previous window are re-read at the start of the next window. This ensures no sentence is ever split in a way that loses its meaning — the overlapping region acts as a bridge.

A middle chunk will typically overlap with both its predecessor and successor. The first and last chunks may have overlap on only one side.

14.4.3 Worked Example — Total Chunks from a Document

Full calculation — Chunking the Transformer paper:

Given the paper Attention Is All You Need as the source document:

  • Document size: \(N = 18{,}200\) tokens
  • Chunk size: \(S = 600\) tokens
  • Overlap: \(O = 120\) tokens

Step 1 — Compute the stride:

\[ \text{stride} = S - O = 600 - 120 = 480 \text{ tokens} \]

The stride is the number of unique (non-overlapping) tokens advanced with each chunk.

Step 2 — Compute the total number of chunks:

\[ \text{chunks} = \left\lceil \frac{N}{S - O} \right\rceil = \left\lceil \frac{18{,}200}{480} \right\rceil = \lceil 37.917 \rceil = 38 \]

Step 3 — Verify with the formula:

The general formula for total chunks with overlap is:

\[ \text{chunks} = \left\lceil \frac{N - O}{S - O} \right\rceil \]

Let us verify: \(\frac{18{,}200 - 120}{480} = \frac{18{,}080}{480} = 37.667\). Ceiling gives 38. Both formulas give 38 because the first chunk starts at position 0 and the overlap only affects subsequent chunks.

Why ceiling? Because even a partial final chunk is counted — if the last chunk has fewer than 480 unique tokens, it still occupies a full 600-token chunk slot in the vector store. A document of 18,200 tokens produces chunks at positions 0, 480, 960, ..., 17,760. The last chunk starts at 17,760 and extends to 18,360 (capped at 18,200), still stored as a 600-token slot.

This PDF file will be stored as 38 chunks in the vector database.

Sense-check: With 38 chunks of 600 tokens each, the total token storage is 38 × 600 = 22,800 tokens — more than the original 18,200 due to the overlapping tokens. The overhead is \(22{,}800 - 18{,}200 = 4{,}600\) tokens, which is the cost of preserving boundary context. This overhead is a worthwhile trade-off: without overlap, important information at chunk boundaries would be lost.

Q: When a chunk in the middle is picked, will it have overlap with both the previous and the next chunk? A: Yes — a middle chunk will overlap with both its predecessor and its successor. Only the very first chunk has overlap on just one side (with the second chunk). Think of it this way: every chunk except the first one inherits 120 tokens from the previous chunk, and every chunk except the last one shares its last 120 tokens with the next chunk.

14.4.4 Variable-Length Chunks

Scope — Fixed vs. variable chunks: The examples above use fixed-length chunks (same \(S\) and \(O\) for every chunk). Recent research explores variable-length chunks — for example, splitting at paragraph or section boundaries rather than at a fixed token count. The advantage is more natural semantic units. The challenge is memory allocation — if chunks vary in size, the vector store may need to allocate space equal to the maximum possible chunk size, wasting memory for smaller chunks. Optimization techniques for variable-length chunking exist and will be covered in advanced courses.

Q: Can we have variable chunk sizes? A: Recent research explores variable-length chunks. The challenge is memory allocation — if chunks vary in size, the vector store may need to allocate space equal to the maximum possible chunk size, wasting memory for smaller chunks. Optimization techniques for variable-length chunking exist and will be covered in advanced courses.

14.4.5 Who Decides the Chunk Size?

Chunk size is a hyperparameter — a design choice made by the developer, not learned from data. The optimal value depends on:

  • The vector data store's capabilities and indexing efficiency
  • The application's context requirements (how much context does each query need?)
  • Cost and latency constraints (larger chunks = more tokens = more money and slower processing)
  • Typical range: 400–600 tokens, though some systems use as few as 200 or as many as 1,000

Pitfall — Setting chunk size too small or too large: If chunks are too small (e.g., 50 tokens), each chunk contains only a sentence or two — retrieval becomes very precise but the LLM lacks enough context to generate a good answer. If chunks are too large (e.g., 2,000 tokens), each retrieved chunk contains a mix of relevant and irrelevant information — the LLM may be confused by noise. The sweet spot depends on the domain and document structure; start with 500 tokens and tune based on retrieval quality metrics.

Recap: Chunking divides documents into retrievable pieces of 400–600 tokens. Overlap (typically 120 tokens) prevents information loss at boundaries. The total number of chunks is \(\lceil N / (S - O) \rceil\) using ceiling because partial final chunks still count. Chunk size is a hyperparameter — too small loses context, too large introduces noise.

14.5 Token Budgeting and the Context Window

Hook: Every LLM has a maximum number of tokens it can process at once — its context window. Think of it as a desk with limited space. You can fit a question sheet, some reference pages, and the system instructions on it. But if you pile on too many reference pages, some will fall off the desk — and the most recently added (often the most relevant re-ranked ones) may be the ones that get dropped. Token budgeting is the discipline of deciding exactly how many reference pages fit on that desk.

14.5.1 The Context Window

Context window — the maximum total number of tokens an LLM can accept in a single session. This limit is set by the model provider and is a hard constraint: exceeding it causes truncation.

Representative context window sizes:

Model Context Window
GPT-4 (varies by version) ~128,000 tokens
Claude (varies by version) up to 1,000,000 tokens
Smaller open-source models 2,048 to 32,768 tokens
Qwen (local) typically 8,192–32,768 tokens

The context window is per session, not per prompt. Across multiple turns of conversation, tokens accumulate. Each new prompt consumes tokens from the remaining window.

14.5.2 Composition of the Context Window

The context window tokens are consumed by three components:

Context window composition:

  1. System prompt — instructions like "Using the following context, answer this question." Typically 100–500 tokens.
  2. User query — the actual question being asked. Variable length.
  3. Retrieved chunks — the RAG context appended to the prompt. Each chunk is \(T_{\text{chunk}}\) tokens.

If the total tokens from these three exceed the context window, the content is truncated — later chunks (including potentially the most relevant re-ranked ones) may be dropped without warning.

14.5.3 Worked Example — Token Budgeting

Full calculation — How many chunks fit?

Given:

  • Maximum context window: \(C = 8{,}192\) tokens
  • System prompt tokens: \(T_{\text{sys}} = 200\)
  • User query tokens: \(T_{\text{query}} = 320\)
  • Chunk size: \(T_{\text{chunk}} = 512\) tokens

Step 1 — Compute available tokens for RAG chunks:

\[ T_{\text{available}} = C - T_{\text{sys}} - T_{\text{query}} = 8{,}192 - 200 - 320 = 7{,}672 \text{ tokens} \]

Step 2 — Compute maximum chunks that can be appended:

\[ \text{max chunks} = \left\lfloor \frac{T_{\text{available}}}{T_{\text{chunk}}} \right\rfloor = \left\lfloor \frac{7{,}672}{512} \right\rfloor = \lfloor 14.984 \rfloor = 14 \]

Step 3 — Verify the token usage:

14 chunks × 512 tokens = 7,168 tokens. Total: 200 + 320 + 7,168 = 7,688 tokens. This is within the 8,192 limit.

15 chunks × 512 tokens = 7,680 tokens. Total: 200 + 320 + 7,680 = 8,200 tokens. This exceeds 8,192 by 8 tokens — the content would be truncated. So 14 is the correct maximum.

Why floor (not ceiling)? For chunking, we used ceiling because a partial final chunk still occupies a full slot in the vector store. For token budgeting, we use floor because exceeding the context window causes truncation — we must not go over. The floor operation guarantees we stay within the limit.

Pitfall — Confusing ceiling and floor: This is a classic exam trap. Chunking uses ceiling (partial chunks still count). Token budgeting uses floor (must not exceed the limit). Mixing them up gives the wrong answer on both calculations.

14.5.4 Multi-Turn Sessions

The token budget applies per session, not per prompt. If the first prompt uses up tokens, subsequent prompts have fewer available tokens for RAG chunks.

Intuition — The shrinking desk: Imagine your desk starts with 8,192 units of space. The first question uses some space for the system prompt, the query, and 14 reference pages. The answer the LLM generates also takes up space (though this is on the output side). In the next turn, the conversation history (previous questions and answers) is still on the desk, so there is less room for new reference pages. As the conversation progresses, the maximum number of retrievable chunks decreases. This means each retrieval must be carefully chosen — there is less room for irrelevant context in later turns.

Q: Is the token limit per prompt or per session? A: Per session. You can have multiple turns (multi-turn queries), and tokens accumulate. The maximum chunks shown in the example are for the first prompt. In subsequent prompts, the available chunks will be further reduced.

Q: How is the maximum token size determined? A: Each LLM provider decides the context window size for their model. GPT might allow 50,000 tokens; Claude might allow 1,000,000. If you deploy a local model, you choose a model with a context window that fits your needs.

Q: Does the output token count also consume the context window? A: The context window discussion here concerns the input side — what goes into the model. Output tokens are generated after the input is processed. The input consists of the system prompt, user query, and retrieved chunks together.

14.5.5 Tokens, Cost, and Carbon Footprint

Tokens are not free. Each token processed costs money (API fees) and consumes compute (servers, electricity, carbon).

The economics of tokens: Even startup companies spend thousands of dollars per day on LLM tokens for production RAG systems. Observability and telemetry tools track per-query token consumption, system prompt tokens, and total usage — essential for cost management. The professor emphasizes: the token budget decides the carbon footprint and organizational cost — lesser tokens is better. This is not just an engineering constraint; it is a business and environmental one.

Real-world: Free-tier API access typically has daily token limits. Once exhausted, you either wait or pay. For production systems operating 24/7, costs scale linearly with token usage.

14.5.6 Similarity Threshold vs. Token Budget — Dual Constraint

Dual constraint on chunk selection:

Two constraints govern which chunks are appended to the prompt:

  1. Similarity threshold — only chunks above a minimum similarity score (e.g., 0.80) qualify.
  2. Token budget — at most \(N\) chunks fit within the available context window.

Both must be satisfied simultaneously. The process is:

  • Step 1: Filter by similarity threshold → keep only chunks with similarity ≥ 0.80.
  • Step 2: Sort by similarity (descending) → highest-similarity chunks first.
  • Step 3: Take the top \(N\) chunks that fit within the token budget.

Pitfall — Padding with irrelevant chunks: Adding chunks with very low similarity (e.g., 0.01) just because there is room in the budget introduces noise and can cause the LLM to hallucinate. The professor's warning: never pad the context with irrelevant chunks. It is better to send fewer, high-quality chunks than to fill the context window with noise. Quality over quantity.

Recap: The context window is a hard token limit per session. Token budgeting computes how many chunks fit: floor of available tokens divided by chunk size. Two constraints — similarity threshold and token budget — must both be satisfied. Never pad with low-similarity chunks just to fill the budget. In multi-turn conversations, the budget shrinks as history accumulates.

14.6 Latency in the RAG Pipeline

Hook: A user asks a question and waits. One second passes. Two seconds. Three seconds. They switch to Google. In production RAG systems, latency is not just a technical metric — it is the difference between a product people use and one they abandon. Every millisecond in the pipeline must be accounted for, budgeted, and justified.

Latency — the total time to answer a query using RAG — is critical for user experience. If the system takes too long, users switch to alternative tools.

14.6.1 Components of RAG Latency

RAG latency components:

Component Symbol Description Required?
Embedding time \(T_{\text{embed}}\) Time to convert the query into a vector using the sentence transformer Yes
Retrieval time \(T_{\text{retrieval}}\) Time to search the vector store, compute similarity, and return top-K chunks Yes
Re-ranking time \(T_{\text{rerank}}\) Time to re-order chunks by relevance after initial retrieval Optional
Generation time \(T_{\text{gen}}\) Time for the LLM to generate the final answer from the augmented prompt Yes

Total latency:

\[ T_{\text{total}} = T_{\text{embed}} + T_{\text{retrieval}} + T_{\text{gen}} + T_{\text{rerank}} \]

Intuition — The assembly line: Think of the RAG pipeline as a factory assembly line with four stations. The product (the answer) must pass through each station in order. Stations 1, 2, and 4 (embed, retrieve, generate) are mandatory — the product cannot skip them. Station 3 (re-ranking) is optional — it improves quality but adds time. If the assembly line has a deadline (the latency budget), the manager must decide whether there is enough time to include the optional station.

Generation time is typically the largest component because LLMs are computationally expensive — generating text token by token requires running a multi-billion-parameter neural network for each token.

14.6.2 Worked Example — Latency Budget

Full calculation — Can we afford re-ranking?

Given:

  • Allowed total time: \(T_{\text{allowed}} = 2{,}500\) milliseconds
  • Embedding time: \(T_{\text{embed}} = 200\) ms
  • Retrieval time: \(T_{\text{retrieval}} = 370\) ms
  • Generation time: \(T_{\text{gen}} = 1{,}600\) ms

Step 1 — Compute time consumed by the three mandatory steps:

\[ T_{\text{mandatory}} = T_{\text{embed}} + T_{\text{retrieval}} + T_{\text{gen}} = 200 + 370 + 1{,}600 = 2{,}170 \text{ ms} \]

Step 2 — Compute remaining time for re-ranking:

\[ T_{\text{rerank}} = T_{\text{allowed}} - T_{\text{embed}} - T_{\text{retrieval}} - T_{\text{gen}} = 2{,}500 - 2{,}170 = 330 \text{ ms} \]

Step 3 — Decision:

If re-ranking takes more than 330 ms, it must be skipped — sacrificing some quality for speed. If it takes less (e.g., 200 ms), it can be included and the total latency stays within budget.

Sense-check: The mandatory steps consume 2,170 ms out of 2,500 ms — that is 86.8% of the budget. Only 13.2% remains for optional enhancements. This illustrates why generation time dominates: it accounts for \(1{,}600 / 2{,}500 = 64\%\) of the total budget alone.

Pitfall — Ignoring the pipeline order: Re-ranking must happen before generation because it changes which chunks are sent to the LLM. This means the decision to include re-ranking must be made before generation starts. But generation time is not known until it completes. So the system must predict how long generation will take (using telemetry from prior queries) and decide whether re-ranking fits within the remaining budget. If the prediction is wrong, the total latency may exceed the budget.

14.6.3 The Quality-Cost-Time Triangle

The three-way trade-off:

There is a fundamental tension between three desirable properties:

  • Higher quality (larger model, more chunks, re-ranking) → higher cost and longer time
  • Lower cost (smaller model, fewer chunks, no re-ranking) → lower quality
  • Lower latency (smaller model, fewer chunks) → lower quality and lower cost

You can optimize for any two, but not all three. This is sometimes called the "iron triangle" of RAG system design. Choosing the right balance depends on the application's requirements:

  • A customer-facing chatbot needs low latency (< 2 seconds) — accept lower quality.
  • A legal document analysis tool needs high quality — accept higher cost and latency.
  • An internal research tool needs low cost — accept higher latency and moderate quality.

14.6.4 Predicting Generation Time

Q: Re-ranking must happen before generation. Do we need to predict how long generation will take? A: Yes — since re-ranking precedes generation in the pipeline, the decision to include re-ranking must be based on a predicted generation time. This prediction uses telemetry data from prior queries to estimate how long the LLM will take. In practice, systems maintain a rolling average of recent generation times and use that as the prediction. If the predicted time plus embed plus retrieval plus re-ranking exceeds the budget, re-ranking is skipped.

Recap: RAG latency has four components: embedding, retrieval, re-ranking (optional), and generation. Generation dominates. The latency budget forces a decision: if mandatory steps leave enough time, re-ranking can be included; otherwise it is skipped. The quality-cost-time triangle means you can optimize for two of the three, never all three.

14.7 Graph RAG

Hook: You ask a vector database: "Which board meetings in the last twelve months had at least two members abstain from a vote?" The vector database searches for chunks that sound similar to your question. It finds fragments about board meetings, fragments about voting, fragments about abstentions — but no single chunk contains the complete answer. A knowledge graph, however, maps members, meetings, votes, and abstentions as explicit relationships. It can traverse these connections to give a precise answer. This is the power of Graph RAG.

The naive RAG approach stores knowledge as independent vector chunks with no relationships between them. This works well for simple, factoid questions ("Who is the CEO?") but fails for complex queries that require reasoning across multiple pieces of information.

Graph RAG addresses this by storing the knowledge base as a graph structure — nodes (entities) connected by edges (relationships) — in addition to or instead of a flat vector store.

14.7.1 Knowledge Graphs as a Knowledge Base

Knowledge graph — a structured representation of information as nodes (entities) and edges (relationships). Each node represents a real-world entity (a person, place, concept, or thing), and each edge represents a typed relationship between two entities.

Example from earlier lectures on ontologies:

  • Nodes: "Obama," "Honolulu," "USA"
  • Edges: "Obama —born_in→ Honolulu," "Honolulu —located_in→ USA"

From this graph, we can infer: "Obama was born in USA" — a fact not explicitly stated but derivable through graph traversal (following two edges).

Intuition — The difference between a filing cabinet and a mind map: A vector store is like a filing cabinet — each document is an independent card, and you find relevant cards by how similar they look. A knowledge graph is like a mind map — entities are nodes, and relationships are lines connecting them. You can follow the lines to discover connections that no single card would reveal. The filing cabinet is faster to set up; the mind map is more powerful for complex questions.

In a flat vector store, multi-hop inference is difficult. The chunks containing "Obama born in Honolulu" and "Honolulu located in USA" might be stored far apart, and cosine similarity may not connect them — because the query "Which country is Obama from?" does not lexically resemble either chunk. The graph structure makes these relationships explicit and traversable.

14.7.2 Hybrid Architecture

In practice, Graph RAG often combines both storage types:

Hybrid Graph RAG architecture:

  1. Vector data store (e.g., Qdrant) — stores chunks as embeddings for fast semantic similarity search. Good for unstructured text and topical queries.
  2. Graph data store (e.g., Neo4j) — stores entities and relationships as a graph for structured reasoning. Good for multi-hop queries and relationship traversal.

When a query arrives:

  • Both stores are searched in parallel.
  • Context from both the vector store (semantic matches) and the graph store (related entities and relationships) are combined.
  • The combined context is appended to the user's prompt and sent to the LLM.

Worked example — Multi-hop reasoning in a knowledge graph:

Query: "Which country is Obama from?"

Graph traversal:

  1. Find node "Obama"
  2. Follow edge "born_in" → node "Honolulu"
  3. Follow edge "located_in" → node "USA"
  4. Infer: Obama is from USA

Why vector search struggles here: The query "Which country is Obama from?" does not contain the word "Honolulu." A vector search might find chunks about Obama (similarity high with "Obama") but the chunk saying "Obama was born in Honolulu" does not mention "country" or "USA." The two-hop connection (Obama → Honolulu → USA) requires following relationships, which is exactly what a graph does naturally.

Why graph search succeeds: The knowledge graph explicitly encodes the born_in and located_in relationships. The graph traversal engine follows these edges without needing the query to contain the intermediate entity names.

Real-world: Qdrant is popular for vector storage because it is fast and free. Neo4j is the industry standard for graph-based storage in RAG applications.

14.7.3 Storing Graph Information

Two ways to store graph nodes:

  1. String literals — the node name and description stored as text (e.g., "myocardial infarction: same as heart attack"). Enables keyword-based search.
  2. Vector embeddings — each node's description is embedded as a vector, using the same embedding model as the rest of the system. Enables semantic search.

In Neo4j, both are stored together. The node has a text field for its name/description and a vector field for its embedding. This allows both keyword-based and semantic search on graph nodes.

Worked example — Medical knowledge graph node:

A medical knowledge graph stores "myocardial infarction" as a node. Its description:

"A condition where blood flow to the heart muscle is blocked, also known as heart attack. Symptoms include chest pain, shortness of breath, and sweating. Treatment includes aspirin, angioplasty, and bypass surgery."

This description is encoded as a single vector (e.g., 768-dimensional) using the same Sentence Transformer model used for all other embeddings.

Related nodes:

  • "chest pain" — connected by edge symptom_of
  • "aspirin" — connected by edge treated_by
  • "angioplasty" — connected by edge treated_by
  • "heart attack" — connected by edge same_as

Each edge is typed (has a label) and directional (points from one entity to another).

When a user asks "What are the symptoms of a heart attack?", the system can:

  1. Find the "heart attack" node (via keyword or semantic search)
  2. Follow same_as → "myocardial infarction"
  3. Follow symptom_of edges → "chest pain," "shortness of breath," "sweating"
  4. Return these as the grounded answer

When a new document discusses the same condition, its information is integrated into the existing node. If the graph is constructed automatically using LLMs, duplicate records may appear. These can be cleaned using:

  • Reasoners — automated tools that detect and merge duplicate nodes based on semantic similarity and explicit same_as relationships.
  • Human-in-the-loop — manual review and cleanup, especially for high-stakes domains like medicine.
  • Ready-made knowledge graphs — high-quality, pre-built domain-specific graphs (e.g., Hetionet for biomedical data, KEGG for metabolic pathways) that can be used directly, avoiding the duplication problem entirely.

14.7.4 Multi-Hop Reasoning Example

Multi-hop traversal — Obama's country:

Query: "Which country is Obama from?"

Step 1 — Node lookup: Find node "Obama" in the graph.

Step 2 — First hop: Follow the born_in edge from "Obama" → "Honolulu."

Step 3 — Second hop: Follow the located_in edge from "Honolulu" → "USA."

Step 4 — Inference: The answer is "USA." This is a two-hop inference — the fact "Obama is from the USA" is not explicitly stored anywhere in the graph. It is derived by traversing two edges.

Why this matters: In a flat vector store, the chunks containing "Obama born in Honolulu" and "Honolulu located in USA" might be stored far apart. Cosine similarity between the query and each chunk individually might not be high enough to retrieve both. The graph structure makes the connection explicit and traversable — no similarity threshold needed.

Pitfall — Assuming graphs are always better: Graph RAG is more powerful for multi-hop queries, but it is also more complex and expensive to build and maintain. For simple factoid questions ("What is the company's phone number?"), a vector store is faster, cheaper, and perfectly adequate. Use graphs when the query complexity demands it — not as a default.

Q: Do we also store the content (descriptions) in the graph database? A: Yes — each node can have description information. For example, "myocardial infarction" can have its symptoms, causes, and treatment stored as part of the node. All that information is encoded as a single vector embedding for that node. This allows both text-based and semantic search on the node's content.

Q: When a new document about the same topic arrives, how is it integrated? A: The document ID is linked to the existing node. All information is integrated into a single node. If the graph is constructed automatically, duplicates may appear — these can be cleaned using reasoners or human review.

Q: Is there a performance difference between vector search and graph search? A: Graph search provides richer contextual information because it traverses relationships between nodes. Vector search relies on chunk overlap for context. Generally, a well-constructed graph yields better contextual retrieval, but it requires more setup and is more expensive. The companion doc notes: graph gives related information of edges also, providing more valid contextual information among nodes.

Recap: Graph RAG stores knowledge as a graph of entities (nodes) and relationships (edges), enabling multi-hop reasoning that flat vector stores cannot achieve. A hybrid architecture combines vector search (for topical relevance) with graph traversal (for relational reasoning). Graph nodes store both text descriptions and vector embeddings. Graph RAG is more powerful but also more complex and expensive — use it when query complexity demands it.

14.8 When to Use Naive RAG vs. Graph RAG

Hook: You are designing a RAG system for your company. Should you use a simple vector store, a full knowledge graph, or both? The answer depends on one question: How complex are your users' queries? If they ask "What is the refund policy?" — a vector store is enough. If they ask "Which suppliers had delivery delays on orders over ₹10 lakhs in Q3 that affected our top-5 revenue products?" — you need a graph.

The choice between naive (vector) RAG and Graph RAG depends on the application's complexity:

Naive RAG vs. Graph RAG — side-by-side comparison:

Criterion Naive RAG (Vector) Graph RAG
Query complexity Simple, factoid questions ("Who is the CEO?") Complex, multi-hop questions ("Which board meeting in the last 12 months had at least two members abstain from voting?")
Data format Unstructured text Structured graph (nodes + edges)
Retrieval method Cosine similarity on vector embeddings Graph traversal, graph neural networks, Cypher queries
Reasoning Limited — each chunk is independent Multi-hop inference across connected entities
Explainability Low — embeddings are opaque numbers High — relationships and sources are explicit paths
Hallucination risk Higher — no structural grounding Lower — graph constraints reduce fabrication
Integration complexity Easy, fast, cheap More complex, requires graph construction
Industry adoption ~80% of production projects Domain-specific projects (medical, legal, corporate governance)

The 80/20 rule of RAG: The professor states that approximately 80% of production RAG projects use naive vector RAG because it is cheaper, faster, and easier to develop. Graph RAG is used when the application demands complex reasoning, high recall, or strong explainability — particularly in medical, legal, and corporate governance domains. Start with naive RAG; upgrade to Graph RAG only when the query complexity justifies the added cost and complexity.

Pitfall — Over-engineering with Graph RAG: The professor's warning: more advanced techniques always involve more cost and time — ask whether the application truly needs the complexity. Building a knowledge graph when a vector store would suffice wastes engineering effort and budget. The decision should be driven by the queries, not by the technology's novelty.

Cypher — the query language used by Neo4j for graph traversal. Cypher is designed specifically for graph databases and is more similar to natural language than SQL, making it easier to express graph traversal patterns.

Example Cypher query:

MATCH (p:Person)-[:BORN_IN]->(c:City)-[:LOCATED_IN]->(country:Country)
WHERE p.name = "Obama"
RETURN country.name

This query reads almost like English: "Find a person named Obama who was born in a city that is located in a country, and return the country's name."

Modern LLMs can convert natural language queries into Cypher queries automatically, so developers do not need to learn Cypher syntax — the LLM handles the translation. This is itself an example of LLMs enhancing knowledge graphs.

Real-world: In the medical domain, Neo4j is very commonly used because medical data involves many interconnections between diseases, symptoms, treatments, and drugs. Heterogeneous data from different sources (hospital records, clinical trials, drug databases) can be integrated into a single knowledge graph, enabling queries that span multiple data sources.

Recap: Naive RAG (vector) handles ~80% of production use cases — it is simple, cheap, and fast. Graph RAG is needed for multi-hop reasoning, high explainability, and domain-specific structured data (medical, legal, governance). The choice is driven by query complexity, not technology preference. Cypher is Neo4j's graph query language, and LLMs can translate natural language to Cypher automatically.

14.9 Synergy Between LLMs and Knowledge Graphs

Hook: LLMs and knowledge graphs are often presented as competing approaches — one is neural, the other is symbolic; one generates text, the other stores facts. But in practice, they are complementary. Each fills the other's blind spots. Together, they are more powerful than either alone.

14.9.1 Bidirectional Integration

The symbiotic relationship — LLMs and knowledge graphs enhance each other in both directions:

Knowledge graphs enhance LLMs (retrieval direction):

  • Provide factual grounding for RAG, reducing hallucination.
  • Enable multi-hop reasoning that flat vector stores cannot.
  • Improve explainability — the graph shows exactly which relationships led to the answer, making the LLM's reasoning transparent.

LLMs enhance knowledge graphs (extraction direction):

  • Automate knowledge graph construction from text documents using entity extraction and relation extraction. An LLM can read a medical paper and extract the triplet (EGF, upregulate, miR-31) to populate a graph node.
  • Convert natural language queries into Cypher queries for graph databases, removing the need for developers to learn Cypher syntax.
  • Generate and populate graph nodes from unstructured text, dramatically reducing the manual effort of graph construction.

Intuition — The perfect couple: The companion doc describes KGs and LLMs as "the perfect couple." Knowledge graphs provide what LLMs lack: structured, factual, interpretable knowledge. LLMs provide what knowledge graphs lack: the ability to understand natural language, generalize across domains, and generate human-readable text. Each compensates for the other's weakness.

Worked example — LLM-powered graph construction:

A medical research paper contains the sentence: "EGF up-regulates miR-31 in breast cancer cell lines."

An LLM reads this sentence and extracts the structured triplet:

  • Subject: EGF (Epidermal Growth Factor)
  • Relation: upregulates
  • Object: miR-31 (microRNA-31)
  • Context: breast cancer cell lines

This triplet is pushed into a Neo4j graph database, creating:

  • Node: "EGF" (type: protein)
  • Node: "miR-31" (type: microRNA)
  • Edge: "EGF —upregulates→ miR-31" (context: breast cancer)

Thousands of papers can be processed this way, automatically building a comprehensive knowledge graph without manual annotation.

This bidirectional integration is an active area of research, with applications in:

  • Explainable AI (XAI) — using knowledge graphs to provide transparent reasoning trails for black-box model outputs. In defense, medicine, and finance, AI must answer: "Why did you make this decision? When can I trust you?" Knowledge graphs provide the explicit reasoning chain.
  • Federated learning — domain-specific knowledge graphs for Indian languages and multilingual RAG systems.
  • Search engines and dialogue systems — synergized LLMs + KGs where data representation learning and neural-symbolic reasoning work together in a continuous loop.

Recap: LLMs and knowledge graphs have a bidirectional symbiotic relationship. KGs enhance LLMs by providing factual grounding, multi-hop reasoning, and explainability. LLMs enhance KGs by automating graph construction, translating natural language to Cypher, and populating nodes from text. This integration is an active research frontier with applications in XAI, federated learning, and multilingual systems.

14.10 Multimodal RAG

Hook: Your company's annual report is not just text — it contains charts showing quarterly revenue trends, satellite images of factory sites, and tables of financial data. A text-only RAG system can answer "What was the total revenue?" but cannot answer "Which product line had the steepest growth curve in Q3 according to this chart?" Multimodal RAG extends the framework to see, hear, and read — not just parse plain text.

Multimodal RAG extends the RAG framework to handle non-text data — images, tables, charts, audio, and video.

14.10.1 The Need for Multimodal RAG

Much real-world information is not plain text. PDFs contain photographs, satellite images, graphs, charts, and tables. A tool like PyPDF can extract text but often fails on tables and images — it may extract them as image files without understanding their content.

Queries that require multimodal understanding:

  • "Which shares performed best in Q3 according to this chart?" — requires reading a bar/line chart
  • "Which product looks most like this picture?" — requires image similarity
  • "Summarize the key points from yesterday's meeting recording" — requires audio processing
  • "What were the Q3 revenue trends in this table?" — requires table extraction

14.10.2 The Core Idea

The strategy is the same as text RAG, but the embedding step is more complex:

Multimodal RAG pipeline:

  1. Ingestion — convert each modality into vector embeddings. Images may use vision-language models; audio may be converted to text first using speech-to-text.
  2. Storage — store embeddings in a vector or graph data store.
  3. Query — embed the query (text, image, or voice) and compute similarity against stored embeddings.
  4. Retrieval and Generation — retrieve top-K matches and send to a multimodal LLM for answer generation.

Intuition — Same recipe, different ingredients: The RAG recipe (retrieve → augment → generate) does not change. What changes is the embedding step. Instead of only converting text to vectors, you also convert images, audio frames, and table structures to vectors. Once everything is in the same vector space, the same cosine similarity search works across all modalities.

14.10.3 Modality-Specific Approaches

How each modality is handled:

Modality Ingestion Approach Query Approach
Audio Convert speech to text using speech-to-text, then use standard text RAG pipeline Text query matched against transcribed text chunks
Images Use vision-language models to extract features and metadata; store image embeddings Image or text query matched against stored image embeddings
Tables/Charts Specialized extraction tools (active research area) Text query matched against extracted table data
Video Treat as sequence of frames; apply image-based techniques with temporal context Text or image query matched against frame embeddings

Audio: Meeting recordings, voice notes, and audio files are converted to text using high-quality speech-to-text converters, then processed through the standard text RAG pipeline. For output, text can be converted back to speech using text-to-speech.

Images: Vision-language models extract features and metadata from images. Image embeddings are stored alongside text embeddings. For queries like "Which product looks like this picture?", cosine similarity is computed between the query image embedding and stored image embeddings.

Tables and Charts: These are the hardest to process. Current tools have challenges extracting meaningful data from complex table layouts and charts. This is an active area of research.

Video: Treated as a sequence of images (frames). The same image-based techniques apply, but with the additional challenge of temporal relationships between frames — understanding what happens over time, not just in a single snapshot.

14.10.4 Multimodal Vision Embeddings

Shared embedding space: For images and text to be searchable together, their embeddings must be in a shared vector space. The image of a dog and the text "a golden retriever playing in the park" should produce similar vectors. Multimodal vision embedding models (like those from Gemini) are designed to create such shared representations — they map images and text into the same coordinate system.

Real-world: Gemini is the most powerful and popular family of models for multimodal tasks, excelling at processing images, video, audio, and text together. GPT-4o is another strong multimodal model.

Q: Is the technique for multimodal RAG the same as for text, like chunking? A: Chunking is the same concept, but for visual data there are vision-language models that handle the embedding differently. For speech, audio is converted to text first. Similar techniques apply to some parts but not all modalities.

Recap: Multimodal RAG extends RAG to images, audio, video, and tables using the same retrieve-augment-generate pipeline. The key difference is the embedding step: each modality uses specialized models (speech-to-text for audio, vision-language models for images). All embeddings must be in a shared vector space for cross-modal search. Gemini leads in multimodal capabilities.

14.11 Agentic RAG

Hook: Standard RAG is a passive pipeline — the system retrieves, augments, and generates, every time, in the same way. But what if the system could think about the query first? What if it could decide "This part of the question needs my company's documents, but that part needs a web search, and I should check my answer before presenting it"? That is Agentic RAG — RAG with an autonomous agent at the helm.

14.11.1 Agent Capabilities and Orchestration

Agentic RAG represents the next evolution, where an AI agent — not just a passive retriever — orchestrates the RAG pipeline with intelligence and autonomy.

What makes an agent different from a plain LLM:

An agentic AI system has four capabilities beyond a plain LLM:

  • Memory — it remembers past interactions and context across turns, building a persistent understanding of the user's needs.
  • Tools — it can call external APIs, databases, web search engines, calculators, and other services. It is not limited to text generation.
  • Planning — it breaks complex queries into sub-tasks, decides the order of operations, and allocates resources.
  • Action — it can execute actions in the real world — send emails, update databases, trigger workflows — not just generate text.

How Agentic RAG works:

In agentic RAG, the agent:

  1. Analyzes the user's query and decomposes it into sub-questions.
  2. For each sub-question, intelligently decides which data source to use: the vector store, the graph store, an external web API, or its own general knowledge.
  3. Some sub-questions may not need proprietary data at all — the agent decides to use general knowledge or web search instead.
  4. Combines results from multiple sources and generates a comprehensive, well-cited answer.
  5. If the first retrieval attempt returns low-quality results, the agent refines the search and tries again — it iterates until it has sufficient information.

Intuition — From assembly line to project manager: Standard RAG is an assembly line: every query goes through the same steps in the same order. Agentic RAG is a project manager: it looks at the query, decides who to ask (vector store? graph? web? calculator?), breaks the work into tasks, checks the results, and iterates if needed. The agent has agency — it makes decisions, not just follows a fixed recipe.

Worked example — Agentic RAG for a complex query:

User query: "Compare the environmental impact of nuclear vs. solar energy globally in 2024."

Agent's plan:

  1. Decompose into sub-queries:
  • "What is the global nuclear energy output in 2024?" → search the web (up-to-date data needed)
  • "What is the global solar energy output in 2024?" → search the web
  • "What are the environmental impacts of nuclear energy?" → search the vector store (company research docs)
  • "What are the environmental impacts of solar energy?" → search the vector store
  • "Convert both to the same units for comparison" → use a calculator tool
  1. Execute each sub-query with the appropriate tool.
  1. Synthesize results into a structured comparison with citations.
  1. If the web search for "nuclear output 2024" returns incomplete data, refine the search query and try again.
  1. Present the final comparison with source citations.

Real-world: Agentic RAG systems also use LLM-as-RAG — one LLM can use another LLM as its knowledge source, generating and appending information from one model to feed into another. This "meta-search" approach is increasingly used in production systems.

Scope — Current state of Agentic RAG: The professor notes that agentic RAG, along with re-ranking, evaluation measures, and variable-length chunks, are advanced topics that will be covered in the next semester's NLP applications course. This section provides an overview of the concept and its capabilities.

Recap: Agentic RAG adds an autonomous agent on top of the RAG pipeline. The agent decomposes queries, selects data sources, iterates on poor results, and synthesizes multi-source answers. It has memory, tools, planning, and action capabilities. This represents the evolution from passive retrieval to intelligent orchestration.

14.12 Tools and Technologies

Hook: Knowing the theory of RAG is essential, but so is knowing the tools. In production, you will use specific libraries and platforms for each stage of the pipeline. This section is your reference card — the named tools the professor expects you to know for the exam and for real-world implementation.

14.12.1 Vector Data Stores

Vector databases for RAG:

Tool Type Key Property
Qdrant Open source Fast, free; most popular choice for naive RAG
Pinecone Managed (cloud) Fully managed; no infrastructure to maintain
FAISS Library (Facebook) High-performance similarity search; not a full database
Weaviate Open source Vector search engine with built-in vectorization
Milvus Open source Purpose-built vector database for AI applications

14.12.2 Graph Data Stores

  • Neo4j — the industry standard for graph-based RAG. Uses Cypher query language. Very common in medical and legal domains. Supports both text and vector embeddings on nodes.

14.12.3 Embedding Models

Embedding models for converting text to vectors:

Model Type Notes
Hugging Face Sentence Transformers Open source Widely used; many model variants; free
Cohere Commercial API High-quality embeddings; paid
OpenAI Embeddings Commercial API High quality; paid; easy integration
BGE (BAAI General Embedding) Open source Strong performance; free

14.12.4 LLMs for RAG

Language models used as the generation component:

Model Type Notes
GPT-4 Proprietary High quality, expensive, large context window
Claude Proprietary Very large context window (up to 1M tokens)
Qwen Open source Smaller; suitable for local deployment; keeps data in-house
Llama Open source Widely used; multiple sizes available
Gemini Proprietary Best for multimodal tasks (images, video, audio)

14.12.5 Document Processing

  • PyPDF — PDF text extraction. Extracts text from PDF files but may struggle with tables and images.
  • Spacy — NLP pipeline for entity extraction, dependency parsing, and tokenization. Used in graph construction.

14.12.6 Graph Construction

  • LLMs (Gemini, Qwen, Llama) — can automatically construct knowledge graphs from text by extracting entities and relations.
  • Dependency parsers (Spacy) — extract entities and relations from text using syntactic analysis.

14.12.7 Research References

  • Dragon — a research model for joint pre-training on text and knowledge graphs for retrieval and generation. These are supplementary references, not required for the exam.
  • Cypher is the query language used by Neo4j for graph traversal.

Pitfall — Confusing the tools: On the exam, be clear about which tool does what. Qdrant and Pinecone are for vector storage. Neo4j is for graph storage. Hugging Face Sentence Transformers is for embedding (converting text to vectors). PyPDF is for document extraction (converting PDFs to text). Mixing these up is a common source of errors.

Recap — Named tools to know: Qdrant, Pinecone, FAISS, Weaviate, Milvus (vector stores). Neo4j (graph store, Cypher). Hugging Face Sentence Transformers, Cohere, OpenAI Embeddings, BGE (embedding models). GPT-4, Claude, Qwen, Llama, Gemini (LLMs). PyPDF, Spacy (document processing). Gemini excels at multimodal tasks.

Exam Guidance Summary

Exam note: RAG is one of the most important architectures in modern NLP — every AI project today typically implements some form of RAG. The professor expects you to be comfortable with both the conceptual framework and the calculations.

  • Understand the three stages: retrieval, augmentation, generation.
  • Know the limitations of LLMs that RAG addresses: knowledge cutoff, hallucination, lack of specificity.
  • Be able to work through token budgeting calculations: available tokens = context window − system prompt − query; max chunks = floor(available / chunk size).
  • Be able to work through chunking calculations: total chunks = ceiling(N / (S − O)), where stride = S − O.
  • Understand chunk overlap and stride — why overlap exists and how it prevents information loss.
  • Know the latency components (embedding, retrieval, re-ranking, generation) and how to calculate remaining time for optional steps like re-ranking.
  • Understand the difference between naive RAG and Graph RAG — when to use each. Graph RAG enables multi-hop reasoning, better explainability, and lower hallucination, but is more complex and expensive.
  • Multimodal RAG extends the framework to images, audio, video — same core idea, different embedding techniques.
  • Advanced topics (re-ranking, evaluation measures, agentic RAG, variable-length chunks) will be covered in the next semester's NLP applications course.
  • Named tools to know: Qdrant, Neo4j, Pinecone, FAISS, Weaviate, Hugging Face Sentence Transformers, PyPDF, Qwen, Llama, Gemini.

Common exam mistakes to avoid:

  • Confusing ceiling (chunking) with floor (token budgeting).
  • Forgetting that the same embedding model must be used everywhere.
  • Not knowing that the context window is per session, not per prompt.
  • Mixing up vector stores (Qdrant) with graph stores (Neo4j).
  • Thinking RAG only works with proprietary models — it works with any LLM.

Key Industry Applications

The professor's observation: RAG is like databases in traditional IT — today, every AI project implements some form of RAG. It has become the standard architecture for grounding LLM responses in real data.

  • Medical domain — ~90% of medical AI projects use RAG. Knowledge graphs with Neo4j are very common due to the interconnected nature of medical data (diseases, symptoms, treatments, drugs). Systems like Hetionet and KEGG integrate heterogeneous medical data into unified knowledge graphs.
  • Legal domain — RAG for document analysis, contract review, and regulatory compliance. Lawyers use RAG to search thousands of contracts for specific clauses without reading each one.
  • Engineering — uploading papers and product documentation for comparison and analysis. Engineers use RAG to query technical specifications across product lines.
  • Recommendation systems — RAG architecture applied to personalized recommendations, retrieving relevant user history and product information.
  • Question answering and conversational AI — grounding chatbot responses in proprietary knowledge. Customer support bots use RAG to answer questions from product manuals and support tickets.
  • Explainable AI (XAI) — using knowledge graphs to provide transparent reasoning for black-box model outputs. In defense, medicine, and finance, AI must explain its decisions.
  • Social good — research on using RAG and knowledge graphs for societal benefit, including the Human Trafficking domain (KGSG project) for building entity-centric search engines.
  • Education — RAG for student query systems, automated tutoring, and course material Q&A.
  • Finance — querying financial reports, quarterly trends from charts and tables. Multimodal RAG is especially valuable here for processing charts and graphs in annual reports.
  • Multilingual systems — developing RAG and knowledge graphs for Indian languages and federated learning scenarios, making AI accessible across language barriers.

NLP Lecture 14 notes · Retrieval Augmented Generation

Natural Language Processing· postgraduate· 2026-08-09

Sections Breakdown

114.1 Limitations of Large Language Models

The three fundamental LLM limitations — knowledge cutoff, hallucination, and lack of specificity — and why fine-tuning fails where RAG succeeds.

214.2 RAG Pipeline — Retrieval, Augmentation, Generation

The two-phase RAG pipeline: offline indexing (extract, chunk, embed, store) and online querying (embed, search, augment, generate), with a company travel policy chatbot worked example.

314.3 Vector Embeddings and Semantic Search

How word embeddings capture meaning, the cosine similarity formula, the same-model consistency rule, and sparse, dense, and hybrid retrieval.

414.4 Chunking Strategies

Chunk size, overlap, and stride; total chunk count = ceiling(N / (S - O)); variable-length chunks and who decides chunk size.

514.5 Token Budgeting and the Context Window

Context window composition, floor-based token budgeting worked example, multi-turn sessions, and the dual constraint of similarity threshold plus token budget.

614.6 Latency in the RAG Pipeline

The four latency components, the latency budget worked example, and the quality-cost-time triangle.

714.7 Graph RAG

Knowledge graphs as a knowledge base, hybrid vector-graph architecture, storing node descriptions and embeddings, and multi-hop reasoning.

814.8 When to Use Naive RAG vs. Graph RAG

Side-by-side comparison of naive and graph RAG, the 80/20 rule, and Cypher queries for graph search.

914.9 Synergy Between LLMs and Knowledge Graphs

Bidirectional integration: knowledge graphs grounding LLMs, and LLMs automating graph construction and NL-to-Cypher translation.

1014.10 Multimodal RAG

Extending RAG to images, audio, video, and tables with modality-specific embeddings in a shared vector space.

1114.11 Agentic RAG

Autonomous agents with memory, tools, planning, and action orchestrating retrieval across multiple sources.

1214.12 Tools and Technologies

Reference card: vector data stores, graph data stores, embedding models, LLMs, and document processing tools.

13Exam Guidance Summary

Exam-ready revision of RAG stages, calculations, and the named tools to know.

14Key Industry Applications

Real-world RAG adoption across medical, legal, engineering, finance, education, and multilingual systems.

Postgraduate students in Natural Language Processing

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Limitations of Large Language Models

Must-know: Three LLM limitations: knowledge cutoff, hallucination, lack of specificity. RAG addresses all three by retrieving external knowledge at inference time without retraining.

⚠️ Top pitfall: Confusing LLM fluency with accuracy — a hallucinated answer sounds just as confident as a correct one.

Self-check: Name the three fundamental limitations of LLMs that RAG addresses.

Connects to: 14.2 RAG Pipeline.

RAG Pipeline — Retrieval, Augmentation, Generation

Must-know: Three RAG stages: Retrieval (find relevant chunks), Augmentation (append chunks to prompt), Generation (LLM produces grounded answer). Two phases: offline indexing and online querying.

⚠️ Top pitfall: Using different embedding models for indexing vs. querying produces incompatible vectors and meaningless similarity scores.

Self-check: List the four steps of the RAG online query phase.

Connects to: 14.1 Limitations of Large Language Models; 14.3 Vector Embeddings and Semantic Search; 14.4 Chunking Strategies.

Vector Embeddings and Semantic Search

Must-know: Cosine similarity formula: dot product divided by product of norms. Same embedding model must be used for all vectors. Score range: -1 to 1, higher = more similar.

\[ \text{similarity}(\mathbf{e}_Q, \mathbf{e}_C) = \frac{\mathbf{e}_Q \cdot \mathbf{e}_C}{\|\mathbf{e}_Q\| \cdot \|\mathbf{e}_C\|} \]

⚠️ Top pitfall: Mixing embedding models produces incompatible vectors — different models have different dimensionalities and different meaning spaces.

Self-check: What is the range of cosine similarity, and what does a score of 1 mean?

Connects to: 14.2 RAG Pipeline; 14.4 Chunking Strategies.

Chunking Strategies

Must-know: Chunking formula: chunks = ceiling(N / (S - O)). Stride = S - O. Overlap preserves boundary information. Chunk size is a hyperparameter (typically 400-600 tokens).

\[ \text{chunks} = \left\lceil \frac{N}{S - O} \right\rceil \]

⚠️ Top pitfall: Using ceiling for chunk count (partial chunks still count) vs. floor for token budgeting (must not exceed context window).

Self-check: A document has 18,200 tokens. With chunk size 600 and overlap 120, how many chunks are produced?

Connects to: 14.3 Vector Embeddings and Semantic Search; 14.5 Token Budgeting and the Context Window.

Token Budgeting and the Context Window

Must-know: Token budgeting: T_available = C - T_sys - T_query. Max chunks = floor(T_available / T_chunk). Floor because exceeding causes truncation. Dual constraint: similarity threshold AND token budget.

\[ T_{\text{available}} = C - T_{\text{sys}} - T_{\text{query}}, \quad \text{max chunks} = \left\lfloor \frac{T_{\text{available}}}{T_{\text{chunk}}} \right\rfloor \]

⚠️ Top pitfall: Confusing ceiling (chunking) with floor (token budgeting). Padding with low-similarity chunks introduces noise and causes hallucination.

Self-check: With context window 8192, system prompt 200, query 320, and chunk size 512, how many chunks fit?

Connects to: 14.4 Chunking Strategies; 14.6 Latency in the RAG Pipeline.

Latency in the RAG Pipeline

Must-know: T_total = T_embed + T_retrieval + T_gen + T_rerank. Re-ranking is optional. If T_allowed - T_embed - T_retrieval - T_gen < T_rerank_needed, skip re-ranking. Generation time is typically the largest component.

\[ T_{\text{total}} = T_{\text{embed}} + T_{\text{retrieval}} + T_{\text{gen}} + T_{\text{rerank}} \]

⚠️ Top pitfall: Forgetting that re-ranking precedes generation in the pipeline — the decision to include it must be based on predicted generation time, not actual.

Self-check: Given T_allowed = 2500ms, T_embed = 200ms, T_retrieval = 370ms, T_gen = 1600ms, how much time remains for re-ranking?

Connects to: 14.5 Token Budgeting and the Context Window; 14.7 Graph RAG.

Graph RAG

Must-know: Graph RAG enables multi-hop reasoning via graph traversal. Hybrid architecture: vector store (Qdrant) + graph store (Neo4j). Graph nodes have text descriptions and vector embeddings. More powerful but more complex and expensive than naive RAG.

⚠️ Top pitfall: Assuming graphs are always better — for simple factoid queries, vector stores are faster and cheaper.

Self-check: Explain why vector search fails on the query "Which country is Obama from?" while graph search succeeds.

Connects to: 14.3 Vector Embeddings and Semantic Search; 14.8 When to Use Naive RAG vs. Graph RAG.

When to Use Naive RAG vs. Graph RAG

Must-know: ~80% of production RAG uses naive vector RAG. Graph RAG for complex multi-hop queries, high explainability, domain-specific data (medical, legal). Cypher is Neo4j's query language.

⚠️ Top pitfall: Over-engineering with Graph RAG when a vector store would suffice. Let query complexity drive the choice.

Self-check: Name three criteria that would make you choose Graph RAG over naive RAG.

Connects to: 14.7 Graph RAG; 14.9 Synergy Between LLMs and Knowledge Graphs.

Synergy Between LLMs and Knowledge Graphs

Must-know: Bidirectional integration: KGs enhance LLMs (grounding, reasoning, explainability) and LLMs enhance KGs (automated construction, NL-to-Cypher, node population).

⚠️ Top pitfall: None specific — this is a conceptual overview section.

Self-check: Give one example of how an LLM can enhance a knowledge graph.

Connects to: 14.7 Graph RAG; 14.8 When to Use Naive RAG vs. Graph RAG.

Multimodal RAG

Must-know: Multimodal RAG handles images, audio, video, tables. Audio → speech-to-text → text RAG. Images → vision-language models → shared embedding space. Tables/charts are hardest (active research). Gemini is the leading multimodal model.

⚠️ Top pitfall: Assuming text-only RAG can handle charts and tables — PyPDF extracts images but cannot understand their content.

Self-check: How does multimodal RAG handle audio data?

Connects to: 14.2 RAG Pipeline; 14.3 Vector Embeddings and Semantic Search.

Agentic RAG

Must-know: Agentic RAG: agent decomposes queries into sub-questions, selects appropriate data sources per sub-question, iterates on poor results, synthesizes with citations. Four capabilities: memory, tools, planning, action.

⚠️ Top pitfall: None specific — this is an overview section. Advanced topics (re-ranking, evaluation, agentic RAG) are covered next semester.

Self-check: What four capabilities distinguish an agentic AI system from a plain LLM?

Connects to: 14.2 RAG Pipeline; 14.7 Graph RAG.

Tools and Technologies

Must-know: Named tools: Qdrant (vector), Neo4j (graph, Cypher), Hugging Face Sentence Transformers (embedding), PyPDF (PDF extraction), Gemini (multimodal), Qwen/Llama (open-source LLMs).

⚠️ Top pitfall: Confusing which tool does what — vector stores vs. graph stores vs. embedding models vs. document processors.

Self-check: Which tool is the industry standard for graph-based RAG, and what query language does it use?

Connects to: 14.2 RAG Pipeline; 14.3 Vector Embeddings and Semantic Search; 14.7 Graph RAG.

Exam Guidance Summary

Must-know: Three RAG stages. Three LLM limitations. Token budgeting: floor. Chunking: ceiling. Latency: 4 components. ~80% production RAG is naive vector. Named tools: Qdrant, Neo4j, Hugging Face, PyPDF, Gemini.

⚠️ Top pitfall: Confusing ceiling (chunking) with floor (token budgeting). Mixing embedding models. Confusing vector stores with graph stores.

Self-check: What is the formula for token budgeting — ceiling or floor? Why?

Connects to: 14.1 Limitations of LLMs; 14.2 RAG Pipeline; 14.4 Chunking Strategies; 14.5 Token Budgeting; 14.6 Latency; 14.7 Graph RAG; 14.10 Multimodal RAG; 14.12 Tools and Technologies.

Key Industry Applications

Must-know: ~90% of medical AI uses RAG with Neo4j knowledge graphs. Finance uses multimodal RAG for charts. Multilingual RAG for Indian languages.

Self-check: Which industry has ~90% adoption of RAG, and what graph tool is commonly used?

Connects to: 14.7 Graph RAG; 14.10 Multimodal RAG.

Previous LectureLecture 13 · Knowledge Graphs and Semantic Web
Next LectureLecture 01 · Software Engineering for Machine Learning — Foundation and Overview

Was this lecture useful?

Loading comments…
🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Key

Select Provider & API Key
🔑 Enter API key above to fetch live models from provider, or enter model name manually.
OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.