Information Retrieval

Introduction to Information Retrieval

Published: 2026-09-13

Level: undergraduate

Audience: Undergraduate students studying Information Retrieval

1.1 What Information Retrieval Is

1.1.1 A Working Definition

What if you ask for one thing and get back a pile ordered from best to least helpful? That is the promise of this field.

An information need (a gap in what a person knows, for example "what treats this illness") drives everything. The need turns into a short query (the words typed or spoken, for example "lung cancer biomarkers"). The system searches a corpus (the full set of stored documents it can search, for example all papers, pages, and mails). It returns a ranked list (an ordered output with the most helpful item first).

An information need is a gap in what a person knows. The person wants to fill that gap from a large store of material. An information retrieval system takes that need in the form of a query and returns a ranked list of items that help fill the gap. A query is the short text a user types or speaks. It can be keywords or a short phrase. A document is any stored item that can be returned. It can be a web page, a research paper, an email, a file, an image, or a video. A corpus is the full set of documents the system can search. A ranked list is an ordered output where the item judged most helpful appears first.

The core idea stays the same across all examples. Start from the query. End with relevant material ordered by how well each item fits the query. In words: the relevance score of a document $d$ for a query $q$ . As LaTeX: $scor e (d, q)$ , where $d$ is one document from the corpus and $q$ is the user query and the value grows when the match is stronger. A close variant says the same thing with shorter symbols. In words: relevance of $d$ given $q$ . As LaTeX: $r e l (d, q)$ , where $r e l$ is a relevance function and $d$ is a document and $q$ is a query.

A relevance score (how well one document fits one query, written $scor e (d, q)$ , for example $scor e (d, q) = 0.83$ ) is the central number in this course. Here $d$ is a single document from the corpus. Here $q$ is the user query. The output is a plain number. A larger number means a stronger match. The system sorts all scored documents from high to low and shows that order to the user. A close shorthand is $r e l (d, q)$ , where $r e l$ names the same relevance function. Both forms say the same thing: fit of document to query decides rank.

Think of the input-output deal in plain steps. The input is a query. The expected output is relevant documents in rank order. The system never hands back the whole store. It filters the store to a short candidate set and then orders that set. Think of it like asking a librarian for help. You state what you want. The librarian does not hand you the whole library. The librarian hands you a short pile with the best book on top. That librarian picture stays useful for the whole lecture: short pile, best book on top, query decides the order.

Picture the score as a row of books on a table sorted left to right. The horizontal axis is rank position, from 1 on the left to 20 on the right. The vertical axis is the relevance score, from 0 at the bottom to 1 at the top. Bars fall from left to right, tall on the left and short on the right. The landmark is the sharp drop after the first three or four bars. The takeaway is that only the head of the list matters, so scoring must push the best items to the left.

Scope: This definition fits ranked search over a large store. Assumption: the store is large, items vary in fit, and the query is shorter than the need behind it. When the store is tiny and every item fits the same way, ranking adds little. When the ask is a full formal spec with exact field names, exact data lookup fits better than ranked retrieval.

The same definition covers web pages, papers, mails, files, images, and video. The four jobs behind it are representation of items, storage of items, organization of items, and access to items. Representation turns raw items into a form the system can match. Storage keeps them. Organization builds structures for fast lookup. Access runs the query and returns the ordered list. You will meet each job again in the pipeline section.

1.1.2 How Learners First Describe It

The first exchange in the room asked everyone to define the idea in their own words. Several answers came back. One answer said it is finding relevant data from a huge heap of unmanaged data. Another said it is finding learned information from a large collection. Another said it is extracting useful information from raw data. Another said it is finding the right information. Another said it is retrieving relevant information when we truly care about the data.

Each answer points at the same goal. There is a lot of stored material. Stored material alone has little value. Value comes when we pull out the part that fits the ask. The formal version keeps that meaning. It is finding relevant information from a large collection for a given query. Relevance is always judged with respect to the query.

Q: What is information retrieval in your own words? How would you define information retrieval words from scratch?

A: It is finding relevant data from a huge heap of unmanaged data, because stored information has little value unless we pull out the part that fits our need. The agreed formal version is finding relevant information from a large formal collection for a given query, where query relevance is judged with respect to that query. Each learner phrase — huge heap, large collection, raw data, right information — points at the same move from stored material to fitted answer.

Why did so many phrasings agree? Because each one names the same three parts: a large collection, a query that names the need, and a relevance test that picks the fitting slice. The phrase "huge heap of unmanaged data" stresses the starting mess. The phrase "large collection" stresses the searchable store. The phrase "raw data" stresses unprocessed input. The formal definition joins them: start messy and large, filter by query, order by relevance.

1.1.3 Everyday Examples That Count

A web search (typing words into a search box and getting back ordered pages, for example "photosynthesis steps") is the best known example. You type words into a search box. You expect pages about those words, not random pages. Retail search works the same way. You search on a shopping site and expect products that fit the words. Video search works the same way. You search on a video platform and expect videos on that topic.

Real-world: web search on Google, product search on Amazon, video search on YouTube, paper search on PubMed, and file search on a personal computer all follow the query to ranked documents pattern.

File search on a computer is a small but telling case. You type a file name or a keyword such as "visa 2024". You expect only relevant files and folders, for example two PDFs and one mail thread, ranked with the best match first. You do not expect the system to list every file on the disk. That filtering plus ordering is what makes it retrieval rather than plain browsing. If a system only filters without ranking, that is closer to browsing. Ranking is part of the job. The same pattern holds at web scale: query in, relevant files and folders out, ranking plus browsing controls decide what you see first.

Mail search shows the same shape. Type "offer letter". The system scans the body text, the subject lines, and the file names. It returns the fitting mails first. Paper search on PubMed shows it too. Type "lung cancer biomarkers". The system scans titles, abstracts, and index terms. It returns the most fitting papers first. In each case the input is a short query and the output is relevant documents in rank order.

A common trap is to call any search box retrieval. A box that only lists files by name prefix with no fit score is closer to lookup. A box that filters by exact date with no ordering is closer to data lookup. The retrieval label fits when many stored items could help and the system must order them by fit to the query.

1.1.4 Two Classical Models Ahead

The course opens with two classical approaches. The first is the Boolean model (an exact-match model with only present or absent per term, for example "lung AND cancer NOT kidney"). A Boolean model keeps only two states for each term, present or absent. It uses logical operators to combine terms. Queries use AND, OR, and NOT. The model is simple and yet efficient for exact matching.

The second is the vector space model (a ranked model that turns text into number lists, for example a query vector matched against document vectors). A vector space model turns each document and each query into a vector. A vector here is a list of numbers, one number per term or feature. Similarity between the query vector and each document vector decides the order. In words: similarity between query vector $q$ and document vector $d$ . As LaTeX: $s im (q, d)$ , where $q \in R^{n}$ is the query vector with $n$ term weights and $d \in R^{n}$ is a document vector with the same $n$ weights and the output is a number that grows with closeness.

A query vector (the number list for the ask, written $q \in R^{n}$ , for example $q = [1, 0, 2]$ ) and a document vector (the number list for one stored item, written $d \in R^{n}$ , for example $d = [1, 1, 0]$ ) live in the same space with the same $n$ term weights. Here $n$ is the count of index terms. Here $R^{n}$ means a list of $n$ real numbers. The function $s im (q, d)$ returns one number. A larger value means closer meaning and a higher rank.

Both models start from the query and end with relevant items. The Boolean path stresses exact logic. The vector path stresses measured closeness and ranking. Later sessions build each path step by step, from Boolean queries with AND, OR, and NOT to vector documents where similarity decides the rank order.

A side-by-side view helps. Boolean matching asks "does this document have the required terms, yes or no" and returns a set with no order. Vector matching asks "how close is this document to the query, on a scale" and returns an ordered list. Pick Boolean logic when the ask is a hard rule with must-have and must-not-have terms. Pick vector similarity when the ask is soft and many items help to different degrees.

Q: What will the course build first? Will it start with Boolean or vector models?

A: The course builds a strong base in retrieval first, starting with the Boolean model with AND, OR, and NOT Boolean queries, then the vector space model where vector documents and queries become vectors and similarity decides the rank order. Boolean logic comes first because it is simple, then vectors add ranking.

1.1.5 Core Reference Book

A reference book (a trusted text that holds the core methods for a course, for example the Manning book) guides the early path. The main text here is Introduction to Information Retrieval by Manning, Raghavan, and Schutze. Early sessions follow that book closely for core concepts. Later sessions on cross-lingual search, multimedia search, recommenders, and neural methods draw on several books and papers. Those later sources get cited where they are used.

Real-world: students who like paper copies can buy the Manning book, while others can use the soft copy for the early core sessions. Use the book for Boolean retrieval, term weights, the vector space model, and scoring. Use later readings for cross-lingual links, multimedia indexing, recommender methods, and neural rankers.

Exam note: State the definition as query to ranked documents, write $scor e (d, q)$ and $r e l (d, q)$ with each symbol named, and contrast Boolean queries against vector documents with similarity rank order. Name the Manning text as the core reference for the early models.

A one-line recap ties this concept together. Retrieval takes a query and returns relevant documents in rank order, scored by fit. That scored ordering leads straight to where retrieval shows up, from web search to recommenders, which is the next concept.

Ranked documents by query relevance define the field: $scor e (d, q)$ grows with fit, Boolean logic filters sets, vector similarity orders lists, and the best item lands on top like the librarian short pile with the best book on top.

1.2 Where Retrieval Shows Up

1.2.1 Text Collections and Web Search

Where do you meet ranked search in daily life? Almost every box you type into runs it.

A text collection (a large store of written items, for example millions of web pages) plus a query plus a relevance order is the same pattern everywhere. Text mining (pulling useful patterns and facts from very large text stores, for example finding drug names across papers) is the batch cousin of live search. Web search is the live version at web scale.

Text mining means pulling useful patterns and facts from very large text stores. Web search means doing the same over web pages at web scale. Both need the same core. Take a query. Compare it against many stored items. Return the relevant ones first.

Real-world: text mining over large text databases, daily web search, and site search on shopping and video platforms all use retrieval at their core. Type a product name on a retail site and the fitting products rise first. Type a topic on a video site and the fitting videos rise first. The store changes, the ranked-documents deal stays fixed.

Take three quick traces. First, web search: query "passport renewal fee", store of web pages, top hits are fee tables and forms, not random pages. Second, retail search: query "running shoes size 9", store of product pages, top hits match both the item and the size. Third, site video search: query "photosynthesis steps", store of videos, top hits teach the steps. Each trace starts with a short query, compares against many stored items, and returns the relevant ones first.

1.2.2 Search Across Languages

What if the right answer sits in a language you did not type in? A letter matcher would miss it. A cross-lingual system still finds it.

Cross-lingual retrieval (finding documents in more than one language from a query in one language, for example English query returning French pages) fixes that gap. The example used a sweet food item. Suppose you type the English word for small French sandwich cookies. You still want the right documents about those cookies. Suppose you type the French word instead. You still want both English and French documents about the same item.

That behavior needs more than letter matching. The system must know the two words point at the same thing. This is why language links matter. Without them, a French query would miss English pages and an English query would miss French pages.

Work the cookie case end to end. Query in English: the words for small French sandwich cookies. The store holds English recipe pages and French recipe pages about the same item. A plain matcher returns only the English documents because the French documents use the French spelling. A cross-lingual index links both spellings to one shared idea, so both English and French documents about those cookies rise to the top. The sense-check is simple: the top list mixes English and French documents about one item, not pages about unrelated sandwiches.

Picture two word clouds, one English and one French, with arrows from both spellings to one shared node in the middle. The horizontal axis is language, English on the left and French on the right. The landmark is the shared middle node with links fanning out to documents in both languages. The takeaway is that meaning links, not shared letters, join the two sides.

Scope: Cross-lingual links help when the same idea has different spellings across languages. Assumption: the system holds a map from words to shared ideas, built from parallel text or linked pages. When no map exists for a rare term, the system falls back to plain matching and misses the other-language pages.

1.2.3 Images Audio and Video

Can a mail search find words inside a photo? Modern retrieval can, because it looks inside file types and not just body text.

Multimedia retrieval (returning more than plain text, for example images, audio, and video for one query) widens the store. The email example makes this concrete. Take the query words digital watermarking. Try those words in your own mail search after the session. You get mails where those words sit in the message body. You also get mails where those words sit inside an attached document such as a PDF or Word file. You can even get mails where those words appear inside an image in the mail.

So retrieval now looks inside file types. It does not stop at the body text. It reads attachments. It reads text baked into images. That multimodal reach is now normal in mail and photo apps.

Trace the digital watermarking mail demo. Query: digital watermarking. Mail one holds those words in the message body, so it matches on body text. Mail two holds those words inside an attached PDF paper, so it matches after the PDF text is pulled out. Mail three holds those words inside an attached Word file, so it matches after the Word text is pulled out. Mail four holds those words inside an image, so it matches after image text is read. All four mails return for one query because email search reads body, attachments, and images together. That body plus attachments plus images coverage is the multimodal point.

Real-world: mail search that finds digital watermarking in message text, inside PDF and Word attachments, and inside images shows multimedia retrieval in daily use. Photo apps add place and time signals on top, so a place name can pull trip photos even when the file names say nothing.

1.2.4 Recommenders Neural Methods and Assistants

What if you never type a query at all and the system still brings you fitting items? That is the recommender twist on the same core.

Recommender systems (suggesting items a user may like, for example "viewers who watched this also liked") track what you viewed and bought. Shopping and streaming sites use them. They track what you viewed and bought. They retrieve items with similar traits or liked by similar users.

Neural retrieval (using deep learning to improve matching and ranking, for example dense vectors that catch synonyms) learns dense vectors for queries and documents. It learns dense vectors for queries and documents. It can catch meaning even when words differ.

Newer assistants build on the same base. A RAG system (a setup that joins retrieval with generation, for example retrieve passages then draft the answer) joins retrieval with generation. It first retrieves relevant passages from a store. Then it drafts an answer grounded in those passages. An agentic assistant (a helper that plans and calls tools around retrieval, for example search then compare then draft) adds planning and tool calls around that loop. One learner described a career guide built with a vector store. It kept roadmaps for roles in a national job market. It retrieved the right roadmap chunks for each user question. Then it drafted guidance from those chunks.

Real-world: product suggestions on Amazon, show suggestions on Netflix, RAG assistants that ground answers in retrieved passages, and agents that plan with retrieved facts all rest on retrieval. Language models, RAG systems, and agents all sit on top of this base: the retriever finds the passages, the generator shapes the answer.

A common trap is to treat recommenders as a different planet. The input differs — past views and buys instead of typed words — but the output is still a ranked list of fitting items. Another trap is to think neural methods replace the base. They improve the match score, often with dense vectors, but the query-to-ranked-list deal stays the same.

1.2.5 Library Media and Filter Funnel Focus

How do you shrink 1000 records to the few that matter? Filter, funnel, focus — each step narrows the set.

A digital library (a searchable book store, for example a university e-library) must return only relevant books. Ask for World War II. A book about Hitler fits better than a book about Gandhi. That choice is relevance in action.

Media search (search over photo, music, and video stores, for example "beach trip 2024") covers photo, music, and video stores. One learner wanted to type a beach trip or a place name and see all photos from that spot to show a friend. That wish is retrieval over images with place and time signals.

Information filtering (narrowing a stream to what matters, for example keeping only stage-four lung cancer records) narrows a stream to what matters. Think filter, funnel, focus as three steps. Suppose you hold 1000 insurance claim records. First filter for lung cancer. Then funnel to records with stage information. Then focus on stage four lung cancer patients. A good retrieval stage makes the first filter easy. One learner named Power BI as a tool that pulls facts from messy stores into structured views, which fits the same filter-then-shape pattern.

Run the numbers on the filter funnel focus claim. Start with 1000 claim records. Step one, filter for lung cancer: about 180 records stay. Step two, funnel to records with stage information: about 90 records stay. Step three, focus on stage four lung cancer patients: about 25 records stay. Each step keeps only the fitting slice. The sense-check is that the final pile is small, on-topic, and ready for review. That 1000 to 180 to 90 to 25 shrink is why the filter funnel focus labels stick.

Real-world: library search that prefers on-topic books, media search over photos and video, and filter funnel focus over 1000 claim records down to fourth stage lung cancer cases. Library search picks the Hitler book over the Gandhi book for a World War II ask. Media search uses beach names plus time stamps to pull trip photos. Claim filtering uses disease plus stage to reach the fourth-stage lung cancer slice.

1.2.6 A Tree Example and Its Debate

Does walking a company tree count as retrieval? It depends on whether ranking happens.

A corporate linkage tree (a parent company linked to headquarters and branches, for example parent at top with branches as leaves) links a parent company to headquarters and branches. Branches sit as leaves. Headquarters sit above them. One learner asked whether walking that tree to pull branch facts for a parent counts as retrieval.

The reply drew a line. A tree store alone is not enough. A query alone is not enough. The output must hold several candidate nodes in rank order for the label to fit well. If the walk returns one fixed node with no ranking, it is closer to lookup. If one parent has many branches and the system ranks matching nodes, then the retrieval label fits. Real-time branch lists can create that multi-node ranked case.

Q: Does walking a corporate linkage tree count as retrieval? Is tree walking retrieval?

A: It counts when a query faces many candidate nodes and the system returns ranked relevant nodes, as with many branches under one parent in real-time use with parent and branch nodes ranked. It counts less when the walk returns one fixed node with no ranking, which is closer to lookup with a fixed node and no rank order.

Q: Is phone photo search by place a good use case? What about a phone photo search place case?

A: Yes, typing a beach name or hotel area to pull all photos from that location for showing to family is a strong image retrieval case, using beach and hotel photos plus place and time signals over the photo store.

Q: Will this course build full language models end to end? Do we build full language models here?

A: No, end-to-end language model building is not part of this course, but language models, RAG systems, and agents all rest on retrieval as their base, so a firm retrieval base helps you grasp what happens inside those larger systems.

Cross-lingual cookies, multimedia watermarking mails, Amazon and Netflix recommenders, and the ranked tree debate all share one deal: a query faces many candidates and a ranked relevant list comes back.

1.3 How a Retrieval System Runs

1.3.1 From Need to Query

Every search starts before anything is typed. A person feels a gap. That gap must shrink into a few words a machine can match.

An information need (the real gap, for example "I must grasp genomics linked to lung cancer") is broad and personal. A query (the short handle, for example "lung cancer biomarkers") is narrow and shared with the system. Good handles retrieve better lists because they keep the disease plus the angle and drop the rest.

Every run starts with a user who has a gap. That gap is the information need. The need then turns into a query. A query is often keywords or short natural phrases.

Query style has shifted. Before wide chatbot use around 2022, short precise queries mattered most on web search. After that shift, many users type long sentences. Both styles still map to the same step. Turn the need into a compact ask the system can match.

A query is not the need itself. The need is broad. The query is the short handle for it. Good handles retrieve better lists. Short precise queries keep the core nouns and drop filler. Long sentences still work when the system can pull out the core nouns, but extra off-topic words pull in weak matches.

Scope: The need-to-query step fits any ranked search. Assumption: the user can name the topic in a few words. When the need is vague ("find something interesting"), no short handle fits well, and browsing or a recommender fits better than a typed query.

1.3.2 What the System Compares

What sits on each side of the match? A short ask on one side, a huge store on the other, and a scoring step in the middle.

A retrieval system takes two inputs. One is the query. One is the stored collection. Stored items can be web pages, images, metadata, papers, mails, or records. The system compares the query against those items. Matching plus ranking methods decide the output. Those methods form the core of the system.

Think of the flow in plain steps. User need becomes query. Query plus corpus enter the system. Matching finds candidates. Ranking orders them. Ordered relevant items return to the user.

The run has five stages: need turns into query, query plus corpus enter matching, matching keeps candidates, ranking orders candidates, ordered relevant items return. Matching answers "which items touch the query at all". Ranking answers "which of those help most". Both use term weights early on and embeddings plus neural scores later, but the five-stage shape stays fixed.

Picture a funnel lying sideways. The wide mouth on the left is the corpus with millions of items. The narrow neck in the middle is the candidate set of a few hundred. The thin pipe on the right is the top ten shown to the user. The landmark is the sharp shrink at each stage. The takeaway is that matching cuts the store fast, and ranking spends the costly math only on the survivors.

1.3.3 A Cancer Genomics Walkthrough

Suppose you work on cancer genomics. Your need is to grasp genomics linked to lung cancer. If genes sounds more familiar than genomics, read it as genes for now. What query would you type into a scholar search or a paper index?

Learners offered several strong options. Genes of lung cancer. Gene sequence for lung cancer. Genomics for lung cancer. Lung cancer biomarkers. Each option is short and on topic. Any of them could serve as the query. Less off-topic text helps. The shared trick is disease plus gene angle: keep "lung cancer" and one gene word, drop the rest.

Now what is the document store? Past research papers on lung cancer biomarkers. Clinical studies on lung cancer. Radiology and radiogenomics files. Review articles and preprints in archives. Treatment and trial documents. Gene expression repositories and data sets. The store is huge. It keeps growing while you think.

Run the lung cancer biomarkers case end to end. Need: grasp genomics linked to lung cancer. Query: lung cancer biomarkers. Corpus: past research papers on lung cancer biomarkers, clinical studies, radiology and radiogenomics files, review articles and preprints, treatment and trial documents, plus gene expression repositories and data sets. Matching pulls papers that hold those three words and their close variants. Ranking pushes the most fitting papers first: highly cited papers rise, wet-lab proven studies with gene expression proof rise, recent reviews rise. The sense-check is that the top five all join lung cancer with gene proof, not generic cancer pages.

The system must match a short query such as lung cancer biomarkers against all of that. It uses ranking steps. Early methods use term weights. Later methods use embeddings and neural scores. You will study each family in turn. The top output must be the most fitting papers. Highly cited papers help. Wet-lab proven studies with gene expression proof help. You do not want to re-query on top of weak output. You want strong items first.

Exam note: Expect to explain the full chain from need to query to corpus to ranked output, using a concrete case like lung cancer biomarkers. Name the query, name the corpus parts, name the ranking push, and close with why the top items fit best.

1.3.4 What Counts as Information

What can the store hold? Far more than plain pages.

Information can take many forms. Text. Numbers and stats. Summaries. Audio. Video. Images. Source code as on code hubs such as GitHub. Structured text such as XML, JSON, and HTML.

Unstructured text (material with no neat rows and columns, for example a research article with title, abstract, sections, and author affiliations) is the normal case. A research article shows why. It holds title, abstract, sections, author affiliations, and more. That mix does not fit one tidy sheet. Retrieval must still represent it, store it, organize it, and give access to it.

Retrieval covers four jobs here. Representation of items. Storage of items. Organization of items. Access to items. Most corpora are unstructured, which means they do not sit in neat rows and columns. A research article shows why. It holds title, abstract, sections, author affiliations, and more. That mix does not fit one tidy sheet. Representation turns that mix into matchable views. Storage keeps the views. Organization builds fast indexes. Access runs each new query.

A common trap is to think numbers, code, and XML fall outside retrieval. They count as information too. Stats tables, GitHub source code, and XML, JSON, and HTML pages all sit in real corpora. The same represent, store, organize, and access jobs cover them.

Q: What query fits a lung cancer genomics need? Which query fits lung cancer genomics?

A: Short on-topic options such as genes of lung cancer, gene sequence for lung cancer, genomics for lung cancer, or lung cancer biomarkers all fit, because each names the disease plus the gene angle with genes and genomics terms and without extra noise.

Q: What sits in the document store for that need? What corpus sits behind the need?

A: Past research papers, clinical studies, radiology and radiogenomics files, review articles and preprints, treatment and trial documents, plus gene expression repositories and data sets, all searched together as one large growing corpus and document store.

Need turns into query, query plus corpus feed matching and ranking, and a genomics walkthrough shows the chain: lung cancer biomarkers against papers, trials, and gene expression stores with the strongest proof on top.

1.4 Retrieval and Language Understanding

1.4.1 Two Jobs Side by Side

Finding papers is one job. Reading inside them is another. This section keeps the two apart.

Information retrieval (finding relevant items from a large collection, for example top papers for "lung cancer treatment") finds. Natural language processing (grasping and shaping human language, for example drafting a summary) reads and writes. Retrieval outputs a ranked list of documents. Language processing outputs labels, tags, summaries, translations, or drafted text.

Information retrieval finds relevant items from a large collection. Natural language processing grasps and shapes human language. Retrieval outputs a ranked list of documents. Language processing outputs labels, tags, summaries, translations, or drafted text.

Take the query biomarkers of lung cancer typed into web search. Retrieval returns relevant documents about those biomarkers. Language processing can go further. It can read gene expression notes from DNA and RNA sequencing. It can link image-based radiogenomics notes. It can join those threads into a tighter answer.

Take the query lung cancer treatment. Retrieval returns documents about lung cancer treatments. That word relevant will repeat often because fit to the query is the key test. Language processing can then digest the top results. It can list treatment options. It can compare them. It can draft a short summary.

So retrieval finds the right papers. Language processing reads inside them. One finds. The other reads and writes. The find step must come first: no reader can draft a sound summary from weak or off-topic papers.

Trace the treatment case both ways. Query: lung cancer treatment. Retrieval step: scan the corpus, keep papers about lung cancer treatments, rank the most fitting first. Language step: open the top five, pull treatment names from text plus gene expression notes from DNA and RNA sequencing plus image notes from radiogenomics files, then draft a short compare-and-contrast summary. The sense-check is that the summary cites the retrieved papers. Without the find step, the summary floats with no source. Without the read step, the user gets links but no synthesis.

1.4.2 Why Meaning Beats Bare Matching

Same words, flipped order, flipped meaning. A letter counter misses the flip. A meaning reader catches it.

Bare word overlap can mislead. Take dog bites man versus man bites dog. The two strings share the same content words. A plain overlap score would call them the same. In words: cosine similarity between the two phrase vectors equals one. As LaTeX: $s im (A, B) = 1$ , where $A$ is the vector for dog bites man and $B$ is the vector for man bites dog and $s im$ is cosine similarity with range minus one to one. Yet the meanings differ sharply. The two queries must not return the same list.

A phrase vector (a number list for a whole phrase, written $A$ , for example $A = [1, 1, 1]$ for three content words) feeds the score $s im (A, B)$ , where $s im$ is cosine similarity with range minus one to one. Here $A$ is the vector for dog bites man. Here $B$ is the vector for man bites dog. Here $s im (A, B) = 1$ means the plain count vectors match fully. The catch is that count vectors drop word order, so $s im (A, B) = 1$ can hold while the phrase meanings differ sharply. Meaning-aware ranking must split the lists even when the count score ties.

Take doctor for heart problem. A letter-match search would chase those exact words. A smart system returns cardiologist details instead. A cardiologist (a heart doctor, for example a specialist who treats rhythm faults) is a heart doctor. The system must bridge the everyday phrase to the expert term.

Direct and indirect feedback helps. Thumbs up and thumbs down on search and chat tools tune later lists. Trusted sources get more weight. Research articles and health guidelines outrank random blogs for medical queries. That source weight is part of smart retrieval.

Picture two arrows on a line from minus one to one. Mark the plain count score at one, hard right. Mark the true meaning score near zero or below, middle to left. The landmark is the wide gap between the two marks for the same word pair. The takeaway is that overlap alone sits at the wrong end of the scale when order flips meaning.

Scope: Order-free overlap works for bag-of-words asks such as "lung cancer biomarkers", where order changes little. Assumption: word roles do not flip the need. When roles carry the meaning, as in dog bites man against man bites dog, the system must use order, roles, or phrase links, or it returns the wrong list.

Q: Is a summarize query only retrieval or a mix? Is summarize query retrieval plus language mix?

A: It is a mix of both retrieval and language processing, because the system must retrieve documents about lung cancer biomarkers and then draft a summary from them. If it returned non-summary pages when summaries were asked for, that output would miss the need, so query grasp plus post-retrieval summarization both matter.

Q: Does a decoder-only generator skip retrieval when it summarizes? Can generation skip retrieval?

A: Generation alone does not replace retrieval, because retrieval finds the right documents first and language processing then summarizes inside those documents. Token steps help the generator, but the find-then-read split still holds with retrieval plus language roles kept apart.

Q: Is retrieval only about matching methods? Is matching alone the whole job?

A: No, matching alone is too narrow, because the job is to find the right documents from the corpus and rank them well. Hold that question for four or five sessions and the full course path from weights to embeddings to neural rankers will make the scope plain.

1.4.3 Private and Locked Text

Can the system unlock private files to answer better? No — locks stay locked unless rights and reasons allow it.

Encrypted and masked text raised a long thread. Learners asked about consumer stores with masking, health records under privacy rules, hash methods such as SHA-256, and RAG stores with locked files and password-gated NDAs. They asked whether retrieval must decrypt or unmask to answer.

The reply set a boundary. Decryption for its own sake is not the job here. Masking for privacy must stay masked for users without rights. If decryption happens, it needs a valid and ethical reason plus the right keys and security steps. Much of that work belongs to security and access control, not to core ranking.

Role-based masking (hiding fields per role, for example a receptionist view hides age and gender while a treating doctor view shows them) shows why context matters. A receptionist view of health records must hide age, gender, and other direct identifiers. A treating doctor view needs those same fields to give care. Claims and trial stores often keep names mapped to IDs behind several locked layers. Hospital search for all lung cancer patients still needs correct patient links for staff with rights. It must not leak those links to staff without rights.

Compare the two views on one ask. Ask: all lung cancer patients seen this month. Receptionist view: the system searches the same index but the display hides age, gender, and direct identifiers, showing only slots and counts. Treating doctor view: the same search shows age, gender, and linked history needed for care, because that role holds rights. Locked NDA stores in enterprise RAG setups behave the same way: password-gated files never enter the answer for users without access, even when their text matches well. The sense-check is that the rank list respects the lock before it respects the score.

Real-world: role-based views in health record search, locked NDA stores in enterprise RAG setups, and masked claim and trial data all need access control before retrieval runs. SHA-256 hash steps and security checks sit in that access layer, not in the ranking math.

Q: How must retrieval treat encrypted or masked personal data? Must it decrypt masked health data?

A: It must not strip privacy masking for users without rights, because the lock exists for a reason with encrypted and masked personal health data. Role-based views decide what each role can see, as with a receptionist view that hides identifiers versus a doctor view that needs them with privacy and masking rules kept, and any valid unlock belongs to security steps with proper rights.

Q: Can search engine optimization count as retrieval work? Is tuning for better ranking retrieval?

A: Yes, tuning storage, representation, relevance, and ranking to return better ordered lists is retrieval work, since the goal is more fitting results for the query with search engine optimization as rank tuning.

Retrieval finds ranked papers while language reads and writes summaries; dog bites man against man bites dog proves overlap with similarity one still needs meaning-aware split lists, and private text stays locked per role.

1.5 Two Kinds of Information Need

1.5.1 Looking Back at a Fixed Store

Some asks look back. The store sits still. Only the questions change.

A retrospective need (searching the past with changing queries over a fixed store, for example "Newton laws") asks new questions of old material. Queries change. The collection stays fixed. Results do not shift much with time.

A retrospective need searches the past. Queries change. The collection stays fixed. Results do not shift much with time. In words: query varies while collection $C$ stays fixed. As LaTeX: $C_{t + 1} = C_{t}$ , where $C_{t}$ is the stored collection at time $t$ and the equality states the store is static across queries.

A static collection (a store frozen across queries, written $C_{t}$ , for example the same paper index today and next week) obeys $C_{t + 1} = C_{t}$ . Here $C_{t}$ is the stored collection at time $t$ . Here $t + 1$ is the next time step. The equality says no new items entered between the two asks, so the same query returns the same list and only a new query changes the output.

Searching for Newton laws in papers and books is a classic case. Searching for an ice cream recipe is another. Search now or in twenty years. The core facts stay stable. Google-style search over a fixed index with changing one-time queries fits here.

Trace two retrospective asks. Ask one: Newton laws. Store: fixed paper and book index. Run today and next month: the same core laws return on top because $C_{t + 1} = C_{t}$ . Ask two: ice cream recipe. Store: fixed recipe index. Vary the query from "vanilla" to "mango": the list changes because the query changed, not because the store moved. The sense-check is that time alone never reshuffles the list.

1.5.2 Watching a Stream With a Fixed Ask

Other asks stand watch. The question sits still. The stream moves.

A prospective need (watching the future with a fixed ask over a growing stream, for example "ping me when the price tops 200") keeps the query fixed while the collection grows. Results depend on time because new items keep arriving.

A prospective need watches the future. The query stays fixed. The collection keeps growing. Results depend on time. In words: collection grows while query $q$ stays fixed, and a stock alert fires when price $p$ tops 200. As LaTeX: $p > 200$ , where $p$ is the current stock price in rupees and $200$ is the alert level and the system returns the item once the inequality holds.

A standing ask (a fixed query watched over time, written $q$ , for example "price tops 200") pairs with a growing store. Here $p$ is the current stock price in rupees. Here $200$ is the alert level. The rule $p > 200$ fires once per crossing tick: at $p = 198$ nothing returns, at $p = 205$ the item returns. Time drives the output, not a new query.

Waiting for new papers on a topic fits here. Watching cricket news fits here. Tracking prices fits here. You set a standing ask once. The system watches the stream and pings you when fitting items arrive.

The stock case can confuse. It looks like forecasting. It is not forecasting here. No model predicts next week prices. The system only waits for the stored price to cross the set line and then returns that event. Collection change drives the output. No forward guess is made. Stock forecasting with a predictive model would guess the next price; a stock watch only waits for the stored price to cross 200.

Picture a flat line at 200 across a time chart, with a wavy price curve below it. The horizontal axis is time in ticks. The vertical axis is price $p$ in rupees. The landmark is the first tick where the curve pokes above 200. The takeaway is that the alert fires at the crossing, with no forecast of later ticks.

Scope: Prospective search fits streams such as news posts, mails, fresh papers, and price ticks. Assumption: the ask stays fixed while items flow in. When queries also change on every run, the task slides back toward repeated retrospective search instead of a standing watch.

1.5.3 A Small Quiz With Answers

Five quick cases test the split.

Watching real-time election trends on a social feed is prospective. The ask stays set. The feed keeps moving.

Searching an encyclopedia for photosynthesis is retrospective. The ask varies. The store is fixed.

Searching a gene index for known sequences is retrospective. The word known matters. It points at stored facts. Asking for new or novel sequences would flip the same store toward prospective.

Monitoring new drug papers for a rare disease is prospective. The ask stays set. New papers keep arriving.

Searching best restaurants near me is debatable. Both labels can hold. Judging by stored reviews is retrospective. Watching for fresh reviews is prospective. The time frame decides.

Score the quiz as a table. Election trends on a social feed: fixed ask plus moving feed gives prospective, watch for a stream with a standing ask and fresh ticks. Encyclopedia photosynthesis: varied ask plus fixed store gives retrospective, look back at stored facts. Gene index for known sequences: the word known points back, so retrospective; swap in new or novel and the same index turns prospective. New drug papers for a rare disease: fixed ask plus arriving papers gives prospective. Best restaurants near me: stored reviews give retrospective, fresh reviews give prospective, so the time frame decides. That known against novel contrast is the sharpest test signal.

Q: Is ice cream making time bound or stable? Is the ice cream case stable?

A: It is stable for this split, because the core recipe facts in the fixed store change little over time with ice cream making as stored recipe facts, so changing queries against that fixed store stay retrospective.

Q: Does prospective mean predictive? Is prospective the same as predictive forecasting?

A: No, prospective means waiting for new items with a fixed ask, not guessing future values with predictive forecasting. The stock alert with stored price only fires when the stored price crosses 200, with no forecast model in the loop.

Q: Is laptop file search ad hoc retrieval? Is file search ad hoc?

A: Yes in the normal case of ad hoc retrieval, because the laptop file store is fixed and each query stands alone for a one-time need. It turns debatable only if files stream in during search and the same fixed ask watches that stream.

1.5.4 Ad Hoc Search and Filtering Streams

The two needs each have a workhorse setup. One serves one-time asks. The other routes a live stream.

Ad hoc retrieval (many one-time queries against one static store, for example "board exam results 2024") is the retrospective workhorse. Many queries face one static store. Each query is a one-time need. Each runs on its own. Example asks: study material for retrieval, drug discovery notes, prize lists, board exam results. Each ask runs apart from the others.

Filtering retrieval (a standing profile sorting each new arrival, for example "keep only retrieval papers") is the prospective workhorse. A stream of new items flows in. News posts, mails, messages, fresh papers. Each user keeps a standing profile. The system makes a yes or no choice on each new item for each user. One user may want only retrieval-related papers from the stream. Another may want only sports headlines. The system sorts incoming items into bins and routes the fitting ones onward. A stock watch that keeps checking each tick for the 200 line and forwards the hit is the same pattern. A stock price alert stream with a standing ask and ticks shows the same yes-or-no per-arrival choice.

A common trap is to call filtering a forecast. Filtering never guesses the next item. It only tests each arrival: does this new paper match the standing retrieval profile, yes or no; does this tick cross 200, yes or no. Prediction would model the future. Filtering waits for it.

Exam note: Be ready to label new cases as retrospective with changing queries over a fixed store versus prospective with a fixed query over a growing stream, and to justify the label from time behavior. Use the stock price alert stream with a standing ask and ticks for the prospective side and the fixed store with $C_{t + 1} = C_{t}$ for the retrospective side.

A one-line bridge closes the split. Fixed store with changing queries looks back; fixed ask with a growing stream stands watch — and both feed the relevance checks that decide what counts as a good answer next.

Retrospective search varies the query over a static collection with $C_{t + 1} = C_{t}$ ; prospective search holds the query fixed over a stream and fires on arrival with $p > 200$ .

1.6 What Counts as Relevant and Smart

1.6.1 Subject Time Authority and Goal

Why can two fitting papers still deserve different ranks? Because relevance runs four checks, not one.

Relevance (fit of an item to the query plus the goal behind it, for example a current Nature paper on the exact treatment) passes four gates: subject fit, time fit, source weight, and user goal fit. Miss any gate and the rank drops.

Relevance has four checks. Subject fit. Time fit. Source weight. User goal fit.

Subject fit is direct. Ask for lung cancer treatment. A kidney cancer paper misses the subject. It is not relevant even if it is about cancer.

Time fit matters. Ask for lung cancer treatment today. A 2025 paper in a top venue likely beats a 2020 paper if both fit the subject. You want current care options.

Source weight matters. Take a 2026 paper in an unknown venue versus a 2025 paper in Nature or a 2020 paper in IEEE. Many readers trust the known venues more despite the date gap. The venue signals care in review.

Goal fit ties them together. The same query from two users can need different slants. One wants trial options. One wants biology background. The list must serve the goal behind the words.

Rank three papers for the ask "lung cancer treatment today". Paper A: 2025 paper in Nature on lung cancer treatment with trial data. Paper B: 2020 paper in IEEE on lung cancer treatment with older protocols. Paper C: 2025 paper in an unknown venue on kidney cancer treatment. Subject check drops C first despite its fresh date. Time plus source checks then order A above B: both fit the subject, but A is current and strongly reviewed. Goal check can still reshuffle A and B: a trial-seeker keeps A first, a history-seeker may lift B. The sense-check is that subject, time, source weight, and goal checks each moved the order for a stated reason.

Picture a four-gate funnel. Items enter at the top. Gate one keeps subject fits. Gate two prefers fresh items. Gate three lifts trusted venues such as Nature and IEEE. Gate four tilts toward the user goal. The landmark is that most items fall at gate one, while gates two to four only order the survivors. The takeaway is that subject fit decides inclusion and the other three decide order.

Scope: The four checks fit sensitive asks such as care options where freshness and trust matter. Assumption: the system knows the date, the venue, and something about the goal. When goal signals are missing, the system must rank by the first three and learn the goal from later clicks.

1.6.2 Beyond Bare Words

Smart retrieval reads meaning, not just letters. The dog bites man case proves it. Same words. Different roles. Different lists needed.

The heart doctor case proves it again. The query says doctor for heart problem. The right answer names cardiologists. The system must map everyday words to expert terms.

Meaning-aware ranking (ordering by roles and sense rather than shared letters, for example splitting dog bites man from man bites dog) beats bare word overlap. Word overlap counts shared tokens. Meaning-aware scoring tracks who did what to whom and maps everyday words such as doctor for heart problem to expert terms such as cardiologist. Overlap with similarity one can still demand different lists when roles flip.

Feedback loops help. Past clicks, likes, and dislikes shape later lists. Personal history shapes results. That is why the same query can show different orders for different users. The system learns patterns of interaction and tunes ranking per profile.

Source trust is the last piece. Research papers and health guidelines carry more weight for medical asks. Random blogs carry less. Smart systems prefer trusted and authoritative items when the need is sensitive.

A common trap is to treat personalization as noise. It is part of smart ranking: past searches, profile signals, and prior feedback tune the order, so two users with the same words see different top items. Another trap is to trust all sources the same. For medical asks, research papers and health guidelines outrank random blogs by design.

Q: Why does the same query show different results for different users? Why do personalization searches differ per user?

A: Personalization shapes the order, because past searches, profile signals, and prior feedback tune the ranking with profile feedback, so two users with the same words can see different top items from smart retrieval and ranking users.

Q: Must dog bites man and man bites dog return the same list? Must reversed meanings share one list?

A: No, they must return different lists, because the word overlap is full with similarity one yet the meanings are reversed with different lists, so meaning-aware ranking must split them.

Exam note: Explain relevance with subject, time, source weight, and goal checks: a 2025 Nature paper can beat a 2020 IEEE paper on freshness plus trust, and the same query still splits per goal and per user history.

A one-line recap holds the concept. Fit to subject opens the door; time, source weight, and goal decide the order — and that ordered deal is what sets ranked retrieval apart from exact lookup next.

Subject fit includes, time fit prefers current care, source weight lifts Nature and IEEE over unknown venues, and goal plus feedback finish the smart order.

1.7 Data Retrieval and Information Retrieval Are Not the Same

1.7.1 Exact Answers From Neat Tables

Some stores answer with a sharp yes or no. Each row either passes every rule or it fails. No near-misses count.

Data retrieval (finding exact answers in structured stores, for example SQL over a student table) demands exact field names and exact term matches. A structured store (neat tables with named fields, for example a student table with CGPA, department, and country columns) makes that test fast. SQL-style queries are the norm.

Data retrieval finds exact answers in structured stores. A structured store keeps neat tables with named fields. Queries must use those exact field names. SQL-style queries are the norm.

Take a student table. Ask for records with CGPA above nine. In words: CGPA value exceeds 9.0. As LaTeX: $C GP A > 9.0$ , where $C GP A$ is the grade point number on a ten scale and $9.0$ is the cutoff. Suppose that returns ten rows. Add department equals CSE. The list shrinks. Add country equals US while the table only holds home-country rows. The list can drop to zero. Zero is a fine answer here.

An exact clause (a hard rule such as $C GP A > 9.0$ , for example department equals CSE) leaves no partial credit. Here $C GP A$ is the grade point number on a ten scale. Here $9.0$ is the cutoff. A row with $C GP A = 9.2$ in CSE passes. A row with $C GP A = 8.9$ fails even when every other field fits. Adding country equals US to a home-country table drops ten rows to fewer rows and then to zero rows, and that zero is valid.

The rule is all or nothing. Each row either meets every clause or it does not. No partial credit. No ranking needed. Field names must match. Terms must match exactly.

Run the student table demo with counts. Start: full student table. Clause one, $C GP A > 9.0$ : ten rows stay. Clause two, department equals CSE: six rows stay. Clause three, country equals US while the table holds only home-country rows: zero rows stay. Each step narrows rows with exact match and no ranking. The sense-check is that zero is a correct answer here, not a failure — the store simply holds no row that meets all three exact clauses.

1.7.2 Relevant Answers From Messy Stores

Other stores answer with degrees of help. Many items help a little. A few help a lot. Order is the answer.

Information retrieval works the opposite way. Stores are messy. Field names are loose or missing. Queries are short and rough. Many answers can be right at once. Ranking decides the order.

There is no single correct row. Many documents can help. Some help more. The system must order them by fit. That need for ordering is why relevance scores and similarity measures sit at the center.

In one line, data retrieval finds exact answers from structured data while information retrieval finds relevant answers from unstructured data. Keep that contrast ready: exact answers from structured tables against relevant answers from messy stores with ranking order.

Real-world: student record lookup with CGPA and department filters shows data retrieval, while paper search for lung cancer biomarkers shows information retrieval. The first narrows rows to an exact set. The second orders papers by fit with partial credit and ranking order.

Picture two outputs side by side. On the left, a table with ten rows shrinking to zero as clauses pile on. On the right, a ranked list of papers that only reorders as the query shifts, rarely hitting bare zero. The landmark is the left side hitting zero while the right side keeps ranked candidates. The takeaway is that exact structured matching allows empty answers while partial ranked matching almost always returns something.

Scope: Exact answers fit structured data with strict fields. Assumption: field names are known and terms match exactly. When fields are loose, names vary, or the ask is rough, exact matching fails safe but empty, and ranked retrieval with partial credit fits better.

1.7.3 Side by Side Contrasts

Data is structured in data retrieval and unstructured in information retrieval. Field naming must be strict in data retrieval and can be loose in information retrieval. Queries must follow exact terms in data retrieval and can be rough in information retrieval. Matching is exact in data retrieval and can be partial with ranking in information retrieval. Empty results are normal in data retrieval and rare as a bare zero in information retrieval, where some ranked candidates tend to return.

Browsing sits apart from both. Browsing moves through lists without a sharp query. Retrieval starts from a query and ranks by fit. Lookup of one fixed node sits near browsing, while ranked search over many candidates sits at the retrieval end.

Q: Can a data query return nothing? Can a data query with country drop to zero?

A: Yes, adding a clause such as country equals US to a home-country student table can drop ten rows to fewer rows and then to zero, and that empty result with zero exact match is valid because matching is exact with no partial credit and no ranking order.

A comparison view locks the split in place.

Side	Data retrieval	Information retrieval
Store	Structured tables	Messy unstructured stores
Fields	Strict named fields	Loose or missing fields
Query	Exact terms	Rough short asks
Match	Exact, no partial credit	Partial with ranking
Empty	Normal, zero is valid	Rare bare zero, ranked candidates return
Output	Exact set	Ranked relevant list

Pick exact lookup when the ask names fields and needs every clause met. Pick ranked retrieval when the ask is rough and many items help to different degrees.

Exact answers from structured tables allow zero; relevant answers from messy stores use partial credit plus ranking order — that one-line contrast plus the table carries the exam mark.

1.8 From Raw Document to Ranked Output

1.8.1 Building a Logical View Step by Step

Raw sentences match slowly and noisily. A cleaned term set matches fast. That cleaned set is the working stand-in for the document.

A logical view (a cleaned term set that stands in for a document during matching, for example "india, success, land, chandrayaan, 3, march, 2023") is what the system matches. Raw text cannot be matched fast. The cleaned view can. The Chandrayaan walkthrough shows the steps from sentence to index terms.

A logical view is a cleaned term set that stands in for a document during matching. Raw text cannot be matched fast. The cleaned view can. The Chandrayaan walkthrough shows the steps.

Step one states the raw document. India successfully landed Chandrayaan-3 on the Moon in March 2023. That single sentence is the whole document for this demo.

Step two brings uniformity. All letters move to lowercase. India becomes india. Moon becomes moon. This step is called normalization for text. It stops the system from treating the same word as two terms due to case.

Step three drops stop words (common words that add little meaning for matching, for example on, the, in). Stop words are common words that add little meaning for matching. Here on, the, and in leave the set. They do not help tell this document apart.

Step four splits the rest into tokens (single matchable units, also called key phrases, keywords, or index words). Tokens are also called key phrases, keywords, or index words. Each unit stands alone. india is one token. successfully is one token. The chain runs through to 2023.

Step five applies stemming (mapping each token to its base form, for example landed to land). Stemming maps each token to its base form. landed maps to land. successfully maps to success. The base forms group variants of one idea.

The final view is a short list. india, success, land, chandrayaan, 3, march, 2023. Those index terms now stand in for the sentence. Build that logical view through lowercase, stop word removal, token split, and stemming to reach the final index terms.

Run the Chandrayaan logical view demo as a chain. Raw: India successfully landed Chandrayaan-3 on the Moon in March 2023. Lowercase: india successfully landed chandrayaan-3 on the moon in march 2023. Stop removal: drop on, the, in, leaving india successfully landed chandrayaan-3 moon march 2023. Token split: india | successfully | landed | chandrayaan | 3 | moon | march | 2023 as separate tokens and keywords and index words. Stemming: landed to land, successfully to success, moon stays moon. Final logical view: india, success, land, chandrayaan, 3, march, 2023. The sense-check is that case noise and filler words are gone while the core names and numbers stay.

Scope: This five-step view fits alphabetic text with clear word breaks. Assumption: lowercase never merges two ideas that case kept apart, and stop removal never drops a word that carries the ask. When case carries meaning or a stop word is the topic, guard those tokens before cleaning.

1.8.2 Stemming Limits and Better Base Forms

Stemming is fast and rough. It groups forms that share letters even when senses differ. Later steps add sense back.

Stemming is rough by design. It can group words that differ in sense. Bank, banking, and banker all stem to bank, yet a river bank and a banker are not the same. Land as soil and landed as touched down share the base land, yet the senses differ. Those gaps get fixed in later steps with more context.

Proper nouns need care. One question asked whether Chandrayaan would stem to a short chant-like root. The answer is no for a careful setup. Proper nouns name one thing. They have no useful base form to collapse. A good pipeline guards names and keeps Chandrayaan intact. Case hints can help, since names often keep capitals, but the guard rule matters more than the case rule.

Lemmatization (mapping variants to a valid dictionary base with the right sense, for example sodium chloride and salt toward one shared form) goes one step further than stemming. It maps variants to a valid dictionary base with the right sense. Sodium chloride and salt map toward one shared form. Parts of speech tags and noun groups help here. They mark whether a word acts as noun, verb, or modifier. They group multi-word names into one unit. Accent and spelling cleanup also sits in this stage.

Keep three levels apart. Stemming strips ends fast with no dictionary check. Lemmatization returns a valid base with sense and parts of speech tags plus noun groups. Proper-noun guarding keeps names such as Chandrayaan intact with no short chant-like root. Rough form grouping runs first for speed; sense-aware bases and intact proper nouns fix the merges that went too far.

Q: Does Chandrayaan stem to a short root like chant? Does Chandrayaan stem to chant?

A: No, proper nouns name one thing and keep their form with proper nouns intact, so the pipeline guards the name and keeps Chandrayaan intact rather than stripping it to a short chant root. Later steps with parts of speech tags, noun groups, and lemmatization plus stemming handle names and sense with more care than rough stemming.

Q: Are bank, banking, and banker the same after stemming? Do bank forms merge?

A: They share the stem bank but differ in sense, as with land as soil versus landed as touched down, so stemming groups forms fast while later sense steps must split meanings that should not merge.

Exam note: Be ready to turn one sentence into its logical view through lowercase, stop word removal, token split, and stemming, and to state the final index terms such as india, success, land, chandrayaan, 3, march, 2023.

1.8.3 Indexing With Term Weights

Once views exist, the system turns words into numbers. Numbers allow fast math at query time.

After views are built, the system indexes them. Indexing (building structures for fast lookup, for example a term-to-document map with weights) builds structures for fast lookup. A term frequency weight counts how often a term occurs in one document. In words: frequency of term $t$ in document $d$ . As LaTeX: $t f (t, d)$ , where $t$ is an index term and $d$ is one document and the value rises with repeats in that document.

An inverse document frequency weight lowers terms that appear in many documents. In words: rarity of term $t$ across the collection. As LaTeX: $i df (t)$ , where $t$ is an index term and the value falls when $t$ spreads across many documents.

A term frequency weight (in-document count, written $t f (t, d)$ , for example $t f (cat, d_{1}) = 2$ ) rises with repeats in one document. An inverse document frequency weight (cross-collection rarity, written $i df (t)$ , for example $i df (the) \approx 0$ ) falls when the term spreads across many documents. The product joins them. In words: term frequency times inverse document frequency for term $t$ in document $d$ . As LaTeX: $t f i df (t, d) = t f (t, d) \times i df (t)$ , where $t f (t, d)$ is the in-document count weight and $i df (t)$ is the cross-collection rarity weight. Frequent here plus rare elsewhere gives a large product; rare here or common elsewhere gives a small one.

A tiny matrix makes this concrete. Take document one, document two, and document three. Take terms cat and dog. Document one holds cat twice. Document three holds neither cat nor dog. Each cell now holds a number, not a word. Numbers allow fast math at query time. That speed gain is why indexing matters.

Build the cat and dog matrix with document term weights and index numbers. Rows are documents one, two, and three. Columns are cat and dog. Document one holds cat twice, so its cat cell is 2. Document three holds neither cat nor dog, so both its cells are 0. Fill the rest from counts, then scale each count by rarity across the collection. The result is a small number table where each cell is a weight, not a word. At query time the system reads one row per document and scores with fast vector math. That table of term weights and index numbers is the index in miniature.

Picture the matrix as a grid. Rows are documents, columns are terms, cells hold weights. The landmark is the all-zero row for the document that holds neither term. The takeaway is that sparse number rows replace raw strings and make scoring a row-wise math pass.

Scope: Count-times-rarity weights fit text where repeats signal focus and rarity signals power to tell items apart. Assumption: terms act as separate units. When phrases or order carry the meaning, single-term weights underfit and phrase or position signals must join the index.

1.8.4 Query Path Similarity and Ranking

The query walks the same cleaning steps, then meets the index. Closeness math decides the order.

The left path handles the query. Suppose the user types moon landing from India. That query goes through the same text steps. It becomes the logical view moon, land, india. Three index terms remain.

Query operations then search the index built on the right path. The system compares the query view against stored views. Cosine similarity (closeness by angle between vectors, for example near one for near-same direction) is one common choice. In words: cosine of the angle between vectors $A$ and $B$ . As LaTeX:

$cos (A, B) = \frac{A \cdot B}{∥ A ∥∥ B ∥}$

where $A \cdot B$ is the dot product sum of paired weights and $∥ A ∥$ is the length of $A$ with Euclidean length and the same for $B$ and the ratio lies in minus one to one with larger for closer items. Pearson correlation and Euclidean distance are other options. In words for distance: straight-line gap between $A$ and $B$ . As LaTeX: $d i s t (A, B) = ∥ A - B ∥_{2}$ , where $A, B \in R^{n}$ are equal-length weight vectors and the smaller gap means closer items.

Two scores cover most needs. Cosine uses angle: the dot product $A \cdot B$ sums paired weight products, each length $∥ A ∥$ and $∥ B ∥$ scales it, and the ratio $cos (A, B)$ lies in minus one to one with larger for closer items. Euclidean uses gap: $d i s t (A, B) = ∥ A - B ∥_{2}$ with $A, B \in R^{n}$ as equal-length weight vectors, and the smaller gap means closer items. Pick cosine when length should not dominate; pick distance when the raw gap is the signal.

Work a tiny cosine trace. Query weights $A = [1, 1, 1]$ for moon, land, india. Document weights $B = [1, 1, 0]$ for a page about moon land without india. Dot product $A \cdot B = 1 + 1 + 0 = 2$ . Lengths $∥ A ∥ = 1.73$ and $∥ B ∥ = 1.41$ . Ratio $2/2.44 = 0.82$ . A second document $C = [1, 0, 0]$ scores lower at $0.58$ . The first document ranks above the second because its angle to the query is smaller.

Higher similarity means higher rank. Top items return to the browser interface first. Feedback links loop back to improve later runs.

The build splits by time cost. The right path is mostly one-time work. Collect with a crawler or data access module. Clean with text steps. Build views and index matrices once. The left path runs per query. Clean the query. Search the index. Score and rank. That split keeps daily search fast.

A library analogy ties it together. The catalog is the index. Your ask is the query. Finding cards is searching. Recommending the best book first is ranking. The library catalog with index cards plus query finding plus ranking the best book first is the same pipeline in wood and paper.

Trace moon landing from India end to end. Query text: moon landing from India. Logical view: moon, land, india as three index terms after the same text steps. Index search: the system pulls stored views that hold those terms with term weights. Scoring: cosine, Pearson, or Euclidean math compares the query view against each stored view. Ranking: the highest closeness returns first to the browser, with feedback links looping back. The sense-check is that pages joining moon plus land plus india outrank pages with only one of the three.

Q: Does this architecture use vectors already? Are vectors used here?

A: Only as a preview here, because full vector models come later. Term weights and the tiny cat and dog matrix with document term weights and index numbers only show how a logical view turns into numbers for fast search, while cosine, Pearson, and Euclidean choices for scoring get full treatment in later sessions.

Q: What turns moon landing from India into matchable form? What turns the moon landing India query into matching terms?

A: The same text steps turn moon landing from India into the logical view moon, land, india, and those three index terms then search the term weight index for scoring and ranking with query path similarity and library catalog ranking.

Exam note: Build the logical view with lowercase plus stop removal plus tokens plus stemming, state final index terms, show term weights feeding cosine and distance scoring, and close with ranking plus the catalog picture.

Exam Guidance Summary

No mark splits or question patterns for the final test were set in this opening material. The emphasis signals below guide study.

Explain relevance as fit to the query with subject, time, source weight, and goal checks. A 2025 paper in Nature can outrank a 2020 paper in IEEE on freshness plus trust, while a kidney cancer paper misses the subject for a lung cancer treatment ask even when both are about cancer.

Trace need to query to corpus to ranked output with a concrete case. Use lung cancer biomarkers: short query, corpus of papers plus clinical studies plus radiology and radiogenomics files plus reviews and preprints plus treatment and trial documents plus gene expression stores, ranked with highly cited and wet-lab proven items first.

Label retrospective versus prospective from time behavior with fresh examples. Retrospective varies queries over a fixed store with $C_{t + 1} = C_{t}$ , as in Newton laws or photosynthesis. Prospective holds a fixed query over a growing stream with a stock price alert stream and standing ask firing at $p > 200$ , as in election trends or new drug papers.

Contrast exact data lookup with ranked retrieval in one line plus the table contrasts. Data retrieval finds exact answers from structured tables with exact match and zero as valid; information retrieval finds relevant answers from messy stores with partial credit and ranking order.

Build a logical view from a sentence through lowercase, stop word removal, tokens, and stemming to final index terms. Show india, success, land, chandrayaan, 3, march, 2023 from the Chandrayaan sentence, guard proper nouns, and note lemmatization with parts of speech tags and noun groups for sense.

State how term weights and similarity scoring feed ranking. Show $t f (t, d)$ with in-document counts, $i df (t)$ with cross-collection rarity, the product $t f i df (t, d) = t f (t, d) \times i df (t)$ , and scoring with cosine angle plus dot product plus lengths against Euclidean gap, closing with ranking and the library catalog picture.

Real-world cases and Q and A exchanges above carry the same weight as definitions. Revisit the cookie cross-lingual case, the digital watermarking mail case, the filter funnel focus 1000 to stage-four case, the tree ranking debate, and the moon landing India query path before the test.

Key Industry Applications

Web, retail, video, paper, and file search cover daily ranked lookup. Web search on Google, product search on Amazon, video search on YouTube, paper search on PubMed, and file search on a personal computer each take a short query and return relevant files and folders plus pages with ranking.

Cross-lingual search joins English and French documents about one item. The French sandwich cookie case shows meaning links beating letter match, so both English and French documents rise for one query.

Mail search reads body text, PDF and Word attachments, and image text such as digital watermarking. That body plus attachments plus images coverage is now normal in enterprise and consumer mail.

Shopping and streaming recommenders plus neural rankers lift match quality. Product suggestions on Amazon and show suggestions on Netflix use past views and buys, while neural methods add dense vectors that catch synonyms.

RAG and agentic assistants retrieve passages with vector stores such as ChromaDB then draft grounded answers, including a career roadmap guide. The retriever finds the roadmap chunks for each question; the generator drafts guidance from those chunks. Language models, RAG systems, and agents all rest on this retrieval base.

Photo search by place and time supports trip sharing. Typing a beach name or hotel area pulls photos from that location with place and time signals for showing to family.

Library search prefers on-topic books. A World War II ask lifts the Hitler book above the Gandhi book on subject fit.

Filter funnel focus moves from 1000 claim records to fourth stage lung cancer cases. Filter for lung cancer, funnel to staged records, focus on stage four, with tools such as Power BI shaping the final view.

Code search over source hubs and structured search over XML, JSON, and HTML widen the corpus. Source code on GitHub plus structured text in XML, JSON, and HTML all sit behind the same represent, store, organize, and access jobs.

Role-gated health record search and enterprise locked-file search put access control first. Receptionist views hide identifiers while doctor views show them; locked NDA stores never leak to users without rights even when text matches well.

IR Lecture 1 notes · Introduction to Information Retrieval

Information Retrieval· undergraduate· 2026-09-13

Sections Breakdown

1What Information Retrieval Is

Defines IR as query to ranked documents with relevance scores, Boolean and vector models, and the Manning reference.

2Where Retrieval Shows Up

Surveys text, cross-lingual cookies, watermarking mails, recommenders, RAG, libraries, media, filtering, and the tree debate.

3How a Retrieval System Runs

Runs need to query to corpus to ranked output via the lung cancer biomarkers walkthrough and four IR jobs.

4Retrieval and Language Understanding

Contrasts retrieval with language processing, proves meaning beats matching, and sets role-based privacy bounds.

5Two Kinds of Information Need

Splits retrospective fixed-store search from prospective stream watch with quiz, ad hoc, and filtering setups.

6What Counts as Relevant and Smart

Defines relevance via subject, time, authority, and goal with personalization and source trust.

7Data Retrieval and Information Retrieval Are Not the Same

Contrasts exact structured data lookup with ranked messy-store retrieval plus browsing.

8From Raw Document to Ranked Output

Builds logical views with stemming, indexes with tf-idf weights, and ranks with cosine and distance scoring.

Undergraduate students studying Information Retrieval

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

What Information Retrieval Is

Must-know: Retrieval takes a query and returns relevant documents in rank order; Boolean filters sets while vector similarity orders lists.

$scor e (d, q)$

Top pitfall: Calling plain filtering without ranking retrieval; ranking is part of the job.

Self-check: What do score(d, q) and sim(q, d) each mean?

Connects to: 1.2, 1.7

Where Retrieval Shows Up

Must-know: Cross-lingual, multimedia, recommender, RAG, library, media, and filter-funnel cases all share query to ranked list.

Top pitfall: Calling a fixed-node tree walk retrieval when no ranking happens; ranked candidates are required.

Self-check: When does tree walking count as retrieval?

Connects to: 1.1, 1.3

How a Retrieval System Runs

Must-know: Need turns into query, query plus corpus feed matching and ranking, shown by the lung cancer biomarkers walkthrough.

Top pitfall: Mixing the broad need with the short query handle; the query is only the handle.

Self-check: Name the five stages from need to ranked output.

Connects to: 1.1, 1.5

Retrieval and Language Understanding

Must-know: Retrieval finds ranked lists while language reads and writes; meaning beats bare matching and locks stay locked.

$s im (A, B) = 1$

Top pitfall: Trusting bare overlap when roles flip; dog bites man needs a different list from man bites dog.

Self-check: Why must dog bites man and man bites dog return different lists?

Connects to: 1.6, 1.8

Two Kinds of Information Need

Must-know: Retrospective varies queries over a fixed store; prospective holds a fixed ask over a growing stream.

$C_{t + 1} = C_{t}, p > 200$

Top pitfall: Calling prospective search forecasting; it waits for arrivals, never guesses them.

Self-check: Is watching election trends retrospective or prospective, and why?

Connects to: 1.3

What Counts as Relevant and Smart

Must-know: Relevance checks subject, time, source weight, and goal; smart systems add meaning and personalization.

Top pitfall: Ranking by letters alone and ignoring time, venue, goal, and feedback signals.

Self-check: Why can a 2025 Nature paper outrank a 2020 IEEE paper?

Connects to: 1.4

Data Retrieval and Information Retrieval Are Not the Same

Must-know: Data retrieval gives exact answers from structured tables; IR gives ranked relevant answers from messy stores.

$C GP A > 9.0$

Top pitfall: Expecting ranked partial credit from exact data lookup; zero is valid there.

Self-check: Why is zero a valid answer for country equals US on a home-country table?

Connects to: 1.1, 1.8

From Raw Document to Ranked Output

Must-know: Raw text turns into a logical view, then weights, then cosine or distance scoring and ranking.

$t f i df (t, d) = t f (t, d) \times i df (t), cos (A, B) = \frac{A \cdot B}{∥ A ∥∥ B ∥}$

Top pitfall: Stemming proper nouns like Chandrayaan to short roots; guard names and use lemmatization.

Self-check: Turn moon landing from India into its logical view and name the scoring step.

Connects to: 1.1, 1.7

Previous LectureLecture 16 · FedBSS Curriculum, Uncertainty Split, and FedCVT

Next LectureLecture 02 · IR Models: Boolean, Vector, and Probabilistic Retrieval

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.