Natural Language Processing

Vector Semantics and Word Embeddings

Published: 2026-08-13

Level: postgraduate

Audience: Postgraduate students in Natural Language Processing

2.1 Vector Semantics: Representing Word Meaning with Numbers

2.1.1 Why Machines Need Word Meaning

Hook: What is the meaning of life? Ask a dictionary and it answers "life" — a string of letters that explains nothing. A machine given that answer still does not know what life is, and the same problem hits every single word. Before any NLP system can answer a question, translate a sentence, or chat with you, it must first grasp what the words in that sentence mean.

Every NLP application — conversational AI, question answering, machine translation, sentiment analysis — needs a proper understanding of the language, and understanding a language means understanding its words. When we read a statement, we reason about it only after we grasp the meaning of its words; only then do we know what action to take. The challenge is how to make machines do the same.

Some ground terms first. A lexicon is the full set of words of a language — English has its own lexicon, Indian languages have their own, each with its own meanings. Lexical semantics is the study of how word meaning is represented and captured. The word semantics itself just means "meaning of words," and a sense is the meaning of one particular word; we will meet this term again when we study word sense disambiguation (deciding which meaning of a word is active in a given sentence).

A vector is a string of numbers arranged like an array with a certain dimension. For example, $(0.03, 3.4, 1.1, 0.02)$ is a four-dimensional vector. For the moment, do not worry about where the individual values (0.03, 3.4, whatever) come from; that is exactly what the rest of the module will teach. Any word can carry its own vector — lexical forms, morphological forms, and so on — and the machine can process these numeric forms far more easily than it can process letters.

Why numbers? Machines are far better at processing numbers than strings of characters. Even speech gets converted to text (or morphemes and phonemes) before any processing, so in the end everything we process is words, and words must become numbers before a machine can do anything useful with them. That is what this module — vector semantics and word embeddings — is about: how to turn every word into a vector.

Visual intuition — the map of language. Imagine a map of a country where every city is a word. If "dog" and "cat" appear in similar sentences (near "pet," "feed," "walk"), they sit close together on the map; "computer" sits far away, near "data" and "process." The distance between two cities is the similarity of the two words. This one picture — meaning as position in space — is the mental model the whole module builds on.

This is the ABCs, the foundation, of every advanced course that follows: NLP applications, conversational AI, LLMs for Gen AI, speech recognition, deep learning, transformers, and agentic AI systems. In every one of these, the input to the model travels through word embeddings. Get this right and everything downstream is easier. Real-world: ChatGPT itself feeds word embeddings into the transformers under the hood, and agentic AI systems — where an LLM is the brain — are built on the same foundation.

2.1.2 Words, Strings, and Vocabulary IDs Are Not Enough

Scope: What could go wrong if we skip the numeric representation? Two naive plans both fail, and seeing why they fail shows what the real representation must do.

Simply listing the words of the vocabulary and numbering them does not help. If we give each word a running ID — this one is index one, this one index two, this one three, this one four — those IDs carry no information about meaning. The number 5342 tells you nothing about a dog; it is just a label, like a seat number at a stadium. A string of characters is no better: just looking at the letters "dog" tells you nothing about what a dog is.

Worked example — why the dictionary answer fails. The classic joke makes the point: ask "what is the meaning of life?" and the dictionary answers "life." Read strictly, the answer is technically correct but entirely unhelpful: one string ("life") explains nothing about the concept behind the other string. The same joke works for any word — "what is a dog?" → "dog." Without meaning, a machine cannot tell whether two words are related at all: "dog" and "puppy" look nothing alike as strings, and neither does "good" and "bad."

There are also books (like Word Power Made Easy, used by many preparing for the GRE) that teach vocabulary by etymology — for example, if a word contains "logy," it is the study of something ("biology" is the study of life, "geology" of the earth). So a human can infer meaning from small pieces of a word. But that kind of knowledge is hard to hand over to a machine: it would require teaching the machine centuries of word history, and it still would not cover words we have never seen. What we need instead is a numeric representation that carries meaning, so that comparing two words becomes comparing two vectors — and comparing numbers is something a machine does naturally.

2.1.3 Student Questions and Answers

Q: In the earlier session we met stemming and lemmatization. Do these come only with NLTK, or do other libraries offer them too?

A: Both. NLTK and spaCy each have APIs for stemming and for lemmatization, so you are not locked into one library. There are libraries for both operations in NLTK as well as in spaCy.

Exam note: This module runs across two sessions. The exam is open book — and open book is a challenge for students as well as for faculty, so prepare accordingly. The reference lists that end every contact session are useful for understanding and extra knowledge, but may not be particularly useful for exams; the recorded sessions and core concepts are the priority. For extra reading, Jurafsky and Martin's textbook covers the entire vector semantics and word embedding material in its chapter on vector semantics (chapter 6 in the earlier edition; the chapter number may differ in the latest edition, but the topic is definitely there) — a free PDF is available online, including an Indian edition.

2.2 The Distributional Hypothesis

2.2.1 The Core Idea

Hook: You are reading a novel and meet a word you have never seen: oculist. You do not reach for a dictionary — you read on, and after a few sentences you already know what it means. How? You guessed from the words around it. That everyday reading trick is the single most important idea in modern NLP, and it has a famous one-line formulation: "You shall know a word by the company it keeps" (J. R. Firth, 1957).

The distributional hypothesis is the central idea behind everything in this module: words that occur in similar contexts generally have similar meanings. If you meet an unknown word, you can often guess its meaning from its neighboring words. Suppose you come across the word oculist in some text and you have no idea what it is. If the surrounding sentences talk about a doctor, about eyes, and about examinations of the eyes, you can infer that an oculist is a doctor who deals with the eyes. This is what human readers do all the time in books and technical articles, and it is exactly what we want machines to do: represent words so that their vectors capture the context around them and the relations between them.

The hypothesis, stated precisely: the meaning of a word is defined by its environment — the distribution of words around it. Two words whose surrounding words look alike are similar in meaning. That gives us a concrete computational strategy: measure the semantic similarity of two words by measuring the similarity of their contexts.

The whole point of the representation is that related words should lie close together in a numeric space. If you see the words "delicious," "cook," "leaves," and "green" around an unknown word, you conclude it is something edible, probably a leafy vegetable; you know it is not grass, because it is delicious and edible. This context-based inference is the intuition that vector semantics is built on.

2.2.2 The On Choy Worked Example

Worked example — what is on choy?

Step 1 — your first guess, purely from appearance: shown a picture of a plant and asked what it is, your first guess is "spinach."

Step 2 — now you are given context sentences about it:

it is sautéed with garlic,
it tastes good with rice,
its leaves are good with salt and sauces,
it is delicious,
it can be cooked and eaten,
it is green.

Step 3 — infer. From these neighboring words you can conclude that on choy is some kind of leafy vegetable that people eat, even if you have no idea whether it is an Indian leafy vegetable, a Chinese one, or anything else.

Step 4 — check what you did NOT learn. You did not learn the species, the origin, or the exact name — the context cannot tell you that. It told you the kind of thing it is: edible, green, leafy, vegetable-like.

Sense-check: every clue word (sautéed, garlic, leaves, delicious) also fits spinach and collard greens, so the conclusion — a leafy vegetable — is exactly the level of confidence the evidence supports. The words around the unknown word did the work.

Related words — the ones that appear in this food context — lie closer together in a semantic space than, say, a cricket bat or a mobile phone, which are not edible at all. That is the distributional intuition in action: unrelated things stay far apart, related things cluster.

Pitfalls:

Do not over-infer. The on choy example shows context reveals the category (leafy vegetable) but not every property (which cuisine it belongs to). A machine working from contexts has exactly the same limit.
Do not confuse "similar context" with "identical meaning." Words can share contexts yet differ — more on this when we reach word relations in the next section.
Do not think the hypothesis needs a dictionary. It works purely from the surrounding text, which is why it can be computed automatically from any large collection of words.

2.2.3 From Context Words to Vectors

So the plan is: represent each word by numbers, place all the words in a high-dimensional vector space, and ensure that words that are related end up lying close to each other. If a machine never sees the word "bad" in its data but sees "worst" and "terrible," the vectors alone can still reveal that these words belong together. The rest of the module builds exactly this representation — first with counting (frequency-based), later with learned embeddings.

Exam note: the distributional hypothesis is the motivating principle for every vector representation in this module; expect it to come back in any conceptual question about why word embeddings work. Remember the one-line summary — words that occur in similar contexts have similar meanings — and be ready to explain the on choy or oculist example as the intuition behind it.

Recap + bridge: "You shall know a word by the company it keeps" — meaning lives in context, so related words will live close together in a numeric space. Next we ask what kinds of relations between words actually exist (similarity, opposition, sentiment), because those relations are what the vectors must expose.

2.3 Word Relations: Similarity, Opposition, Sentiment

2.3.1 The Many Kinds of Word Relations

Hook: Is "good" related to "bad"? Most people would say yes — they are opposites. But in what sense is an opposite a "relation"? Word meaning is not a single yes/no property; words connect to each other in many different ways, and any representation of meaning has to expose them all.

Words relate to each other in many ways: some words are synonyms, some are similar, some are opposites, some are merely related by topic. A quick taxonomy of the main relations:

Synonymy — words with the same or nearly the same meaning: couch/sofa, car/automobile.
Similarity — words sharing core elements of meaning, often of the same class: cat and dog (both pets, both mammals).
Antonymy — opposites along one specific feature: hot/cold, rise/fall, good/bad.
Relatedness (association) — words that frequently appear together without meaning the same thing: coffee and cup, doctor and scalpel.
Hypernymy — an "is-a" hierarchy: animal is a hypernym of dog, dog of poodle (the more general word is the hypernym, the more specific is the hyponym).
Meronymy — part-whole relations: a wheel is a meronym of a car.

Humans acquire these relations over time — we learn that "bad" is the opposite of "good" — but machines cannot be given a dictionary of synonyms, and synonyms are not even freely interchangeable. Even if water and H2O mean the same thing, you cannot ask for "H2O" at someone's house; you cannot swap "big sister" for "large sister" either. Human beings manage through context; machines struggle.

Worked example — similarity as numbers. A numeric representation solves the problem in a way strings cannot: if you tell the machine that "vanish" and "disappear" have a similarity score of 9.8 (out of 10), while "behave" and "obey" score 7.3, the machine can use those numbers to make judgments — "vanish" and "disappear" are near-synonyms, "behave" and "obey" only loosely similar. Same two-word pair judged by a number, not by a dictionary lookup.

2.3.2 Why Human Judgment Is Too Subjective

But where would such scores come from? If a human supplies them, the exercise is subjective and does not scale. Take coffee and tea: everyone knows they are related, but they are not exactly similar, so what number do we assign? 8? 6? Different people will answer differently, and no one wants to hand-label every pair of words in a language. We need a degree of similarity measured objectively, automatically, from data — the machine must figure out on its own how related two words are. That objectivity is the second big promise of vector representations (the first being that they carry meaning at all).

A key distinction: similarity versus relatedness. "Coffee" is similar to "tea" (both hot beverages, sharing core meaning), but "coffee" is related to "cup" (they appear together constantly without meaning anything alike). A good representation must be able to place coffee close to tea on one axis and close to cup on another — and plain strings cannot express either.

2.3.3 Semantic Fields and Semantic Space

There is a lot of NLP jargon around this idea. A semantic field (also called a semantic space) is the set of words related to a concept — surgeon, nurse, anesthetist, and hospital all belong to the concept of medicine. The vectors of words inside one semantic field will be more similar to each other than to words outside it: the vector of surgeon is closer to the vector of nurse or anesthetist than to the vector of waiter.

Visual intuition — one space, many relations. Picture the semantic space as a map: the medicine neighborhood (surgeon, nurse, anesthetist, hospital) is a tight cluster; waiter lives in a different neighborhood entirely. Now look at directions inside the map. "Long" and "short" lie along the same line but in opposite directions — the opposition shows up as direction, not as distance. "Happy" and "sad" also sit far apart along a feeling axis, yet a shared feature (both are feelings) keeps them linked to the same region of the map. So vectors capture similarity (distance), opposition (direction), and sentiment (position along an emotional axis) at once, all without a human judge.

Beyond plain meaning, words also carry an emotional color: a connotation. "Cheap" and "inexpensive" both literally mean low cost, but "cheap" hints at poor quality while "inexpensive" hints at good value; "slim" sounds fit, "skinny" sounds unhealthy. This emotional layer is the foundation of sentiment analysis — the NLP task of deciding whether a piece of text is positive, negative, or neutral.

2.3.4 Why This Must Be Automatic

Every time we need to know how related two words are, we cannot run to a human for a subjectivity score. Vector semantics is the standard model in language processing for exactly this reason: it is the automatic, computable way to answer "how related are these words?" Two families of representations will appear in this module: sparse vectors, built by counting word frequencies (chiefly TF-IDF), and dense vectors, learned by prediction-based models. Both are used in the industry today.

Pitfalls:

Do not treat similar and related as the same thing: coffee is similar to tea but related to cup — a search engine that cannot tell them apart will answer "cup of tea" queries with recipes.
Do not expect one number to capture all relations: similarity, opposition, and sentiment live on different aspects of the vectors (distance, direction, region).
Do not assume synonyms are interchangeable: "big sister" is fine and "large sister" is odd, even though big and large are near-synonyms — context decides.

Real-world: in agentic AI and retrieval-augmented generation (RAG) systems, a modified version of TF-IDF called BM25 is the workhorse for matching queries to documents — it is a frequency-based, sparse representation, and it appears in the papers behind modern retrieval pipelines.

Recap + bridge: Words relate in many ways — synonymy, similarity, antonymy, relatedness, hypernymy, meronymy — and human-supplied scores are too subjective to scale. Vectors expose similarity as distance, opposition as direction, and sentiment as position. One catch: a single fixed vector per word can show these relations, but a single fixed vector per word cannot handle a word with several meanings — which is exactly the problem we look at next.

2.4 Contextual Word Embeddings: A First Look

2.4.1 One Word, Many Meanings

Hook: The word "bank" appears in "money in the bank" and in "walk along the river bank." Is it one word or two? For a machine with one vector per word, it has to be one — and that single vector has to mean two different things at once. It cannot.

A single vector per word cannot handle a word with several meanings. This is the problem of word sense in context: the sense (the meaning of one particular word in one particular context) of "bank" in a finance sentence is a financial institution, and in a geography sentence it is a riverside. Words that carry several senses are called polysemous — think of "mouse": the rodent and the computer device are two senses of one word. When two meanings of the same word are so different that they behave like two unrelated words, the technical name is homonymy: the financial bank and the river bank are homonyms — written the same, historically unrelated in meaning. The modern answer to the whole problem is contextual word embeddings, the kind used in GPT and transformer models.

Contextual word embeddings: instead of one fixed vector per word, the representation is computed afresh each time the word is used, from the surrounding words. "River bank" gets a vector shaped by river, water, and shore; "financial bank" gets a vector shaped by money, loan, and account. The two uses now have two different vectors, and the machine can compare them, relate them, or keep them apart — exactly what a fixed vector cannot do.

Real-world: BERT — the bidirectional encoder — is one such technique; its details (and the transformer math) are scheduled after the mid-semester exam, because the topic is math-heavy and the module deliberately balances simple and complex material across the two halves of the course.

Scope: what the preview covers, and what it does not. For now, the only promise to remember is the trick's shape: the numbers come from the context around the word. The mechanics — how the transformer architecture actually produces those numbers — come after the mid-semester exam. There is also more than one algorithm for producing such vectors, just as classification can be solved by SVM, logistic regression, and so on; the module will show several.

2.4.2 Student Questions and Answers

Q: Suppose we parse two sentences and they share three or four common words, but each sentence uses those words with a different meaning. How does the system detect which sentence is meant? For example, "bill" could relate to a restaurant bill and to a hospital bill.

A: That is exactly what the plain similarity measures cannot do with one fixed vector per word. This is where contextual word embeddings come in — the ones used in GPT and the transformers. Take the word "bank": a financial bank and a river bank get two separate vector representations. When I say "river bank," the values in the vector (0.1, 0.02, whatever they are) differ from the values for the financial bank, which gets a different vector. The values depend on the context around the word, and they are what let the system see how the two uses relate and how they differ. We will cover this properly after the mid-semester exam; for now, remember the trick is always the same — how do you get those numbers? And there are many algorithms, just as classification can be solved by SVM, logistic regression, and so on.

Q: Before any of this can work, we have to assign some value to every word in the universe, right? Otherwise how do we know which number gets assigned to which word?

A: Hold that question — we are about to see exactly that. The algorithm itself decides which number each word gets; you never hand-assign values word by word. We will start with the simplest algorithm in this session.

Exam note: contextual word embeddings (BERT, transformers) come after the mid-semester exam; the pre-mid-semester material deliberately keeps to the simple topics, so do not expect the heavy transformer math before mid-sem. For this session, the examinable point is the problem — one fixed vector per word cannot handle multiple senses — and the promise of the solution: context-dependent vectors.

Recap + bridge: polysemy and homonymy defeat one-vector-per-word; contextual embeddings compute the vector from context; and the algorithm — not a human — decides the numbers. Next we look at the big picture: what word embeddings are, and how we can see their structure with a visualization.

2.5 Word Embeddings and the Semantic Space View

2.5.1 What an Embedding Is

Hook: We have been saying "vector" for a while. Now the same idea gets its famous technical name — embedding — the word you will meet in every paper, library, and job description from here on. The name itself is the concept: each word's vector is embedded in a space with the other words.

Once words are represented as vectors and placed in a multidimensional space, we call the representation an embedding, because each vector is embedded in that N-dimensional space. Related words lie in the same semantic space: positive words cluster together, negative words cluster together, and so on. This is the technical name you will see everywhere — word embedding — and it solves both problems at once: it represents meaning, and it puts related words near each other.

A baseline to appreciate it against — one-hot vectors. The naive way to vectorize a vocabulary of, say, 10,000 words is one-hot encoding: give every word a vector of length 10,000 with a single 1 at its own position and zeros everywhere else, so "dog" is (0, 0, …, 1, …, 0). This is a vector, but it is a useless embedding: every pair of one-hot vectors is equally (un)related — the angle between any two is the same. Embeddings exist precisely to replace this: they compress meaning into a dense, low-dimensional form in which related words end up close. The professor's t-SNE plot below shows the payoff.

2.5.2 Visualizing Embeddings with t-SNE

To see this clustering, we plot the vectors. You have probably used matplotlib in earlier ML or DL courses; t-SNE is another visualization library that projects high-dimensional vectors into two dimensions for plotting.

Visual intuition — the t-SNE plot. Imagine a scatter plot with no meaningful axis names, because the two axes are just a compressed projection of many real dimensions. In such a plot, all the positive words (good, best, incredible, and similar words) land in one region, while the negative words (bad, worst, incredibly bad, dislike, and so on) land in another, and a third group (prepositions, roughly) sits in its own region. The landmarks are the region boundaries: you can literally point at where "good" lives and where "bad" lives. The one-sentence takeaway: meaning organizes itself into neighborhoods. The vectors are usually drawn starting from the origin (zero) — each arrow from the origin points at a word.

The machine never "knows" the meaning of bad or worst, yet from the vectors alone it can see they are related — it can even figure out that worst and bad are similar even if the word bad never appeared in its prompt.

Pitfall — the plot is a lie, deliberately. The real space is not two-dimensional — we can only draw two axes; in practice every word has its own vector in a space with many dimensions, and there are N words each with a vector. t-SNE squashes that space into two dimensions for human eyes, so exact distances in the picture are approximate. Use the plot to see clusters; do not read precise numbers off it.

2.5.3 Tokens and Their Unique Vectors

Every token has its own embedding. The definition of a token can vary — advanced tokenization algorithms (subword tokenizers) are a next-semester topic — but under the simple definition, every word is a token, and every token has its own unique vector representation.

2.5.4 Unseen Words and Generalization

Embeddings also generalize to words the model never saw in exactly the same sentence.

Worked example — understanding a sentence never seen before.

Training text contains:

"the movie was terrible"
"the weather is terrible"

Test sentence (never seen in training):

"the weather is bad and the temperature is 42 degrees"

Step 1 — notice the novel part: the training data never contained this exact sentence, and "the weather is bad" has never been seen with these words together.

Step 2 — see why it still works: bad and terrible lie close to each other in vector space, because both appear in negative contexts. The model treats "bad" as standing where "terrible" stood, so the sentence makes sense.

Step 3 — extend to generation: if the task is generation, the model can even produce a token it has not seen in that context, using the neighbors of the words it did see.

Sense-check: the sentence is understood not because of the exact words, but because of the neighborhoods of those words — which is exactly the distributional hypothesis doing its job.

String-based measures cannot do this: Levenshtein distance, edit distance, and similar string similarity measures cannot reveal that bad and terrible are related — to a machine, "mobile," "terrible," and "screen" all look alike as strings. Only the vector representation carries the meaning. We will formalize this with cosine similarity: the higher the similarity between two word vectors, the more related the words.

2.5.5 Student Questions and Answers

Q: Is one token represented by one vector embedding in the vector space, or is it stored some other way?

A: Every token gets its own unique vector. The definition of a token itself varies — we will look at advanced tokenization algorithms next semester — but with the simple definition, every word is a token and every token has its own vector representation.

Q: What does each bit of this vector mean — these numbers?

A: Each number is a feature of the word. We are about to talk about how these numbers are produced, so hold on for that.

Recap + bridge: An embedding is a word's vector embedded in an N-dimensional semantic space; it clusters related words (visible in a t-SNE plot), gives every token its own vector, and generalizes to unseen sentences because related words sit close together. That "close together" is about to become a precise formula — the next sections build the counting machinery (term frequency, context windows) and the measure (cosine similarity) that turns "close" into a number.

2.6 Frequency-Based vs Prediction-Based Embeddings

2.6.1 The Two Families

Hook: Every algorithm that produces word vectors belongs to one of two families: one family counts how often words appear, the other family predicts words from neighbors. The counted vectors are cheap and sparse; the predicted ones are powerful, heavy, and dense — and this course deliberately teaches the counting family first.

All the algorithms that produce word vectors fall into two families. Frequency-based (sparse) embeddings count occurrences — the TF-IDF family — and are simple to implement yet effective, with no resource demands: they run on CPUs, edge devices, and mobile phones, at zero cost, so the return on investment is good. Their weakness is that they cannot distinguish context-sensitive information. Prediction-based (dense) embeddings learn vectors from the task of predicting words: skip-gram and CBOW from the word2vec family, GloVe, and the contextual embeddings built on transformers. These are math-heavy, which is why the course places the dense family after the mid-semester exam — pre-mid-sem material stays simple, post-mid-sem carries the heavier math. We start with the simplest, frequency-based representation, because starting simple-to-difficult is the only way to keep it digestible.

Dimension	Frequency-based (sparse)	Prediction-based (dense)
How the numbers are made	Counted directly from the corpus (TF-IDF family)	Learned from a prediction task (skip-gram, CBOW, GloVe, transformers)
Vector length	As long as the vocabulary or document count — can be huge	Short and fixed, typically 50–300 dimensions
Zeros	Mostly zeros (sparse)	Few or no zeros (dense)
Context sensitivity	Cannot distinguish context-sensitive information	Can — contextual embeddings even change per sentence
Cost	Runs on CPUs, edge devices, mobile phones, at zero cost	Math-heavy; needs GPUs at web scale
Where it appears	Search, enterprise retrieval, BM25 in RAG	LLMs, ChatGPT, transformers, agentic AI

When to pick which: use the frequency-based family when the task is short, structured, and context-insensitive (lookups, retrieval, ranking); use the prediction-based family when the task needs real understanding of meaning in context.

2.6.2 Where Frequency-Based Representations Win

Despite its simplicity, the frequency-based approach is still used in practice, especially for short structured queries — for example, looking up course IDs such as AIML 519 or CZ519. For that kind of input, the frequency-based approach gives the better result: there is no ambiguity to resolve, so the expensive context machinery buys nothing. Real-world: at web scale, search engines and retrieval systems build on this idea; even modern pipelines use BM25, which is a modified, more advanced TF-IDF, and it shows up in RAG and agentic systems. That belongs to later courses in the NLP specialization — not everything can fit into one course of 16 two-hour sessions.

Scope: the boundary of today's material. In this session we build only the frequency-based pipeline end to end. The dense family (skip-gram, CBOW, GloVe, and transformer-based contextual embeddings) is scheduled for the following session and beyond the mid-semester boundary — do not expect the heavy transformer math before mid-sem.

2.6.3 Student Questions and Answers

Q: When we talk about embeddings, do we mean that every possible English word has to have a vector embedding stored somewhere, so that when a user interacts with any NLP system — ChatGPT or whatever — it can find the context? And won't that be billions and billions of vectors, since we cannot know the length and breadth of the word set up front?

A: Yes — and you are absolutely right about the scale. Large language models are trained at web scale, which is why they need H100 servers and months of training, and why they have so many parameters — on the order of 150 billion. Everything out there on the internet that should have a space defined for it gets one. This is all done offline: the embeddings are created during pre-training, and that is why these are called pre-trained language models. BERT and the other algorithms do this vector creation offline. If you work on a smaller corpus, you get a smaller vocabulary, and you can run the same kind of model on smaller resources. The general rule stands: whether it is an LLM, a transformer, or an agent system, the input is always vectors — you never feed these systems strings of letters.

Exam note: the dense family (skip-gram, CBOW, GloVe, contextual embeddings) is scheduled for the following session and beyond the mid-semester boundary. Skip-gram is the one place where the math gets a little exhaustive, but it will be explained in very simple words, starting from the basics — do not be scared of it, and optional pre-reading of the uploaded material is fine if you want a head start.

Recap + bridge: two families — counted (sparse, cheap, TF-IDF) and predicted (dense, heavy, word2vec family and beyond) — and both still earn their keep in industry. The rest of this session walks the counting family step by step, starting with the simplest count of all: how often a term occurs in a document.

2.7 Term Frequency: Counting Words in Documents

2.7.1 The Toy Corpus: Four Shakespeare Plays

Hook: Search engines answer "which pages match this query?" millions of times a day, yet they never read the web at query time — all the heavy work happens offline, and only a fast comparison happens per request. The trick behind that comparison is the oldest word-counting idea in this module: term frequency.

The counting idea started with search — the concept was born with Google Search, where documents (millions of web pages) must be matched to a query fast; the heavy work is done offline, and only the comparison of query vectors with document vectors happens at query time. (Search internals like the inverted index belong to an information retrieval course; here we focus on the frequency-based vector representation itself.) To see how, take a tiny toy corpus: four documents, actually four Shakespeare plays — As You Like It, Twelfth Night, Julius Caesar, and Henry V — some comedies, some serious. For simplicity we track only four words: battle, good, fool, and wit. (Real corpora have millions of words and millions of documents; the toy keeps the mechanics visible.) A term is just a word. We count how many times each term occurs in each document:

term	As You Like It	Twelfth Night	Julius Caesar	Henry V
battle	1	0	7	13
good	114	80	62	89
fool	36	58	1	4
wit	20	15	2	3

2.7.2 Word Vectors from the Document Counts

Term frequency (tf) is the count of how many times a term appears in one document — nothing more. Each cell above is a term frequency: battle occurs 13 times in Henry V, fool 36 times in As You Like It, and so on.

Read the table vertically and each column is a document vector: its numbers describe the document. Read it horizontally and each row is a word vector:

Worked example — reading the table as vectors.

Row-wise, the word vectors are:

battle = (1, 0, 7, 13)
good = (114, 80, 62, 89)
fool = (36, 58, 1, 4)
wit = (20, 15, 2, 3)

Each vector has one component per document, in the order As You Like It, Twelfth Night, Julius Caesar, Henry V. So battle = (1, 0, 7, 13) says: battle appears once in As You Like It, never in Twelfth Night, 7 times in Julius Caesar, and 13 times in Henry V — the history play about war, which is exactly where we would expect the word battle to dominate.

Column-wise, the document vectors are: As You Like It = (1, 114, 36, 20), Twelfth Night = (0, 80, 58, 15), Julius Caesar = (7, 62, 1, 2), Henry V = (13, 89, 4, 3).

Sense-check: the comedy As You Like It is heavy in fool and wit and light in battle; the serious plays are heavy in battle and light in fool. The numbers already smell like the genres — before any machinery has looked at a single word meaning.

These numbers are the raw term frequency: how important the term is in each document. Already, without any further machinery, the vectors separate related things from unrelated ones: wit and fool share similar counts and lie close, while battle and fool lie far apart. Comparing numbers is easier than comparing strings — that is the immediate payoff of the representation.

Notice one more thing in the table: good appears in every single play with a large count (114, 80, 62, 89). A word that shows up everywhere is not informative — it cannot tell plays apart. That observation is exactly what will motivate the IDF weighting later in this session.

2.7.3 Plotting Documents to Find Similarity

Visual intuition — the document plot. Take just two words: fool's counts on the X axis and battle's counts on the Y axis, and plot each document as a point:

As You Like It (36, 1)
Twelfth Night (58, 0)
Julius Caesar (1, 7)
Henry V (4, 13)

Both comedies — As You Like It and Twelfth Night — sit low and far to the right: high fool, low battle. Both serious plays — Julius Caesar and Henry V — sit high and far to the left: high battle, low fool. Two documents are related when the angle between their points is small: As You Like It and Twelfth Night lie on nearly the same ray from the origin (both fool-heavy), Henry V and Julius Caesar lie on another ray (both battle-heavy), and the angle between As You Like It and Julius Caesar is much wider — more separated. So the vector plot reveals document relatedness that the bare strings hide.

Pitfall — distance in the plot is not the measure. Notice the comedies sit far apart on the axes (36 vs 58 fool counts) yet are closely related by angle: both points hug the X axis. The relatedness measure we are building is the angle, not the straight-line distance — raw counts inflate the magnitude of long vectors, so later we will normalize by vector length before comparing. Keep this picture in mind; it becomes the cosine similarity formula in a few sections.

The same flip works for words: with the horizontal vectors, we will quantify "close" soon via cosine similarity; even by eye, wit and fool are closer than battle and fool.

2.7.4 Student Questions and Answers

Q: So the numbers here are just the number of occurrences in the book?

A: Correct. That is the simplest measure — the first level of figuring out relatedness. You still cannot confirm anything, but you can at least differentiate: these words are far from each other, those are close. Something lying in a similar semantic space is more related to its neighbors. We just know wit and fool are more related than fool and battle. This is only term frequency; IDF, normalization, and cosine similarity come next.

Q: Could a word with a similar frequency actually be an opposite word instead of a similar one?

A: Yes, it could. The relation could be synonymy, opposition, or some other relation — the point is they are not totally unrelated. Remember the edible versus not-edible example: a leafy vegetable and a phone are totally away from each other. Here we are just simplifying: a term occurring many times in the same documents means those documents discuss the same topic, so they are related — probably two of these are comedies and two are serious. Even this simple count already gives the significance of a term in a document.

Q: How did you draw these graphs with battle and fool?

A: Just by using the values. Fool is on the X axis and battle on the Y axis. For example, fool occurs in As You Like It about 36 times while battle occurs once there, so that document is the point (36, 1); every document becomes one point on the plot.

Recap + bridge: Term frequency is the raw count of a term in a document; reading the count table row-wise gives word vectors, column-wise gives document vectors, and plotting two dimensions shows that related documents share small angles. Raw counts are only the first level — next we stop counting words per document and start counting words near other words, the context-window idea that gives the second family of sparse vectors.

2.8 Context Windows: Co-occurrence Vectors

2.8.1 From Documents to Neighboring Words

Hook: The term frequency idea answered "which documents does this word appear in?" Now flip the question: "which words does this word appear near?" Counting neighbors instead of documents gives each word a vector built from the company it keeps — the distributional hypothesis made into a concrete recipe.

The same counting idea applies to contexts instead of documents. Instead of counting how often a word occurs in a document, we count how often it occurs near other words. For each target word we fix a context window of plus and minus 4 words (a hyperparameter — a value we choose, not one the data computes), and count how many times every other word appears inside that window around the target. Both sides matter: a neighboring word counts whether it sits at minus 4 (left of the target) or plus 4 (right of it).

Co-occurrence count: for a fixed target word (say cherry) and a fixed neighbor word (say pie), the co-occurrence count is the number of times pie appears inside the plus/minus 4 window around any occurrence of cherry in the whole corpus. Walk the entire corpus, look at every occurrence of cherry, and tally how often pie is within four words of it — on either side. Do this for every pair of words, and each word gets a row of counts: its context vector.

2.8.2 The Cherry, Strawberry, Digital, Information Table

This is the instructor's example, following Jurafsky: we want vectors for the words cherry, strawberry, digital, and information. First we take all the unique words in the corpus, sorted alphabetically from A to Z; in the normal scenario this is a matrix with A-to-Z words on both axes (every word is a potential target row and a potential context column), and only the target words are shown for space. Then for each target word we scan the whole corpus and count co-occurrences inside the plus/minus 4 window. For cherry, the word pie co-occurs within the window 442 times in the whole corpus; sugar co-occurs 25 times; computer rarely appears near cherry. The full picture for our four target words looks like this (context words as columns):

target word	computer	data	result	pie	sugar
cherry	2	8	9	442	25
strawberry	0	0	1	60	19
digital	1670	1683	85	5	4
information	3325	3982	378	5	13

Pick the three dimensions used in the worked examples below — pie, data, and computer — and the context vectors become:

Worked example — the context vectors.

Over the dimensions (pie, data, computer):

cherry = (442, 8, 2)
strawberry = (60, 0, 0)
digital = (5, 1683, 1670)
information = (5, 3982, 3325)

Step 1 — look at the shapes. Cherry and strawberry are heavy on pie (442 and 60) and nearly empty on data and computer. Digital and information are heavy on data and computer and nearly empty on pie.

Step 2 — predict the relations. Cherry and strawberry hang out near dessert words, so they should come out related to each other; digital and information hang out near computing words, so they should come out related to each other. (This prediction is confirmed when we compute cosine similarity in the next section.)

Step 3 — spot-check one dimension against the plot. Plotting just two dimensions — data on the X axis and computer on the Y axis — puts digital at (1683, 1670) and information at (3982, 3325): both points lie high and to the right, on nearly the same ray from the origin.

Sense-check: two points that sit on nearly the same line from the origin have similar meaning in this representation — the numbers reveal a relation between digital and information that the bare strings do not.

2.8.3 How Large Should the Window Be?

The window size can vary with the corpus. Practical guidance: people keep it at plus or minus 4 or 5 words, because the average English sentence is about 8 to 10 words long, so a window of 4–5 on each side tends to cover the sentence. This is a statistically determined practical guideline, like using 0.01 for the learning rate eta in gradient descent: nobody derives it from first principles for every new problem, everyone trusts the empirically good default.

The trade-off: more context means more computation but captures longer-range relations; less context is cheaper. A window too small (say ±1) sees only immediate neighbors and misses relations that span the sentence; a window too large starts counting words that belong to neighboring clauses, adding noise. Plus or minus 5 is enough for the majority of applications, even on large corpora.

2.8.4 Student Questions and Answers

Q: Does "context window" mean you divide the whole corpus into small windows and then check whether cherry and pie are inside the same window?

A: No, we are not splitting the corpus into parts at all. We keep the whole corpus, and for every occurrence of a word we look at the plus and minus 4 words around it. Suppose the corpus is four short documents with words like "is," "traditionally," "followed," "by," "dessert," and so on. First we list the unique words sorted A to Z — with suffix stripping, "traditionally" and "traditional" can be treated as one word. Then for every word — take "is" — we walk the entire corpus and count how many times "traditional" appears in its plus-4 window, how many times "followed" does, how many times "by" does, and so on. Every word gets the same plus/minus 4 treatment across the whole corpus. In the real world the corpus has millions of words, but the number crunching is not a big deal — a CPU can easily compute these counts.

Q: Suppose cherry appears at minus 4 and pie also appears at plus 4 in the same context — the same word pair — do we count it twice or once?

A: Twice. Those are two different occurrences of pie: one on the left side of cherry and one on the right side, so both occurrences count.

Those two answers cover the mechanics; the next pair of questions covers the practical choices:

Q: How do we decide the context window size for a bigger corpus — 400 words, 1000 words? You said plus minus 4 works for a small example.

A: Most English sentences are about 8 to 10 words long, so a window of plus or minus 5 covers a typical sentence. That is the practical guideline even for larger corpora — like the 0.01 value people use for the learning rate in gradient descent: statistically determined. More context means more computation, but it helps you capture longer context; less context is cheaper. In most scenarios people go for plus or minus 5 words, and it gives good enough context for the majority of applications.

Q: In the earlier table, cherry is in document one and computer in document four — how can computer appear in cherry's context?

A: The counts are not taken only from these four example sentences. Elsewhere in the whole corpus, cherry and computer occur close to each other, and those counts are included. This is just a toy example; the real counts are gathered over the entire corpus.

Recap + bridge: A context window of plus/minus 4 (or 5) words around every occurrence turns "the company a word keeps" into countable numbers — the co-occurrence matrix — and each target word becomes a context vector. Cherry and strawberry line up on pie; digital and information line up on data and computer. The next question is the one the whole representation has been building toward: how do we quantify "line up"? That is the dot product and cosine similarity.

2.9 Dot Product and Cosine Similarity

2.9.1 Dot Product

Hook: We keep saying "close together," "related," "similar" — but what does that mean as a number? The entire representation built so far is only as good as its yardstick, and the yardstick starts with one simple operation: the dot product.

To compare two vectors we start from the dot product: multiply the two vectors element-wise and add up the products. For $a = (a_{1}, a_{2}, \dots, a_{n})$ and $b = (b_{1}, b_{2}, \dots, b_{n})$ ,

$a \cdot b = i = 1 \sum n a_{i} b_{i} = a_{1} b_{1} + a_{2} b_{2} + \dots + a_{n} b_{n}$

Here $a$ and $b$ are two vectors of the same length $n$ , $a_{i}$ is the $i$ -th component of $a$ , and $b_{i}$ is the $i$ -th component of $b$ . The sum runs over all $n$ positions. The result is a single number — a scalar.

Worked example — the dot product of $a = (1, 2)$ and $b = (3, 4)$ .

Step 1 — multiply component by component: $1 \times 3 = 3$ and $2 \times 4 = 8$ .

Step 2 — add the products: $3 + 8 = 11$ .

So $a \cdot b = 11$ , a scalar.

Sense-check: each pair of components contributes a product, and the result is bigger when the two vectors have large values in the same positions — the dot product is high when two vectors have large values in the same dimensions.

A dot product of zero between two vectors means they are perpendicular — nothing shared in the direction sense: one vector leans entirely "north," the other entirely "east." But the dot product is blind to the sizes of the vectors: a longer vector inflates the value, so a big dot product can just mean one vector is long. In count-based matrices, frequent words like "the" and "of" have long vectors because they occur often — the raw dot product would call "the" similar to everything, conflating frequency with similarity. To remove that effect we normalize by the lengths.

2.9.2 Vector Length

The length of a vector is the square root of the sum of its squared components:

$∥ v ∥ = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2}$

The double bars $∥ v ∥$ are read "the length (or norm) of $v$ "; $v_{1}, v_{2}, \dots, v_{n}$ are the components of $v$ . Geometrically this is the distance from the origin to the tip of the vector.

Worked example — the length of $v = (1, 2)$ .

Step 1 — square each component: $1^{2} = 1$ , $2^{2} = 4$ .

Step 2 — add: $1 + 4 = 5$ .

Step 3 — take the square root: $∥ v ∥ = 5 \approx 2.24$ .

If there were a third component we would add its square inside the root: $1 + 4 + v_{3}^{2}$ .

Sense-check: the arrow (1, 2) is a little over two units long, which matches the picture of a point sitting at distance $5$ from the origin.

2.9.3 Cosine Similarity

Cosine similarity divides the dot product by the lengths of both vectors, giving the cosine of the angle between them. Where does that come from? The geometric definition of the dot product connects it to the angle $θ$ between the two vectors:

$v \cdot w = ∥ v ∥ ∥ w ∥ cos θ$

Formalize — cosine similarity step by step.

Step 1 — start from the geometric definition of the dot product: $v \cdot w = ∥ v ∥ ∥ w ∥ cos θ$

Step 2 — divide both sides by $∥ v ∥ ∥ w ∥$ to isolate the cosine of the angle: $cos θ = \frac{v \cdot w}{∥ v ∥ ∥ w ∥}$

Step 3 — substitute the component formulas for the dot product and the two lengths: $cos (v, w) = \frac{\sum _{i = 1}^{n} v _{i} w _{i}}{\sum _{i = 1}^{n} v _{i}^{2} \sum _{i = 1}^{n} w _{i}^{2}}$

Here $θ$ is the angle between the vectors, $v_{i}$ and $w_{i}$ are the components of $v$ and $w$ , and $n$ is the number of dimensions. The numerator is the dot product; the denominator is the product of the two lengths.

Visual intuition — the angle matters, not the distance. Draw both vectors starting at the origin. If they point in the exact same direction, the angle is $0^{\circ}$ and $cos (0^{\circ}) = 1$ — maximum similarity. If they are perpendicular, the angle is $9 0^{\circ}$ and $cos (9 0^{\circ}) = 0$ — no similarity. If they point in opposite directions, the angle is $18 0^{\circ}$ and $cos (18 0^{\circ}) = - 1$ . This is what we plotted in the semantic space: a small angle means high similarity. A value close to +1 means the vectors are highly similar (pointing in the same direction), close to −1 means opposite directions, and close to 0 means totally unrelated.

What it measures	Dot product $v \cdot w$	Cosine similarity $cos (v, w)$
Formula	$\sum_{i} v_{i} w_{i}$	$(\sum_{i} v_{i} w_{i}) / (∥ v ∥ ∥ w ∥)$
Sensitive to vector length?	Yes — long vectors inflate it	No — lengths divide out
Range	Unbounded	−1 to +1
Best for	Raw aligned magnitude	Comparing words in a semantic space

When to pick which: use cosine similarity for word vectors, because it measures alignment of direction — meaning — while ignoring how often the words occur. This is the standard tool for comparing word vectors, and it is the same measure used everywhere downstream — including contextual embeddings, agentic AI, and RAG.

2.9.4 Worked Example: cherry versus information

Worked example — are cherry and information related?

Take the context vectors from the previous section: cherry = (442, 8, 2) and information = (5, 3982, 3325) over the dimensions (pie, data, computer).

Step 1 — the numerator, the dot product: $442 \times 5 = 2, 210$ ; $8 \times 3982 = 31, 856$ ; $2 \times 3325 = 6, 650$ . Sum: $2, 210 + 31, 856 + 6, 650 = 40, 716$ .

Step 2 — the denominator, the product of the two lengths: $44 2^{2} + 8^{2} + 2^{2} = 195, 432 \approx 442.1$ , and $5^{2} + 398 2^{2} + 332 5^{2} = 26, 911, 974 \approx 5187.7$ . Product: $442.1 \times 5187.7 \approx 2, 293, 000$ .

Step 3 — divide: $cos (cherry, information) = \frac{40 , 716}{2 , 293 , 000} \approx 0.018$

The professor reported the result as 0.17; recomputing from the slide vectors gives about 0.018 — the spoken value was very likely 0.017. Either reading lands far from 1, so the verdict is the same: cherry and information are not related.

Sense-check: cherry is a dessert word and information is a computing word, so a cosine near 0 is exactly what the semantic space should show — and the strings "cherry" and "information" share no letters of meaning whatsoever.

2.9.5 Worked Example: digital versus information

Worked example — are digital and information related?

Take digital = (1683, 1670) and information = (3982, 3325) over the dimensions (data, computer).

Step 1 — the numerator, the dot product: $1683 \times 3982 = 6, 701, 706; 1670 \times 3325 = 5, 552, 750$

Sum: $6, 701, 706 + 5, 552, 750 = 12, 254, 456$ .

Step 2 — the denominator, the product of the two lengths: $168 3^{2} + 167 0^{2} = 5, 621, 389 \approx 2371$ , and $398 2^{2} + 332 5^{2} = 26, 911, 949 \approx 5188$ . Product: $2371 \times 5188 \approx 12.3$ million.

Step 3 — divide: $cos (d, i) = \frac{12 , 254 , 456}{12 , 300 , 000} \approx 0.996$

Sense-check: a cosine of 0.996 is almost 1 — digital and information are highly similar and lie nearly on top of each other in the semantic space. This matches the plot from the previous section, where the two points sat right next to each other on the same ray.

2.9.6 The Range of Cosine Values and Negative Similarity

Scope — where the range comes from, and the slide correction. With frequency counts as vector components, every value is positive, so the cosine always lands between 0 and 1 — never negative. In theory, negative components would give negative cosines: the vectors (10) and (−10) point in exactly opposite directions, 180 degrees apart, so their cosine is −1 — think hot versus cold. That situation does arise in practice once vectors are learned rather than counted: in skip-gram and other prediction-based embeddings the weights are trained by machine/deep learning and can be negative, so negative cosine values genuinely occur there. In frequency-based context vectors they cannot.

2.9.7 Student Questions and Answers

Q: What is a dot product? If the dot product of two vectors is zero, does that mean they have no angle distance between them, or that they are similar?

A: Let us define it precisely. The dot product is the element-wise multiplication of the two vectors, added up: with a = (1, 2) and b = (3, 4), it is 1 × 3 + 2 × 4, and you get a scalar value. A zero dot product means the vectors are perpendicular — there is no shared direction. The dot product does carry a sense of direction and distance between the two vectors, but it is blind to the sizes of the vectors: if one vector is much longer than the other, you cannot see that from the dot product alone. That is why we use cosine similarity, which also considers the sizes by dividing by the lengths.

Q: When will we get a cosine similarity of negative one? We won't go negative at all, right?

A: You are right for this setting — and I meant to correct the slide. With frequency counts as the values, every component is positive, so the cosine stays between 0 and 1; we never reach −1 here. In theory, with negative values you can get −1: take a = (10) and b = (−10), they point in exactly opposite directions, 180 degrees apart, like hot and cold. And that does happen with learned embeddings — in skip-gram and the other algorithms, the vectors come from learning the weights, so weights can be negative and the cosine angle can be negative, as you rightly said. In this frequency-counting situation the occurrences are positive numbers, so you will not see it.

Exam note: cosine similarity is the standard similarity measure for word vectors, and the worked computation (dot product, lengths, divide) follows the exact procedure shown in this session — be ready to reproduce it step by step on a pair of vectors. Remember the contrast: the dot product is blind to vector sizes; cosine similarity divides them out.

Recap + bridge: similarity is now a formula: cosine of the angle between vectors, +1 for identical direction, 0 for perpendicular, and in the counted world never negative. The cherry-and-information pair scored near 0; digital and information scored 0.996. One problem with raw counts remains — common words like "the" pollute the picture — and the next sections fix it with document frequency and IDF.

2.10 Document Frequency and Collection Frequency

2.10.1 Why Raw Counts Are Not Enough

Hook: "the" appears hundreds of times in nearly every document — so by raw term frequency, "the" is the most important word in English. Every search engine knows that is nonsense. Raw counts need a correction, and the correction starts with a second kind of frequency.

Term frequency alone misleads. A term occurring twice in a document is not necessarily twice as important as a term occurring once, and a common word — say "the" or "of" — racks up huge counts in every document without being important in any of them. We need a second ingredient that rewards rare terms: document frequency.

2.10.2 Document Frequency: A Boolean Question

Document frequency (df) asks a yes/no question for each term: out of all N documents in the collection, in how many does the term appear at least once? It never counts how many times — only presence or absence.

Document frequency is a Boolean-style count of documents. For each term $t$ and a collection of $N$ documents, $df (t)$ is the number of documents that contain $t$ at least once. A document containing the term ten thousand times still contributes exactly 1 to the document frequency. Presence or absence — never quantity.

Suppose the corpus has 100 documents, D1 through D100. If the term Romeo appears in only 1 of the 100, it is rare, and the one document containing it deserves a boost in importance: the word Romeo says "this document is about Romeo" with near-certainty, because almost no other document uses the word at all.

2.10.3 Collection Frequency: The Total Count

Collection frequency (cf) is different: the total number of occurrences of the term across all documents — the sum of the term frequencies.

Collection frequency is a plain total of occurrences. For a term $t$ , $cf (t) = \sum_{d} tf (t, d)$ — add up the term frequency of $t$ over every document in the collection. Document frequency counts documents; collection frequency counts words.

The distinction is stressed because people constantly confuse the two, and the confusion is worth resolving because some other formulas (not discussed here) do use collection frequency:

	Document frequency (df)	Collection frequency (cf)
Counts	Documents containing the term	Total occurrences of the term
Question answered	"In how many documents does the term appear at least once?"	"How many times does the term appear in total?"
Example (Romeo)	1 document out of 100	130 occurrences in total
Unit	One per document, at most	One per occurrence, unbounded

2.10.4 The Romeo and action Comparison

Worked example — same collection frequency, very different document frequency.

Corpus: 100 documents, D1 through D100.

Step 1 — Romeo: appears in only 1 document (D12, say), yet it may occur 113 times in that one document — so its collection frequency is 130 (the professor read the count as 113 at one point and 130 at another; either number makes the same point). That gives:

df(Romeo) = 1
cf(Romeo) = 130

Step 2 — action: appears in 31 of the 100 documents; its collection frequency also happens to be 130 (a coincidence — the example deliberately uses the same total). That gives:

df(action) = 31
cf(action) = 130

Step 3 — read the contrast: two words with identical collection frequency 130, but Romeo concentrates all 130 occurrences in a single document while action spreads 130 occurrences across 31 documents.

Sense-check: a small df can hide a large cf — and it is the document frequency, not the collection frequency, that will carry importance in TF-IDF. Romeo is a sharper signal: a document that contains it is very likely about it.

2.10.5 Student Questions and Answers

Q: If the term is present in only one document, isn't the collection frequency the same as the term frequency?

A: No — they are different ideas. Term frequency is the count within one particular document. Collection frequency is the total count of the term across all the documents. In this example they happen to coincide because the term appears in only one document, but in general they differ.

Q: What is the "I" in IDF?

A: Inverse. Inverse document frequency, because you take one over the document frequency — that is why it is called inverse.

Q: What is the purpose of collection frequency if it is not used in this formula?

A: To avoid a very common confusion. When people compute document frequency, they often end up counting how many times the term occurs in each document — that is collection frequency. Document frequency only checks how many documents contain the term at all. Collection frequency is used in some other formulas, which we are not discussing here; for now it exists so the two ideas stay distinct.

Recap + bridge: Raw term frequency overvalues common words; document frequency (a Boolean count of documents) rewards rare terms; collection frequency (a total occurrence count) is a separate idea people constantly mix up with it. Next we turn the document frequency into a number that grows as a term gets rarer: inverse document frequency.

2.11 Inverse Document Frequency

2.11.1 The Formula

Hook: A word that appears in every document tells you nothing — it cannot separate one document from another. A word that appears in exactly one document tells you everything: you know what that document is about. IDF turns that intuition into a number: the rarer the term, the bigger the weight.

Inverse document frequency (IDF) inverts the document frequency: the rarer a term is across the collection, the larger its IDF. With $N$ the total number of documents and $df (t)$ the number of documents containing term $t$ ,

$idf (t) = lo g_{10} (\frac{N}{df ( t )})$

Formalize — reading the formula piece by piece.

$N$ is the total number of documents in the collection.
$df (t)$ is the document frequency of term $t$ : how many documents contain it at least once.
The ratio $N / df (t)$ is the inverse: as $df (t)$ shrinks, the ratio grows. A term in every document gives $N / N = 1$ ; a term in one document out of a million gives $1, 000, 000/1$ , a huge multiplier.
The log (base 10) squashes the value — without it, on a corpus of millions of documents, the ratio would be a very small or very large number; the log base 10 keeps things in a workable range.
Note there is no +1 smoothing inside the IDF (unlike the TF formula we will see next), which is why the IDF can be exactly 0 for terms that appear in every document: $lo g_{10} (1) = 0$ .

2.11.2 Worked Example: The 37-Document Corpus

Worked example — IDF on a 37-document corpus.

In the example table, N = 37 (a toy number — real corpora run to millions and billions of documents; nothing special about 37).

Step 1 — Romeo appears in 1 of the 37 documents: $idf (Romeo) = lo g_{10} (\frac{37}{1}) = lo g_{10} (37) \approx 1.57$

Step 2 — Salad appears in 2 of the 37 documents: $idf (Salad) = lo g_{10} (\frac{37}{2}) = lo g_{10} (18.5) \approx 1.27$

Step 3 — good appears in all 37 documents: $idf (good) = lo g_{10} (\frac{37}{37}) = lo g_{10} (1) = 0$

Step 4 — one more check on a larger collection, N = 100: a word A appearing in 5 documents gets $idf = lo g_{10} (100/5) = lo g_{10} (20) \approx 1.30$ (a high weight); a word B appearing in all 100 documents gets $lo g_{10} (100/100) = lo g_{10} (1) = 0$ (zero weight).

Sense-check: the ranking falls exactly in the order rarity suggests — Romeo (rarest, 1.57), then Salad (1.27), then any term in every document (0). Rarity translates directly into weight.

2.11.3 Why Rare Terms Gain Importance

Watch what the formula does: the less a term is present across the collection, the bigger its IDF. A term that occurs in all the documents carries little importance — it discriminates nothing: knowing that a document contains "the" does not narrow down which document it is, because every document contains "the." A rare term carries a lot of importance, because its presence is informative: a document containing "Romeo" is very likely the one document about Romeo. That is the justification for the inverse: rarity signals discriminative power.

Scope — the boundary cases. The IDF is exactly 0 whenever $df (t) = N$ , because $lo g_{10} (1) = 0$ and there is no +1 smoothing in this formula (the smoothing lives in the TF formula instead). There is no way for a counted IDF to go negative, since $df (t) \leq N$ always, so the ratio inside the log is always at least 1. And a term with $df (t) = 0$ cannot occur — a term in no document never shows up in the corpus at all.

2.11.4 Student Questions and Answers

Q: Why do we get zero for good?

A: Because good appears in all 37 documents, so df(good) = 37 and N = 37: the ratio is 37/37 = 1, and log base 10 of 1 is 0. And note we are not adding the plus-one smoothing here, so the IDF really is exactly zero. Later we will multiply by the TF, and the zero will carry through.

Exam note: the worked IDF computation follows the exact procedure shown in this session — divide N by df, then take log base 10 — and the rarer-term-gets-bigger-IDF pattern is the concept behind it. Know the formula and both boundary behaviors: $idf = 0$ for a term in every document, and the largest IDF for a term in one document.

Recap + bridge: IDF rewards rarity: $lo g_{10} (N / df (t))$ , from 1.57 for Romeo down to exactly 0 for a term in every document. Now we combine the two ingredients — log-smoothed TF for within-document importance, IDF for across-collection rarity — into the workhorse of information retrieval: TF-IDF.

2.12 TF-IDF: Term Frequency Times Inverse Document Frequency

2.12.1 Smoothing the Term Frequency with log and +1

Hook: Term frequency says "how often," and IDF says "how rare." Used alone, TF drowns in common words and IDF cannot tell how relevant a term is to one particular document. The industry answer is to multiply the two — but first the term frequency gets two adjustments of its own.

Before combining, the term frequency itself gets two adjustments. First, instead of raw counts we take the log — at web scale, base 10 is the convention (base 2 is sometimes used for easier calculation). Logs make computations faster: multiplications turn into additions in the log domain. Second, we add 1 inside the log:

$tf (t, d) = lo g_{10} (count (t, d) + 1)$

Formalize — the two adjustments, one at a time.

$count (t, d)$ is the raw number of times term $t$ occurs in document $d$ .
Adjustment 1 — the log: it squashes the counts. "The" with 1000 occurrences maps to $lo g_{10} (1001) \approx 3.0$ , while "data" with 10 occurrences maps to $lo g_{10} (11) \approx 1.04$ . The gap between a common word and a moderately frequent word shrinks from 990 raw occurrences to under 2 log units — reducing the bias toward "the."
Adjustment 2 — the +1: this is smoothing, to accommodate zero occurrences. A word absent from a document would otherwise give log of zero, which is undefined; with the +1 you get a small but finite value, $lo g_{10} (0 + 1) = 0$ exactly. This is the same spirit as Laplace smoothing — the plus-one trick you met in Naive Bayes.

So the concept of term frequency is unchanged — counting how often each term occurs in each document — only the raw number is now squashed into a decimal value.

2.12.2 The TF-IDF Combination

Industry practice is to use both factors together, not either alone:

$tf-idf (t, d) = tf (t, d) \times idf (t)$

Why the product works. The formula balances two desires: the term gets credit for occurring often in the document (TF), and credit for being rare across the collection (IDF). A common word such as "the" scores near zero even with a huge count, because its IDF is near zero: $tf \times idf \approx big \times 0$ . A rare, discriminative term scores high: a moderate TF multiplied by a large IDF. The output values are fractions because of the logs.

Visual intuition — the two-factor balance. Picture two dials: one dial (TF) turns up the weight as the word appears more in the document; the other dial (IDF) turns up the weight as the word appears in fewer documents. A word only scores when both dials agree it is meaningful — frequent in this document AND rare elsewhere. A word that is frequent everywhere (the) has dial two at zero, so the product stays at zero no matter what dial one reads.

2.12.3 Worked Example: wit in the Shakespeare Corpus

Worked example — the TF-IDF of wit in As You Like It.

Step 1 — the smoothed term frequency. Wit occurs 20 times in As You Like It (the first document of the four-play table): $tf (wit, d_{1}) = lo g_{10} (20 + 1) = lo g_{10} (21) \approx 1.322$

Step 2 — the IDF, from the 37-document collection. Wit appears in 34 of the 37 documents: $idf (wit) = lo g_{10} (\frac{37}{34}) = lo g_{10} (1.0882) \approx 0.037$

Step 3 — multiply: $tf-idf (wit, d_{1}) = 1.322 \times 0.037 \approx 0.049$

Step 4 — the key property: the IDF is the same for every document, because IDF depends only on the collection, not on the document. Wit's IDF is 0.037 whether we look at As You Like It, Twelfth Night, Julius Caesar, or Henry V — the whole column of IDF values repeats across documents. That is the concept of collection-level statistics in action: IDF is computed once per collection, TF once per document.

Sense-check: 0.049 is a small fraction, as expected — wit is common across the collection (34 of 37 documents), so its rarity bonus is tiny; the weight comes mostly from wit being frequent in this particular comedy.

2.12.4 TF-IDF Vectors and Ranking

Replacing the raw counts with TF-IDF values gives each word a TF-IDF vector, and every word in the vocabulary now has such a vector. The value 0.049 physically signifies the importance of the term wit in that document. TF-IDF serves two purposes: it measures importance of a term within a document, and it supports ranking.

Worked example — ranking search results for "battle".

Suppose someone searches for the word battle: should we return As You Like It, Twelfth Night, Julius Caesar, or Henry V?

Step 1 — look up the raw counts of battle in the four plays: As You Like It 1, Twelfth Night 0, Julius Caesar 7, Henry V 13.

Step 2 — apply the smoothing and the log to each count:

Twelfth Night: $lo g_{10} (0 + 1) = 0$
As You Like It: $lo g_{10} (1 + 1) = lo g_{10} (2) \approx 0.301$
Julius Caesar: $lo g_{10} (7 + 1) = lo g_{10} (8) \approx 0.903$
Henry V: $lo g_{10} (13 + 1) = lo g_{10} (14) \approx 1.146$

Step 3 — multiply each by $idf (battle)$ , the same number for every document. Since IDF is a constant per term, it cannot change the ordering — the document with the highest weight for battle comes first, then the next, then the next.

Step 4 — rank: Henry V first, then Julius Caesar, then As You Like It, then Twelfth Night. Henry V carries the maximum battle weight and so is the first result — exactly right for a history play about war.

Sense-check: the log preserves the order of the raw counts (it is a strictly increasing function), so the ranking matches the raw counts: 13 > 7 > 1 > 0.

TF-IDF is also a type of word embedding — a sparse one, because the vectors are long and mostly zero — but it is the same family of ideas that later feeds dense word embeddings.

2.12.5 Student Questions and Answers

Q: Can we not use the same cosine similarity and context-window idea with term frequency that you showed earlier?

A: We can. Any vector similarity works the same way: take the dot product of the two vectors and divide by the sizes of the vectors. Cosine similarity applies to any pair of vectors — TF-IDF vectors here, and later the contextual embeddings from BERT, agentic AI, and RAG use it everywhere too. That is why these ideas are the ABCs of the advanced AI/ML technologies — you should be very clear on them.

The next two questions circle the smoothing: where the plus one comes from, and what the resulting number really means.

Q: In the formula, TF is log base 10 of 20 plus one — why is that one added?

A: To avoid the zero. If the count were zero, log of zero would be undefined (or minus infinity). The +1 smoothing guarantees a small but finite value. It is the same concept as Laplace smoothing — the plus-one and divide by vocabulary trick from Naive Bayes.

Q: What does the TF-IDF value 0.049 physically signify?

A: It is the importance of the term wit in that document. It tells you how important the term is within the document, and comparing values across words lets you figure out the relations among words better than raw frequencies alone. It also helps with ranking: someone searches for "battle" — do we show As You Like It, Twelfth Night, Julius Caesar, or Henry V? The play with the highest weight for battle is shown first, then the next, and last the lowest. The same ideas feed into word embeddings later.

Two more questions pin down what the vector is and where its zeros come from.

Q: So to summarize: TF-IDF is a technique that creates a vector — for battle, a four-dimensional vector where one dimension is As You Like It, the second is Twelfth Night, and so on?

A: Correct, exactly that. And one clarification: battle is not a scalar — the whole row, all the document values together, is the vector; each document is one dimension.

Q: When we found zeros in the term frequency table we added one; here, while calculating the TF-IDF value, aren't we adding a weightage to every value even when there are zeros?

A: The zeros in the TF-IDF table come from the IDF, not the TF. Good has a term frequency of 114 in one document, but IDF(good) = 0, so 0 multiplied by whatever the TF is gives 0 — because we are not doing the smoothing in the IDF. Even if we smoothed the IDF, the value would be very close to zero. So yes: if the IDF is 0, the TF-IDF is 0 even when the term occurs in the document — a term that appears in every document, like "the," is not important no matter how often it occurs.

The last three questions clean up table-reading confusions and the edges of the method.

Q: Some articles say that instead of adding ones in the place of zeros, we should remove values from the existing numbers to balance the matrix — say a 5 by 4 matrix with one zero in a cell, take values out of the other cells?

A: No — that is manipulating the data, and it is not correct. You cannot change the actual data to make your matrix look nicer. And in practice, with a billion-word vocabulary, many elements are exactly zero or almost zero for every document — that is the sparsity problem, and no amount of reshuffling fixes it.

Q: The "TF" in the TF-IDF title and the final TF-IDF value — are they the same?

A: The raw count is the term frequency only: it says good occurs 114 times in As You Like It, with no document frequency and no log. The TF-IDF value is the multiplication of the term frequency and the IDF. In the table the two numbers happened to sit in the same row, which confused everyone — they are different things.

Q: Is the dimension of a term's vector always derived from the number of documents?

A: For the document-based representation, yes — the dimension equals the number of documents. For the context-window representation, the dimension equals the number of unique words in the vocabulary. Either way the size is very long, and that is exactly the problem we will discuss next.

Q: What if there is a spelling mistake that happens to be a real word — say a typo of battle? Won't TF-IDF treat it as a rare word and give it importance?

A: Spell check is a completely different application with its own algorithms — it is not as easy as it looks. You need edit distance, you need to find probable correct words that are close to the wrong word, and there are many nuances. TF-IDF is not used for that. If you take the NLP applications course you will see how spell check is actually implemented; it is the basic, classic NLP application, but it has nothing to do with TF-IDF.

Exam note: expect the worked computations — log-smoothed TF, IDF, TF-IDF product, cosine similarity — to follow the exact procedure shown in this session: $tf = lo g_{10} (count + 1)$ , $idf = lo g_{10} (N / df)$ , then multiply. Remember that zeros in a TF-IDF table come from the IDF, never from the TF.

Recap + bridge: TF-IDF multiplies a log-smoothed, within-document term frequency by a collection-level IDF, producing a sparse vector per word and ordering search results by weight. The representation works — but the vectors are astronomically long and mostly zeros. That size problem is exactly what motivates the next section's question: can we build something smaller and denser?

2.13 Problems with Sparse Vectors and the Road to Dense Embeddings

2.13.1 The Billion-Dimensional Problem

Hook: Give every word a vector the size of the vocabulary, and a single word's vector at web scale has about one billion components. One billion numbers — mostly zeros — for the word "the" alone. That is the problem with the frequency-based representations, and it is the reason the entire field moved on.

The real challenge with the frequency-based representations is size. With the context-window approach, the vector dimension is the size of the unique vocabulary; at web scale that can reach about 1 billion. Every word's vector would then have 1 billion components — a very, very long vector. Storing it and searching with cosine similarity needs enormous compute power. The document-based TF-IDF has the same disease: with a billion documents, each word vector has a billion dimensions.

The dimension bill, in one place. Context-window vectors: dimension = number of unique words in the vocabulary (about 1 billion at web scale). Document-based TF-IDF vectors: dimension = number of documents in the collection (also about 1 billion at web scale). In a small example the dimension was 4 (four plays); in production it is a billion — nothing about the idea changed, only the size.

This representation still serves web search and enterprise search — it is used even today as one parameter among many in ranking — but the size is a genuine problem. There are other frequency-based measures as well — pointwise mutual information (PMI) is similar in spirit to TF-IDF — but TF-IDF is the most common, which is why this module focuses on it.

2.13.2 Sparsity Wastes Computation

The vectors are not just long, they are sparse: most elements are 0 or close to 0, because no word appears in most documents and no context word appears near most other words. A 1-billion-word vocabulary means each document's vector is mostly zeros — unnecessary information lying idle in the vector space. Both the length and the sparsity cost computation.

Worked example — a sparse vector versus a dense one.

A sparse TF-IDF vector for a typical word looks like this (shown conceptually): $[0, 0, 5.2, 0, \dots, 0, 1.1, 0, \dots] length 50, 000 (or a billion at web scale)$

Most cells are zero: the word appears in almost no documents and co-occurs with almost no other words. A dense word2vec-style vector looks like this: $[0.25, - 0.11, 0.67, \dots, 0.03] length 300$

Every cell carries a real value; nothing sits idle. The sparse vector is 50,000 numbers wide to say what the dense vector says in 300.

Sense-check: the same word, the same meaning, but 99% of the sparse vector's storage and compute is spent on zeros — and the zeros are exactly what the dense family throws away.

Some students wonder whether the zeros could just be removed or rebalanced; that would be manipulating the data, so the standard answer is to find a different representation entirely.

2.13.3 How Research Moves: The Pattern

This is how research happens: every solution has challenges, and the next solution addresses them. TF-IDF's long, sparse vectors motivated the search for smaller, denser vectors. The same pattern repeats across the field: LLMs hallucinate, so RAG and fine-tuning appeared; LLMs need huge compute and have an expensive carbon footprint, so nano-models, mini-models, and edge models appeared — smaller versions of the larger language models that run on constrained hardware; agents brought new problems, so ethical AI and responsible AI followed. Each generation of models fixes a known weakness of the previous one.

Scope — why the frequency family still matters. The sparse representation is not dead: it still serves web search and enterprise search today, one parameter among many in ranking, and BM25 in modern retrieval pipelines. The question is not "which one is correct" but "which one fits the task": short structured lookups still favor the counted vectors, while meaning-in-context favors the learned ones.

2.13.4 What Comes Next: Dense Embeddings

The alternative we are moving toward is the dense representation: can we produce a vector that is shorter and has few or no zeros — a small, dense, non-zero vector that still represents the word well? That is the task of the prediction-based family: skip-gram and CBOW (the word2vec family), GloVe, and the contextual embeddings built on the transformer architecture. Where the sparse vectors are counted, these are learned: the values come out of a training process, not out of a tally.

Exam note: the dense family (skip-gram, CBOW, GloVe, contextual embeddings) is scheduled for the following session and beyond the mid-semester boundary, so do not expect the heavy transformer math before mid-sem. Skip-gram is the one place where the math gets a little exhaustive, but it will be explained in very simple words, starting from the basics — do not be scared of it, and optional pre-reading of the uploaded material is fine if you want a head start.

Recap + bridge: sparse vectors work but cost a billion dimensions of mostly idle zeros; research moves in fix-the-previous-weakness cycles; the next session's dense embeddings replace counting with prediction to get short, full vectors. Before we close the lecture, there is one practical stop: the two sklearn functions that implement the whole counting pipeline in a few lines.

2.14 Implementation: Count Vectorizer and TF-IDF in sklearn

2.14.1 CountVectorizer

Hook: Everything built in this session — tokenize, count, weigh, compare — reduces to two ready-made functions in the scikit-learn (sklearn) library. The coding is not rocket science: you hand over clean text and get vectors back.

The whole TF-IDF idea reduces to two ready-made functions in the scikit-learn (sklearn) library, so the coding is not rocket science — you hand over clean text and get vectors back. The first function, CountVectorizer, does exactly the plain counting we started with.

CountVectorizer — what it does.

Purpose: turn a corpus of documents into the raw count matrix of section 2.7 — the term-document counts — without writing any counting loop.
Inputs & outputs: you pass a collection of documents (lists of strings, one per document); it returns a matrix whose rows are documents, whose columns are unique words, and whose cells are the counts. The output rows are sparse: most cells are zero, because no document contains every word.
Steps inside: tokenize each document into words, build the vocabulary of unique words, then count how many times each vocabulary word occurs in each document — the plain term frequency.

These sparse count vectors are exactly the representation whose challenges we discussed — the long, mostly-zero vectors of section 2.13.

2.14.2 TfidfVectorizer

The second function, TfidfVectorizer, performs the full TF-IDF computation automatically.

TfidfVectorizer — what it does.

Purpose: compute the complete TF-IDF representation — term frequencies, document frequencies, logs, and the multiplication — in one call.
Inputs & outputs: you pass the same documents plus parameters, such as normalization options; the function returns the TF-IDF values as decimals, one per term per document.
Steps inside: compute $tf (t, d) = lo g_{10} (count (t, d) + 1)$ , compute $idf (t) = lo g_{10} (N / df (t))$ , and multiply the two. You do not implement any of the arithmetic yourself.

The same code also covers preprocessing: after stemming and lemmatization the counts change (different word forms collapse into one), and without them each form is treated as a separate word.

2.14.3 Preprocessing: Stemming and Lemmatization

The demo code shows the pipeline end to end: tokenization (splitting the text into words, as seen in the previous session), then stemming and lemmatization, then counting. The code works for other languages too, not only English. The code was shared in open form so everyone can run it; a webinar is planned where the code for skip-gram, CBOW, and GloVe implementations will be shown hands-on — participants execute the code in parallel in the virtual labs, so the session is interactive rather than a passive walkthrough.

Scope — what the functions do not do. They compute from the corpus you give them; they carry no built-in vocabulary of the whole world. At web scale the embeddings are pre-computed offline — that is why training needs so many GPUs, and why only a few big players (Claude, Gemini, Microsoft, and the xAI/OpenAI side) run that market. On a smaller corpus you can run the same kind of pipeline yourself on small resources.

2.14.4 Student Questions and Answers

Q: In the TF-IDF example, does the library already have access to the whole corpus and frequencies, or do we have to pass it?

A: You pass the text. The library computes everything from the corpus you give it — there is no built-in vocabulary of the whole world; that would not be useful anyway. For a real-world web corpus it will create the vectors quickly too. For the biggest corpora the embeddings are pre-computed offline — that is why training needs so many GPUs, and why only a few big players — Claude, Gemini, Microsoft, and the xAI/OpenAI side — run that market. Later you will see that word embeddings are pre-computed for web-scale corpora, and you need a lot of GPUs to do that; on a smaller corpus you can do it yourself on small resources.

Q: In my previous reading, term frequency was a ratio of the number of words against the number of documents. Is that how it is used?

A: No — term frequency is a very simple concept. It is just the frequency of that term in that document: count how many times the term occurs in the document. That's it.

One question about search deserves its own answer, because it is the real-world story of this entire session:

Q: You said TF-IDF is used in Google search. Is it still used?

A: Yes, but its role changed. Earlier, TF-IDF was essentially the only ranking signal. Today it is one parameter among many. Search engines create vectors for all those billions of web pages offline — pre-computed in data centers scattered all over the world — and only the comparison of the query with the pre-computed vectors happens at query time.

Recap + bridge — the session in two functions. CountVectorizer produces the raw count matrix; TfidfVectorizer applies log-smoothed TF times IDF in one call; preprocessing (tokenize, stem, lemmatize) feeds both; and the same pipeline that ranks billions of web pages runs on your laptop for a smaller corpus. This closes the frequency-based half of vector semantics — the next session opens the prediction-based half with skip-gram.

Exam Guidance Summary

The module spans two sessions: this one covers vector semantics through TF-IDF; the following one covers the skip-gram model, which needs a bit more mathematical understanding. TF-IDF is comparatively extremely easy; skip-gram is the one place where the math gets a little exhaustive, but it will be explained in very simple words, starting from the basics — do not be scared of it, and optional pre-reading of the uploaded material is fine if you want a head start.
The exam is open book — open book is always a challenge for students as well as for faculty, and closed book is easier to set questions for; expect the open-book format to be the one that matters for how you prepare.
The reference lists that end every contact session are useful for understanding and extra knowledge, but "may not be particularly useful for exams" — do not treat them as exam material; the recorded sessions and core concepts are the priority.
Jurafsky and Martin's textbook (chapter 6 in the earlier edition; the chapter number may differ in the latest edition, but the topic is definitely there) covers the entire vector semantics and word embedding material — refer to it for extra reading; a free PDF is available online, including an Indian edition.
Contextual word embeddings (BERT, transformers) come after the mid-semester exam; pre-mid-sem material deliberately keeps to the simple topics, so do not expect the heavy transformer math before mid-sem.
Expect conceptual understanding of the distributional hypothesis, sparse versus dense representations, and the mechanics of the frequency-based pipeline; the worked computations (log-smoothed TF, IDF, TF-IDF product, cosine similarity) follow the exact procedure shown in this session — rehearse the wit and battle examples until the arithmetic is automatic.
Open-book preparation tips from the session: know the exact formulas (smoothed TF, IDF, TF-IDF product, cosine similarity) with every symbol named, keep the four-play and 37-document numbers ready to hand, and be able to explain why TF-IDF zeros come from the IDF rather than the TF.

Key Industry Applications

Real-world: word embeddings are the universal input to modern NLP — transformers, LLMs, ChatGPT, conversational AI, question answering, machine translation, sentiment analysis, speech recognition, and agentic AI all consume embeddings; an LLM is the brain of agentic AI, and its input travels through word embeddings.
Real-world: Google Search pioneered this vector idea. TF-IDF was originally the only ranking signal; today it is one parameter among many. Document vectors for billions of web pages are pre-computed offline in data centers around the world (from Vishakhapatnam to everywhere), and only query-versus-vector comparison happens at request time.
Real-world: RAG and agentic AI systems use BM25 — a modified, more advanced version of TF-IDF — for matching queries to documents; it is a frequency-based, sparse representation that appears in modern retrieval papers.
Real-world: enterprise search systems and course-identifier lookups (course IDs such as AIML 519 or CZ519) still favor the simple frequency-based approach; it gives better results for such short structured queries.
Real-world: frequency-based embeddings run on CPUs, edge devices, and mobile phones with zero cost — a strong return on investment whenever context sensitivity is not required; dense, learned embeddings (skip-gram, CBOW, GloVe, BERT) trade that cheapness for context awareness.
Real-world: large language models are trained at web scale on H100 servers for months, with on the order of 150 billion parameters; embeddings are pre-computed offline by algorithms such as BERT, and only a handful of companies — Claude, Gemini, Microsoft, and the xAI/OpenAI side — run that market.
Real-world: t-SNE plots (a visualization library, in the spirit of matplotlib) are the standard way to eyeball embedding spaces — positive words, negative words, and prepositions form visible clusters, which is how practitioners sanity-check learned representations.

NLP Lecture 2 Notes · Vector Semantics and Word Embeddings

Natural Language Processing· postgraduate· 2026-08-13

Sections Breakdown

12.1 Vector Semantics: Representing Word Meaning with Numbers

Machines need word meaning in numeric form; a lexicon, lexical semantics, and a sense are defined, and a vector is introduced as a string of numbers that can carry meaning. Word IDs and strings fail because they carry no meaning, which motivates the module's numeric representation of words.

22.2 The Distributional Hypothesis

Words that occur in similar contexts have similar meanings, captured by Firth's 'you shall know a word by the company it keeps'; the oculist and on choy examples show context reveals category. The plan is to place words as vectors in a high-dimensional space so related words cluster.

32.3 Word Relations: Similarity, Opposition, Sentiment

Words relate through synonymy, similarity, antonymy, relatedness, hypernymy, and meronymy; human-supplied similarity scores are subjective and do not scale, so vectors must expose similarity, opposition, and sentiment automatically in a semantic field/space.

42.4 Contextual Word Embeddings: A First Look

A single fixed vector per word cannot handle polysemous or homonymous words (bank, bill, mouse); contextual word embeddings compute a fresh vector from the surrounding context, as used in GPT, transformers, and BERT, and are scheduled after the mid-semester exam.

52.5 Word Embeddings and the Semantic Space View

An embedding is a word's vector embedded in an N-dimensional semantic space that clusters related words; t-SNE plots reveal the clusters, every token has its own vector, and embeddings generalize to unseen sentences because related words sit close together, which string measures cannot capture.

62.6 Frequency-Based vs Prediction-Based Embeddings

Word vector algorithms split into frequency-based (sparse, counted, cheap: TF-IDF family) and prediction-based (dense, learned, math-heavy: skip-gram, CBOW, GloVe, transformers); frequency-based methods win on short structured queries like course IDs, and web-scale embeddings are pre-computed offline.

72.7 Term Frequency: Counting Words in Documents

Term frequency is the raw count of a term in a document; the four-play Shakespeare corpus yields word vectors as rows and document vectors as columns (battle = (1, 0, 7, 13) over As You Like It, Twelfth Night, Julius Caesar, Henry V), and plotting fool against battle shows related documents share small angles.

82.8 Context Windows: Co-occurrence Vectors

Instead of counting words per document, count how often every word appears inside a plus/minus 4-word context window around each target; the co-occurrence matrix gives context vectors such as cherry = (442, 8, 2) and information = (5, 3982, 3325) over pie, data, computer, with the window size set to plus/minus 5 by the 8-10 word average sentence.

92.9 Dot Product and Cosine Similarity

The dot product sums element-wise products of two vectors (1,2)-(3,4) = 11 but is blind to vector sizes; vector length is the square root of the sum of squared components; cosine similarity divides the dot product by both lengths, giving cos of the angle, with worked examples cherry-information about 0.018 (professor said 0.17) and digital-information about 0.996.

102.10 Document Frequency and Collection Frequency

Document frequency is a Boolean count of how many documents contain a term at least once; collection frequency is the total number of occurrences across all documents. Romeo (df 1, cf 130) versus action (df 31, cf 130) shows a small df can hide a large cf.

112.11 Inverse Document Frequency

IDF inverts document frequency: idf(t) = log10(N / df(t)), so rarer terms get bigger weights; Romeo in 1 of 37 documents scores about 1.57, Salad in 2 scores about 1.27, and a term in every document (good) scores exactly 0.

122.12 TF-IDF: Term Frequency Times Inverse Document Frequency

TF-IDF multiplies log-smoothed term frequency (log10(count + 1)) by inverse document frequency (log10(N/df)): wit in As You Like It scores 1.322 x 0.037 = 0.049, the vector dimension equals the number of documents, and search ranking puts Henry V first for battle (count 13).

132.13 Problems with Sparse Vectors and the Road to Dense Embeddings

Sparse vectors are astronomically long (vocabulary size or document count, about 1 billion at web scale) and mostly zeros, wasting compute; research moves in fix-the-previous-weakness cycles, and the road forward is the dense, learned prediction-based family (skip-gram, CBOW, GloVe, transformer contextual embeddings).

142.14 Implementation: Count Vectorizer and TF-IDF in sklearn

CountVectorizer produces the raw term-document count matrix, and TfidfVectorizer computes the full TF-IDF values automatically from the corpus you pass; preprocessing runs tokenization then stemming/lemmatization, and TF-IDF evolved from the only ranking signal in Google search to one parameter among many.

15Exam Guidance Summary

Exam guidance carried through: two-session module split with skip-gram next session, open-book exam format, reference lists not exam material, Jurafsky and Martin chapter 6 for extra reading, contextual embeddings after mid-sem, and worked computations following the exact procedure shown in this session.

16Key Industry Applications

Industry applications carried through: embeddings as the universal model input, Google Search's TF-IDF history, BM25 in RAG and agentic AI, frequency-based methods for short structured queries and edge devices, web-scale pre-training by a few big players, and t-SNE for sanity-checking embedding spaces.

Postgraduate students in Natural Language Processing

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

2.1 Vector Semantics: Representing Word Meaning with Numbers

Must-know: Lexicon is the full word set of a language; lexical semantics is the study of word meaning; a sense is one meaning of a word. Words must become vectors (strings of numbers) because IDs and strings carry no meaning, and every modern NLP model feeds on word embeddings.

⚠️ Top pitfall: Thinking a vocabulary ID or a string like "dog" carries meaning; it carries none — only a numeric vector can.

Self-check: Why does the dictionary answer "life" fail as a meaning of life?

2.2 The Distributional Hypothesis

Must-know: The distributional hypothesis: words that occur in similar contexts generally have similar meanings; it is the motivating principle for all vector representations in this module and can be probed with the oculist/on choy example.

⚠️ Top pitfall: Over-inferring from context: on choy is revealed to be a leafy vegetable but not which cuisine; context gives the category, not every property.

Self-check: How would you guess the meaning of an unknown word from its neighbors?

Connects to: 2.3

2.3 Word Relations: Similarity, Opposition, Sentiment

Must-know: A semantic field (semantic space) is the set of words related to a concept; vectors make similarity visible as distance, opposition as direction, and sentiment as position, all automatically. Synonyms are not freely interchangeable (water vs H2O, big vs large sister).

⚠️ Top pitfall: Confusing similarity with relatedness: coffee is similar to tea but related to cup; similarity scores from humans are subjective and do not scale.

Self-check: Why can the vector of surgeon be closer to nurse than to waiter without any human labeling?

Connects to: 2.2, 2.4

2.4 Contextual Word Embeddings: A First Look

Must-know: A single vector per word cannot handle multiple senses; contextual word embeddings (GPT, transformers, BERT) give the word a context-dependent vector, so river bank and financial bank get different representations.

⚠️ Top pitfall: Expecting one fixed vector per word to resolve ambiguity like bill in a restaurant bill versus a hospital bill; the plain similarity measures cannot do that.

Self-check: Why does the word bank need two different vectors?

Connects to: 2.3, 2.5

2.5 Word Embeddings and the Semantic Space View

Must-know: A word embedding is the vector of a word embedded in an N-dimensional space; related words cluster (visible with t-SNE), every token has its own vector, and embeddings generalize to unseen sentences where strings and edit distance cannot.

⚠️ Top pitfall: Reading the 2D t-SNE plot as the real space; the true space has many dimensions and the plot is only a projection. Also treating tokens as fixed words, when advanced tokenization is a separate topic.

Self-check: Why can a model understand 'the weather is bad' after only ever seeing 'the weather is terrible'?

Connects to: 2.2, 2.9

2.6 Frequency-Based vs Prediction-Based Embeddings

Must-know: Two families of embeddings: frequency-based (sparse, counted, cheap, TF-IDF family) and prediction-based (dense, learned, math-heavy: skip-gram, CBOW, GloVe, transformer contextual embeddings). The input to any LLM, transformer, or agent system is always vectors, never strings.

⚠️ Top pitfall: Expecting frequency-based vectors to distinguish context-sensitive information; that is their named weakness and the reason the prediction family exists.

Self-check: Why are web-scale embeddings created offline during pre-training?

Connects to: 2.5, 2.13

2.7 Term Frequency: Counting Words in Documents

Must-know: Term frequency is just the count of how many times a term occurs in a document. In the four-play corpus, battle = (1, 0, 7, 13) over (As You Like It, Twelfth Night, Julius Caesar, Henry V); comedies are fool-heavy, serious plays are battle-heavy, and related documents meet at small angles.

⚠️ Top pitfall: Reading straight-line distance in the document plot as relatedness; the measure is the angle between the vectors, and raw counts inflate vector length.

Self-check: What is the vector of fool in the four-play corpus?

Connects to: 2.8, 2.9

2.8 Context Windows: Co-occurrence Vectors

Must-know: Co-occurrence counts come from a context window of plus/minus 4 words around every occurrence of a target word across the whole corpus; the row of counts is its context vector. Cherry = (442, 8, 2), strawberry = (60, 0, 0), digital = (5, 1683, 1670), information = (5, 3982, 3325) over (pie, data, computer).

⚠️ Top pitfall: Thinking the context window means splitting the corpus into chunks; the window slides over the whole corpus around each occurrence, and both sides (minus 4 and plus 4) count separately.

Self-check: If pie sits at minus 4 and at plus 4 around two different cherry occurrences, how many times does pie count for cherry?

Connects to: 2.7, 2.9

2.9 Dot Product and Cosine Similarity

Must-know: Cosine similarity = dot product divided by both lengths: cos(v,w) = sum_i v_i w_i / (sqrt(sum_i v_i^2) sqrt(sum_i w_i^2)). With frequency counts every component is positive so cosine stays in [0,1]; learned embeddings can give negative values. cherry vs information: 0.018; digital vs information: 0.996.

$cos (v, w) = \frac{\sum _{i = 1}^{n} v _{i} w _{i}}{\sum _{i = 1}^{n} v _{i}^{2} \sum _{i = 1}^{n} w _{i}^{2}}$

⚠️ Top pitfall: Calling the dot product a distance or a similarity: it is blind to the sizes of the vectors, so a longer vector inflates the value; divide by the lengths to get cosine similarity.

Self-check: Why can the cosine never be negative in this lecture's frequency-counting vectors?

Connects to: 2.8, 2.12

2.10 Document Frequency and Collection Frequency

Must-know: Document frequency counts documents containing the term at least once (Boolean); collection frequency counts total occurrences. Romeo: df = 1, cf = 130; action: df = 31, cf = 130. The IDF formula divides by df, not cf.

⚠️ Top pitfall: Confusing collection frequency with document frequency: df only checks presence of the term in each document, never how many times it occurs.

Self-check: Two words both have cf = 130; can their df differ, and why does that matter?

Connects to: 2.11

2.11 Inverse Document Frequency

Must-know: idf(t) = log10(N / df(t)). The rarer the term across the collection, the larger its IDF; a term in every document gets exactly 0 because there is no +1 smoothing in the IDF. Romeo: log10(37/1) ≈ 1.57; good: log10(37/37) = 0.

$idf (t) = lo g_{10} (\frac{N}{df ( t )})$

⚠️ Top pitfall: Adding +1 smoothing to the IDF like the TF formula: the IDF has no smoothing, so terms appearing in every document really do get exactly 0.

Self-check: Why does a term appearing in every document get an IDF of exactly zero?

Connects to: 2.10, 2.12

2.12 TF-IDF: Term Frequency Times Inverse Document Frequency

Must-know: tf(t,d) = log10(count(t,d) + 1); idf(t) = log10(N/df(t)); tf-idf(t,d) = tf(t,d) x idf(t). wit in As You Like It: tf = log10(21) = 1.322, idf = log10(37/34) = 0.037, tf-idf = 0.049. Zeros in the TF-IDF table come from the IDF. For battle, Henry V ranks first (count 13).

$tf-idf (t, d) = lo g_{10} (count (t, d) + 1) \times lo g_{10} (\frac{N}{df ( t )})$

⚠️ Top pitfall: Mixing up the raw TF count with the final TF-IDF value, and thinking the +1 smoothing applies to IDF; the TF-IDF zeros come from IDF(t) = 0, not from the TF.

Self-check: Why does good score a TF-IDF of zero in a document where it occurs 114 times?

Connects to: 2.11, 2.13

2.13 Problems with Sparse Vectors and the Road to Dense Embeddings

Must-know: Sparse vectors have dimension = vocabulary size (context-window) or document count (TF-IDF), up to about 1 billion at web scale, and are mostly zeros; the dense prediction-based family (skip-gram, CBOW, GloVe, contextual embeddings) is scheduled after the mid-semester boundary.

⚠️ Top pitfall: Thinking the zeros of sparse vectors can be removed or rebalanced; that manipulates the data, and the standard answer is a different representation.

Self-check: Why does a 1-billion-word vocabulary make every context-window vector a billion dimensions long?

Connects to: 2.6, 2.12

2.14 Implementation: Count Vectorizer and TF-IDF in sklearn

Must-know: CountVectorizer gives the raw count matrix (rows = documents, columns = unique words, sparse); TfidfVectorizer computes TF-IDF in one call from the corpus you pass; TF-IDF went from the only Google ranking signal to one parameter among many.

⚠️ Top pitfall: Thinking term frequency is a ratio of words to documents; it is simply the count of the term in the document.

Self-check: Where do the counts come from when you call CountVectorizer?

Connects to: 2.7, 2.12

Exam Guidance Summary

Must-know: The exam is open book; expect conceptual questions on the distributional hypothesis and sparse versus dense representations, plus worked computations (log-smoothed TF, IDF, TF-IDF product, cosine similarity) exactly as shown in this session.

⚠️ Top pitfall: Treating reference lists as exam material; they are for understanding, and the recorded sessions and core concepts are the priority.

Self-check: What is the expected format of the exam, and what computations must you rehearse?

Key Industry Applications

Must-know: Word embeddings are the universal input to modern NLP systems; TF-IDF evolved from the only Google ranking signal to one parameter among many; BM25 (a modified TF-IDF) powers RAG and agentic retrieval.

Self-check: Why do frequency-based embeddings still win for short structured queries like course IDs?

Previous LectureLecture 01 · Introduction to Natural Language Processing

Next LectureLecture 03 · Word Embeddings and Word2Vec

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.