CBOW, GloVe, and Statistical Language Modeling

4.1 Skip-gram with Negative Sampling: Recap of the Update Procedure

4.1.1 Why Static Word Embeddings Still Matter

This session completes the family of static word-embedding algorithms: skip-gram with negative sampling (SGNS), continuous bag of words (CBOW), and GloVe. The word "static" matters. A static embedding gives each word one fixed vector, unlike the contextual embeddings inside transformer models, which change a word's vector depending on its neighbors. Even though contextual embeddings now dominate modern NLP, the static algorithms remain useful and are widely used in production environments. The intuition behind contextual word embeddings actually grew out of these static methods, so understanding SGNS, CBOW, and GloVe is the foundation for the transformer material later in the course.

These algorithms are low-resource by design: they run comfortably on a local CPU, with no GPU cluster needed. Pre-trained word-embedding models can be downloaded and plugged into an application, much like downloading GPT or LLaMA checkpoints. The gensim open-source library provides ready implementations of all these word-embedding algorithms, in the same spirit that scikit-learn provides implementations of classical machine-learning algorithms and TensorFlow provides deep-learning primitives. You only pass the hyperparameters; the algorithms themselves are already implemented, and a demo codebase and a spreadsheet walkthrough of the SGNS update accompany this material.

Static embeddings, especially SGNS, are popular in production precisely because they are cheap to train and cheap to run. A search or recommendation system that needs to compare millions of words cannot afford a transformer for every lookup; a precomputed table of vectors does the same job for a fraction of the cost, and it still appears in real applications today.

4.1.2 The Update Procedure, Step by Step

The previous session walked through one full SGNS update cycle on a worked example, and this recap retraces every step. The setup starts with initial word embeddings for a target word and for its context words. In the worked example, the positive context word appears as "stark" in the recording — likely "start", the exact word name does not change the math — and the target word as "net". We retrace the same six steps below, and in 4.1.4 we run them with explicit numbers.

Why does this work? Because the sigmoid score measures how "real" the model thinks a pair is. When the score for the positive pair is too low, the error is large and the update pulls the two vectors together. When the score for a negative pair is too high, the error is large in the other direction and the update pushes the vectors apart. After billions of such pushes and pulls across the corpus, words that appear in the same contexts (like "coffee" and "tea") settle near each other, and words that never co-occur sit far apart.

4.1.3 The Mathematics Behind the Update

Let us write the procedure in symbols, one formula at a time. Every symbol is named where it first appears.

Two sanity checks on the sign convention before we compute anything. First, if the model is already perfect on the positive pair (${\hat{y}}_{+}=1$), then $e_{+}=0$ and nothing moves — correct, there is nothing to fix. Second, if the model wrongly scores a negative pair at ${\hat{y}}_k=0.5$, then $e_k=-0.5$ and the update subtracts $0.5\eta \cdot {\mathbf{v}}_k^{-}$ from the target vector — pushing it away from the negative word. Both checks match the intended push-pull behavior.

4.1.4 Fully Worked Example: One Update Cycle with Real Numbers

Here is a complete numerical cycle in two dimensions, with the learning rate 0.05 used in the lecture. The words are illustrative; the arithmetic is the exact pattern of the lecture's spreadsheet example.

4.1.5 Assumptions & Scope

4.1.6 Visual Intuition

Picture a flat map with one arrow per word. The target "net" sits at coordinates $(0.10,0.30)$. Draw the positive word "start" at $(0.20,0.90)$, high up and to the right; draw the negative word "table" at $(0.70,0.10)$, far to the right and low down. The gradient $[-0.278,0.335]$ points up-and-left — toward "start" and away from "table" — and the update nudges "net" exactly $5\%$ of the way along that arrow. After the step, "net" sits closer to "start" and farther from "table". Repeat this millions of times for every word, and the map settles into a shape where words that share contexts share neighborhoods. The takeaway: each training step is one small tug-of-war between friends pulling words together and strangers pushing them apart.

4.1.7 Pitfalls

4.1.8 Recap, Exam Note, and Real-World Connection

Where this matters in the real world: SGNS-style embeddings power production search, recommendation, and document-ranking systems that must run on modest hardware. A news app can embed every headline once with a static model and then rank by vector similarity in microseconds per query — a job that would cost far more with a transformer. This is why the low-resource property praised in 4.1.1 is not nostalgia; it is a budget decision made daily in industry.

4.1.9 Choosing the Context Window

How big should the context window be? A smaller window is easier and faster. But a small, domain-specific corpus may need a larger window to produce enough context and out-of-context words. For a larger training corpus, a window of plus/minus 2 to plus/minus 4 words is enough. This window choice determines which words count as context, which is why the training data itself plays such a big role in what these embeddings learn.

The window is a dial that trades two kinds of quality. A narrow window captures syntax (how words combine); a wide window captures topic (what words tend to appear in the same discussion). Systems that need synonym-style similarity (search, recommendations) often widen the window; systems that need grammatical patterns keep it narrow. There is no single right answer — which is why the window is a hyperparameter you choose, and why two teams with the same code and different windows end up with visibly different vector spaces.

4.2 Training Data Quality and Bias in Word Embeddings

4.2.1 Data Shapes Meaning

Because embeddings are built from context, the training data shapes everything. The quality of the training data, the ethics, and responsible AI concerns all matter at the word-embedding step itself, because these embeddings are the input layer to every fancy model downstream — GPT, transformers, large language models, small language models. If the embeddings come from noisy data, everything downstream inherits the noise. The meanings of words also change with the data, and meanings are domain-specific.

This pattern shows up in every model seen so far — SGNS, CBOW, GloVe, and later the attention mechanism — context plays the central role in generating word embeddings. Good context gives a good output.

4.2.2 Words Change Meaning: The "Broadcast" Story

Think of the word "broadcast". It started in the farming industry, where it meant scattering seeds over a wide area. Today we use it for news channels and social media. The same string of letters, two completely different contexts, and so two different neighborhoods in a modern vector space — the farming sense would sit near "sow" and "harvest", while the media sense sits near "channel" and "audience".

The language keeps growing: new words appear constantly, and the context keeps changing. Word embeddings adapt themselves to the context they are trained on. This is why embeddings are never trained once and frozen forever: an embedding model trained on 1990s newspapers does not know what "tweet" means as a verb, and one trained on 1950s texts places "virus" only in medicine, never in computing. Every embedding has a birthday — the date of its training corpus — and its blind spots match that date.

4.2.3 Bias Enters at the Embedding Layer

If the training articles are biased, the embeddings will carry that bias, which is not a good thing. There are practices and mathematical formulations from fairness in ML and Responsible AI coursework that try to remove these biases from the training data so you get better quality embeddings. Cultural biases, gender biases, and similar problems must be removed because of the classic rule: garbage in, garbage out. Since these embeddings sit at the very input of the pipeline, they play a critical role in everything that follows.

A concrete example: if a corpus of news articles mentions "nurse" mostly near female pronouns and "engineer" mostly near male pronouns, the trained vectors will encode that association as a geometric fact — "nurse" will sit closer to female-gendered words. A resume-screening system built on such embeddings can then downgrade candidates in ways nobody explicitly programmed. The fix has two halves: curate and balance the training data (the lecture's emphasis), and apply debiasing techniques from the fairness-in-ML toolbox, such as removing the gender direction from the vector space or projecting neutralized words away from it.

4.2.4 Pitfalls and How to Respond

4.2.5 Recap and Bridge

Real-world placement: the data-quality conversation is not a side note — it is the reason companies with production NLP systems maintain their own corpora and periodically retrain embeddings. Search engines, recommendation platforms, and LLM vendors all face the same trade-off the lecture describes: pre-trained embeddings are cheap and fast, but only as trustworthy as the corpus they were trained on, which is why responsible-AI review of training data happens at the embedding layer first.

4.3 Continuous Bag of Words (CBOW)

4.3.1 The CBOW Idea

The second static word-embedding algorithm is CBOW, which stands for continuous bag of words. The name says it: it takes into consideration a bag of words that are continuous — the nearby words in a window, treated as a bag.

CBOW is the mirror image of skip-gram. In skip-gram, we give a target word and try to predict the probability of the context words (and out-of-context words) around it. In CBOW, we give the context words for a given target word and try to predict the target word itself. This is a fill-in-the-blanks problem. The same idea shows up later in the attention mechanism, where certain words in a sentence are masked and the model predicts them using probabilities from the training corpus. Here the same idea applies: give the context word embeddings, and train the model so it can predict the target word correctly.

In the spoken example, the four context words "I am because I" were used to predict the target word "happy" — the full sentence was likely something like "I am happy because I …", with the blank sitting between "am" and "because". Before training, some predictions will be wrong. The model checks the probability of every possible target word given these context words, and the word with the maximum probability is predicted as the target.

The training data is self-supervised, exactly as in skip-gram: the corpus provides its own labels, because the word that actually appeared in the center is the correct answer. The one difference: here we predict the probability of the target word, not of the context words. You still need the target word's embedding to end up close to the embeddings of its context words. And the important point carries over: you are not interested in the final classification task — you are interested in the weight matrix that the training produces, because that weight matrix is the word embeddings. The "fake task" of filling in blanks is only the excuse; the learned weights are the treasure.

4.3.2 How CBOW Predicts

Given the embeddings of the context words (computed from the features), CBOW combines them and computes the probability of every possible target word in the input vocabulary. Each unique word in the vocabulary gets its own probability based on those context words. The word with the highest probability wins and is predicted as the target:

$$w^{\ast }=\arg \underset{w}{\max }P(w\mid \text{context words})$$

Read this as: $w^{\ast }$ (the predicted word) is the word $w$ that makes the conditional probability $P(w\mid \text{context words})$ — "probability of word $w$ given the context words" — as large as possible. Computing this maximum needs a probability for every word in the vocabulary, and that is exactly the softmax function, covered in the next section.

4.3.3 The Forward Pass Under the Hood

How does "combine the context embeddings" actually work? The bag-of-words recipe is deliberately simple, and tracing it once makes the whole model concrete.

The two-vector trick from skip-gram reappears here: each word has an input vector (used when the word is part of the context) and an output vector (used when the word is a candidate target). The input vectors are the ones typically kept as the final word embeddings.

4.3.4 CBOW versus Skip-gram

CBOW is faster and simpler, because it only predicts the target word probability — it does not have to score context and out-of-context words the way skip-gram does. The trade-offs go both ways.

When to pick which: if you have a large corpus and care about speed and frequent words, CBOW is the cheaper tool; if your corpus is small or your task cares about rare words, skip-gram earns its extra cost. In production, SGNS is the more popular choice, and skip-gram is more powerful and useful for many real-world applications.

4.3.5 Student Questions

One more important point: all of these word embeddings are built during training time, so they happen offline. None of this happens at inference time or runtime, so there is no latency cost from these heavy computations — the embeddings are pre-trained models. Training and inference have different requirements. A recent article about Google's TPU (competing with NVIDIA) described exactly this: Google separated the processing units for inference from the units for training, because training builds the offline static embeddings for the whole training corpus, while at test time you process real-world user input — you need a little more RAM during inference time and ROM during training time.

This training-versus-inference split is why pre-trained embeddings can be heavy to build but cheap to serve: pay the computation once, at training time, and then treat the vectors as a read-only table that any small service can query.

4.3.6 Assumptions, Scope, and Pitfalls

4.3.7 Recap, Exam Note, and Real-World Connection

Where CBOW lives in industry: the model (and its Word2Vec sibling) is the default "get embeddings fast" tool in search and retrieval pipelines. Because training and serving are split — heavy counting offline, cheap vector lookup online — CBOW embeddings end up in products that must answer in milliseconds on modest hardware, from document similarity in knowledge bases to feature vectors for recommendation models.

4.4 Softmax and Sigmoid: Purpose and Difference

4.4.1 The Exchange

A question about the difference between sigmoid and softmax led to a useful clarification, since softmax returns later with contextual word embeddings.

That last word — compete — is the heart of the second question, which followed immediately.

The division by the sum is not decoration: it is what makes the ten numbers a genuine distribution — each value is a share of the total, so the shares sum to 100%, and "the biggest share" becomes a meaningful way to pick one winner.

4.4.2 The Formulas, Side by Side

4.4.3 Worked Example with Numbers

4.4.4 Why CBOW Uses Softmax

CBOW uses softmax at the output side for predicting the target word. The number of output options equals the unique vocabulary size. The softmax turns the raw scores into a probability distribution over all possible target words, and the maximum is picked. For CBOW there is no exam math problem — the intuition is what matters: softmax gives you one probability per vocabulary word, all positive, all summing to one, so "the word with the highest probability" is a well-defined choice.

4.4.5 Assumptions and Pitfalls

4.4.6 Recap and Bridge

Real-world placement: this same softmax is the final layer of virtually every modern language model — GPT, Whisper, and friends all emit a softmax distribution over their vocabularies and sample or argmax from it to choose the next token. The lecture's "ten options adding to 100%" is, scaled to 50,000 tokens, the exact mechanism that generates every word you have ever seen an LLM produce.

4.5 Word Vector Arithmetic, Analogies, and t-SNE Visualization

4.5.1 Visualizing the Vector Space

A t-SNE plot is a standard way to look at the whole vector space. It is a nice interface, similar in spirit to matplotlib, showing the words with color coding so you can see which words are similar to each other. These plots are commonly used because they help immediately. GloVe ships with exactly such a visualization.

What does such a plot look like? Imagine a 2D scatter plot where each dot is a word. The horizontal and vertical axes have no simple names — t-SNE squeezes a 300-dimensional space down to two dimensions while trying to keep nearby neighbors together, so the axes mean "position in the squeezed map", not a real feature. You see clusters: months of the year in one corner, countries in another, verbs of motion bunched nearby. Color coding separates word classes so the clusters jump out at a glance. The landmark to notice is not any single dot but the gaps between clusters — words with related meanings fall into the same cloud, and unrelated words are separated by empty space. The takeaway: if similar words look close on the plot, the embeddings are healthy; a plot with no visible clusters signals a poorly trained or badly chosen space.

4.5.2 Math on Words

The beauty of vector word embeddings is that you can perform math operations on words just like you do on numbers. This vector translation of words lets you combine and subtract meanings. If you want the word embedding of queen, you can take the vector embedding of king, subtract the embedding of man, and add the embedding of woman:

$$\mathbf{v}(\text{king})-\mathbf{v}(\text{man})+\mathbf{v}(\text{woman})\approx \mathbf{v}(\text{queen})$$

The same trick infers relations: given the relation between grape and wine, you can infer the analogous relation between apple and tree, because the word embeddings lie close to each other. This is why vector representation of words is extremely useful in many real-world applications.

4.5.3 Why the Arithmetic Works

4.5.4 Real-World Use

Analogy arithmetic is a standard quality check for word-embedding models, and the t-SNE visualization is how these vector spaces are explored interactively. When a team publishes a new embedding model, its score on analogy test sets (thousands of "king : queen :: man : ?" style questions, answered by nearest-neighbor search) is one of the first numbers reported — it proves the space has learnable structure, not just memorized clusters. In production, the same arithmetic powers lightweight semantic tools: rewriting a search query by moving it along a known direction ("cheap" minus "expensive" applied to a product vector) without retraining anything.

4.6 GloVe: Global Vectors for Word Representation

4.6.1 Global Vectors

GloVe is not a complicated concept. Its full form is global vectors for word representation. Recall that we have seen two methods of word embedding: the frequency-based approach using TF-IDF (basically calculating the frequency of different things), and the prediction-based approach with denser word vector representations. GloVe combines both — it combines counting with prediction, which is probability.

Skip-gram and CBOW only look at the plus/minus 2 context window for each word; they never look at the entire corpus. GloVe counts the co-occurrence of the entire vocabulary across the whole corpus, then uses unsupervised learning. The algorithm was developed at Stanford, and it captures both global and local statistics from the corpus. To start, you build a co-occurrence matrix for all the words in the vocabulary.

4.6.2 The Co-occurrence Matrix

For a toy corpus — say three sentences, standing in for the millions and trillions of sentences of a real corpus — first pick out the unique words. Then build a matrix that captures the co-occurrence of every unique word with every other unique word, within a context window of plus/minus 2 words. In the real world these counts can reach $10^{20}$ or more; with three sentences they stay small. This is just counting, and machines are very good at number crunching, so no fancy calculation is needed.

This reuses the co-occurrence concept from the earlier discussion: in that example, pie and cherry appeared together 42 times within a window of plus/minus 4 words in the training corpus. GloVe uses the same counting concept. High co-occurrence means the words are semantically related to each other; low co-occurrence means they are not. That is how context is captured in GloVe.

4.6.3 Conditional Probabilities and Ratios

The next step is conditional probabilities. For a word $w$ and a conditioning word $c$:

$$P(w\mid c)=\frac{C(c,w)}{C(c)}$$

where $C(c,w)$ is the number of times $w$ and $c$ co-occur within the window, and $C(c)$ is the total count of $c$ in the training corpus. For example, the probability of "solid" given "ice" is the count of solid and ice occurring together — 180 in the worked example — divided by the number of times ice occurs in the training corpus, because ice can occur with words other than solid. The more this probability grows, the more related the two words are.

Sometimes you want to distinguish which of two words is more related to a third word: is "solid" more related to "ice" or to "steam"? Take the ratio:

$$\frac{P(\text{solid}\mid \text{ice})}{P(\text{solid}\mid \text{steam})}$$

The worked numbers: $1.9\times 10^{-4}$ divided by $2.2\times 10^{-5}$, which gives about 8.9. Since the ratio is much larger than 1, solid is more related to ice than to steam. The same table shows water: $P(\text{water}\mid \text{ice})/P(\text{water}\mid \text{steam})=1.36$, and fashion: 0.96. Both are close to 1, but water scores higher than fashion, so water is more related to ice than fashion is.

In this way GloVe captures both things: the global counts using all the words in the corpus, and the local context for each word. That is why this approach is said to capture both global and local statistics. Yet it uses only counting — there is no training as such, unlike CBOW and SGNS. There is no question of partial derivatives; nothing like that is done to compute the vectors. The counting part is the global statistics, computed across the entire training corpus. When you calculate probabilities and ratios, that part is local only.

4.6.4 Why Ratios Beat Raw Probabilities

4.6.5 Student Questions

Several questions came up around the counting window and the ratio table; the four distinct confusions are kept below.

The next confusion was about where the headline number came from.

The most discussed row of the table was gas, and it led to a correction worth remembering.

The remaining questions were about reading the near-1 rows and the table's scope.

One more structural question closed the topic: why only ice and steam on the left side?

A note on scope for that last answer: ice and steam are only a running example with a smaller number of words. You repeat the same counting and ratio procedure for all your words in the training corpus — every pair of words gets its own ratio table, which is exactly why the co-occurrence matrix has to be computed once, globally, before any vectors exist.

4.6.6 Recap, Exam Note, and Real-World Connection

In the real world, GloVe's pre-trained vectors (trained on billions of tokens by the Stanford team) were for years the default word embeddings shipped in search, chatbots, and classifiers — and they still turn up where a fast, dependency-free, static embedding is the right size of tool. The ratio insight itself outlived the model: modern contextual models still learn by contrasting what appears together against what does not, the same ice-versus-steam logic at transformer scale.

4.7 Language Modeling: What It Is and Where It Is Used

4.7.1 The Definition

Language modeling is a core concept of NLP, and it came up many, many years ago — the statistical approach is today's topic, and the next session introduces neural language modeling, large language models, small language models, and prompt engineering.

What is language modeling? A first guess is "making a model understand a language", but it is more than understanding. These are generative models: given an input prompt from the user, the machine generates new tokens — answering a conversational question, making a PPT, writing code. Generation tasks are the concept of generative models, and that is the concept of language modeling.

The basic idea behind all language models — nano, edge, mini, small, large, open source or paid — is predicting a set of words given a set of previous words. It all boils down to probability, even in transformers. Given a set of words, you predict a set of new words: a next word, a sentence, a group of sentences, an article, a document, a PPT, or code. The statistical approach means finding the probability of the next set of words or tokens given a set of words or tokens — the probability of a sequence of words.

So a language model has two jobs that share one engine: score a whole word sequence (how natural does "high winds tonight" sound?) and rank the possible next words (what usually follows "See you"?). Both jobs read from the same probabilities, and the rest of this session builds those probabilities by counting text.

4.7.2 Predict or Generate?

The terminology matters because the job is production, not guessing. Prediction suggests picking the one true next word; generation means creating new content one token at a time. Every modern chat assistant is exactly this loop repeated: score the next token, pick one, append it, score again. When the lecture says "generative", it points at that loop.

4.7.3 Applications

The applications number in the thousands. Most real-world applications are language-dependent, and wherever there is language there is an NLP application — that is why LLMs like ChatGPT became such a breakthrough.

• Autocomplete of queries: the phone-keyboard example at scale — search engines and editors rank the continuations of what you typed.
• Spell check and grammar check: compare the probability of one sequence of words against another, and suggest the one with the higher count in the training data. "About fifteen minuets" loses to "about fifteen minutes" because the second sequence is far more common.
• Machine translation: you need to correctly translate the words. A wrong word choice changes the meaning — in the Hindi example given, translating a phrase for a strong wind must not substitute the wrong adjective (the recording has "tej hawa" — standard Hindi for "strong wind" is tez hawa — versus "jada hawa", possibly zyada hawa; the point stands either way: the adjective carries the meaning).
• Speech recognition: statistical language modeling is widely used in audio analysis and automatic speech recognition systems like Whisper. If one transcription has more counts in the training corpus, it is the more likely correct speech-to-text. "I saw a van" beats the identically-sounding "eyes awe of an" because only one of them reads like English.
• Next-token prediction, used for example by Google's systems, and answer generation from a web corpus for bots.

Everywhere, everything is dependent on the training data — whether it is a large language model, an agentic AI system, or statistical language modeling, it boils down to data. How good is your data? From that data you generate or predict new text.

4.7.4 What a Language Model Is Not

4.7.5 Recap, Exam Note, and Real-World Connection

Real-world placement: the perplexity tool, used widely in research papers, is built around the standard evaluation measure for these models, covered in the next session. When a new model is announced, its perplexity on a standard test set is one of the first numbers reported — it is this section's sentence probability, turned into one headline number. Autocomplete, spell check, machine translation, speech recognition, and chat assistants all rest on the same probability engine; the statistical version built today is the small, cheap, fully understandable ancestor of every one of them.

4.8 Conditional Probability, Independence, and the Chain Rule

4.8.1 Independence versus Dependence

Language modeling is built on plain probability theory, starting with conditional probability. Conditional probability is used when the outcome depends on whatever is available to you: what is the probability of something given something?

If you want the probability of two events happening together, there are two cases. With an independence assumption, the two events do not depend on each other, and the joint probability of events $A$ and $B$ is simply the multiplication of both individual probabilities:

$$P(A,B)=P(A)\cdot P(B)$$

These individual probabilities are also called marginal probabilities. But when we condition, that means $A$ depends on $B$, so you cannot simply multiply the individual probabilities to get the joint. You condition on the other event occurring:

$$P(A,B)=P(A)\cdot P(B\mid A)$$

That is: the probability of $B$ given that $A$ has already occurred. Equivalently, rearranging the same equation gives the definition of the conditional probability itself:

$$P(B\mid A)=\frac{P(A,B)}{P(A)}$$

All three formulas say the same thing in different orders: the joint probability of two events is the probability of the first times the probability of the second given the first, and conditioning just re-weights the joint by the probability of what you already know.

4.8.2 The Chain Rule for Word Sequences

Now extend this to the language modeling problem: predicting a sequence of words. For a four-word phrase "chain rule in general", because the words of normal English depend on each other, the probability is:

$$P(\text{chain})\cdot P(\text{rule}\mid \text{chain})\cdot P(\text{in}\mid \text{chain},\text{rule})\cdot P(\text{general}\mid \text{chain},\text{rule},\text{in})$$

We capture the context here. In language, all the words of a sentence relate to each other — you don't randomly pick words and form a sentence; that would be a garbage sentence. English works from left to right, so we calculate the probabilities from left to right, and we get a chain of probabilities. In general:

$$P(w_1,w_2,\dots ,w_n)=P(w_1)\cdot P(w_2\mid w_1)\cdot P(w_3\mid w_1,w_2)\cdots P(w_n\mid w_1,\dots ,w_{n-1})$$

4.8.3 Worked Example: "its water is so transparent"

The lecture's running phrase (the recording garbles one word — "so" is heard as "soap", but the intended phrase is the one below) shows both how the chain rule works and why it is hard.

This is the challenge that motivates the Markov assumption: computing these factors is not easy from the training corpus — computationally it is a real burden, because you need the counts — and the counts you need most (long histories) are the ones most likely to be zero.

4.8.4 Student Question

4.8.5 Pitfalls and Recap

Real-world placement: the chain rule is the generation engine of modern LLMs. When a chatbot writes a sentence, it computes $P(\text{next token}\mid \text{everything so far})$, picks one token, appends it, and repeats — the product of all those steps is exactly the probability of the text it wrote. Today's models handle the long history with attention instead of exact counting, but the left-to-right chain of "given" factors is the same structure built in this section.

4.9 The Markov Assumption and n-gram Language Models

4.9.1 The Markov Assumption

Hidden Markov models came up in the deep learning course (a prerequisite call worth remembering). The idea: instead of remembering the complete set of previous words, remember a restricted set of $K$ previous words.

In the Markov assumption, the current word is not dependent on the complete set of previous words, but on some set of previous words. Instead of looking at the whole history, you look at the previous one word, previous two words, or previous three words. In symbols, the exact chain-rule factor,

$$P(w_i\mid w_1,w_2,\dots ,w_{i-1})$$

is replaced by a short window of the last $K$ words:

$$P(w_i\mid w_1,\dots ,w_{i-1})\approx P(w_i\mid w_{i-K},\dots ,w_{i-1})$$

The symbol $\approx$ means "is about equal to": it is an approximation, not the truth. Some real sentences do depend on far-back words, and the model accepts a small error in exchange for counts that actually exist. Instead of all $N-1$ previous words, you approximate with the previous $K$ words. This is a simplifying assumption that makes computation easier and better.

The value of $K$ depends on your training data. Usually $K$ is taken as two or three previous words, because those combinations are highly likely to occur in the training corpus. Combinations of five or six words are much less likely to be present, so we don't go that far.

4.9.2 Unigram, Bigram, Trigram

These are called n-gram language models: an n-gram is a sequence of tokens or words on which the current word depends. The number $n$ counts the words in the window including the word being predicted, so each model keeps $n-1$ previous words.

• Unigram: $n=1$. The current word does not depend on any previous word; its probability depends only on itself:

$$P(w_i)$$

In practice unigram is of hardly any use, because a real sentence has dependencies between words — it is mentioned for completeness.

• Bigram: $n=2$. The current word depends on one previous word, since two words appear in the probability function:

$$P(w_i\mid w_{i-1})$$

• Trigram: $n=3$. The current word depends on the two immediately previous words:

$$P(w_i\mid w_{i-2},w_{i-1})$$

The same pattern extends to 4-grams, 5-grams, and so on. Sometimes English sentences have longer dependencies, where the last word depends on the first or second word after five or six words. The n-gram model cannot capture those because the combination will not be in the training data — the attention mechanism, covered with neural language modeling, handles longer context.

4.9.3 Start and End Symbols

Extra start and end symbols are generally used in NLP applications to indicate the start and the end of the sentence. Some algorithms use tags like opening and closing <s> and </s>, others use a star symbol, and current transformer architectures use CLS for the start of a sentence and SEP to separate sentences. These tokens are automatically introduced when you do language modeling. The reason: for the first word of a sentence, the bigram needs a previous word, and without a start tag you would not know which word is first. A trigram needs two start symbols before the first word.

4.9.4 Student Questions

A second question was about the other boundary: when generation should end.

The most important question of the set was about how much context a trigram really uses.

Zero counts came up twice more, about denominators and about production.

A bridge question connected today's strings to the earlier vectors.

4.9.5 Scope, Pitfalls, and Recap

Real-world placement: classic n-gram models are pure Markov models — fixed short windows — and they remain the go-to for fast, cheap text scoring on limited hardware: keyboard suggestions, SMS autocorrect, and domain-specific command systems. Modern neural models broke the $K$-word limit with attention, which is why the lecture repeatedly points forward: the Markov assumption is the simple baseline that later models were built to beat.

4.10 Bigram Probabilities: MLE Estimation and the Toy Corpus

4.10.1 The MLE Formula

The maximum likelihood estimate (MLE) for a bigram comes straight from the training corpus. Count how many times $w_i$ comes after $w_{i-1}$, and divide by the count of $w_{i-1}$ (because that word can occur with other words too):

$$P(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)}{C(w_{i-1})}$$

One rule before any calculation: when you do bigram, trigram, or any n-gram calculations, the order of the words is critical. The probability of I given am is different from the probability of am given I — the pair "am I" is counted separately from the pair "I am".

4.10.2 The Three-Sentence Corpus

The toy corpus for practice has three sentences with start and end symbols:

1. <s> I am Sam </s>
2. <s> Sam I am </s>
3. <s> I do not like green eggs and ham </s>

(The third sentence is reconstructed from the spoken counts in the lecture; its words — do, not, like, green, eggs, and, ham — are exactly the ones that appear in the discussion, each occurring once.)

4.10.3 Student Questions

A closely related question was about the denominator in the other direction.

The last confusion was about what kind of probability this is.

4.10.4 Pitfalls, Exam Note, and Recap

Real-world placement: this hand count is exactly what a training pipeline does on billions of words — build a giant table of which word follows which, then normalize each row into probabilities. The table is the model. Everything downstream, from a speech recognizer choosing between transcriptions to a keyboard suggesting the next word, is a lookup in that table followed by multiplication.

4.11 Sentence Probability, Logs, and the Zero-Count Problem

4.11.1 A Bigger Corpus

Scale up to a larger toy corpus: 9,222 sentences from a restaurant-information system. The bigram table gives counts of the next word given the previous word — next word on the rows, previous word on the columns. There is some noise in real training data: grammatically incorrect sentences produce odd counts, like I coming after I five times. Want coming after I occurs 827 times. The preposition "to" coming after I occurs 0 times (generally a preposition does not come after I). Eat coming after I occurs 9 times. There are many 0 values in this table.

To turn raw counts into probabilities, use the unigram counts — the number of times each word occurs in these 9,222 sentences. The count of I is 2,533. The probability calculation uses the same formula as before: the joint occurrence divided by the unique occurrence. For example:

$$P(\text{want}\mid \text{I})=\frac{C(\text{I},\text{want})}{C(\text{I})}=\frac{827}{2533}\approx 0.33$$

Do this for every word. Some entries are 0 in the numerator, so their probability is 0. The method does not change with size — it is still count and divide — the numbers are just messier now.

4.11.2 Multiplying Bigram Probabilities

To score a whole phrase, multiply the individual bigram probabilities under the Markov assumption. For "I want to eat Chinese food", some of the needed bigrams are zero in the corpus, so the entire sentence probability becomes 0. For a phrase like "I want English food" the pieces exist, so the sentence probability is easy to compute:

4.11.3 Logs Avoid Underflow

One more practical issue before smoothing. Instead of multiplying the raw probabilities, which can get a diminishing value — probabilities like $10^{-20}$, or $10^{-100}$ for longer sequences — we squash the values by taking the log. This avoids underflow: we take the log of all the probabilities, and instead of multiplication we just add the log probabilities. Addition is always less expensive computationally than multiplication for the machine, which is another reason this is better. Everywhere, including transformers, log is taken to reduce the computations:

$$\log P(w_1,\dots ,w_n)=\log P(w_1)+\sum _{i=2}^n\log P(w_i\mid w_{i-1})$$

4.11.4 Ready-Made n-gram Corpora

You don't need to compute these counts yourself. A lot of free n-gram datasets are available — for example the Google web corpus, a large repository with counts of all possible combinations of words, ready-made for applications. You can see the counts of 4-grams occurring together, like every continuation of "serve as the" with its own tally. For domain-specific work (medical or finance), the open-source generic English datasets (Wikipedia and others) may not fit, but otherwise they are freely downloadable and easy to use. Even web-scale counts run out — most possible long phrases were still never written — so zeros never fully disappear; scale reduces the problem but does not remove it.

4.11.5 Student Questions

A follow-up asked about predicting two words at once.

4.11.6 Pitfalls and Recap

Real-world placement: log-space scoring is the everyday language of model training — the cross-entropy loss used by modern neural networks is an average of negative log-probabilities. The restaurant-corpus counting here and a GPT's training objective are the same family of computation, separated by sixty years and six orders of magnitude.

4.12 Laplace (Add-One) Smoothing

4.12.1 The Add-One Formula

The zero problem: I have the words "offer" and "loan" individually in my training data, but not in that sequence, so the bigram probability is 0, and any sentence containing it scores 0. To avoid the 0 probability, Laplace smoothing adjusts the MLE counts: add 1 in the numerator and add the vocabulary size $V$ in the denominator:

$$P(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V}$$

where $V$ is the number of unique words in the training vocabulary.

Why divide by $V$, instead of just adding 1 in the numerator? Two reasons. First, a probability should be between 0 and 1, and dividing keeps the smoothed values in that range. Second, balance: when you add 1 in the numerator you affect the counts, and to balance that and generalize it so the counts are not too large, we divide by the entire vocabulary size. The count becomes close to 0, but it is not exactly 0 — and dividing by the vocabulary avoids giving too much significance to the zero-count words. If you divided by 1, you would manipulate the data far more.

There is a deeper reason the denominator grows by exactly $V$: we added 1 to each of the $V$ possible next words, so the row total grows by $V\times 1=V$, and adding the same $V$ below keeps each row of probabilities summing to 1. Note the numerator update applies to every count, not just the zeros: 827 becomes 828, 5 becomes 6 — all the counts are updated by 1 in the numerator, and the vocabulary is added in the denominator.

4.12.2 Worked Numbers

For the running example (the restaurant corpus has vocabulary size $V=1446$ unique words — the number needed to reproduce the lecture's rounded values):

$$P(\text{want}\mid \text{I})=\frac{827+1}{2533+1446}=\frac{828}{3979}\approx 0.208\approx 0.21$$

4.12.3 The Blunt-Instrument Problem

There is a problem with Laplace smoothing. The counts have been manipulated, and the change is large. If you take the smoothed probability and try to recover the counts — multiplying the smoothed probability by the denominator count $C(w_{i-1})$ — the "counts" you get back are substantially changed. The original count was 827, and the adjusted count comes out as about 527:

$$0.208\times 2533\approx 527$$

Just by doing Laplace smoothing, you are manipulating the data quite a lot, and you don't want to do that. The lesson is to match the strength of the fix to the size of the problem: over-smoothing hurts, just like too much regularization in a neural network underfits.

4.12.4 Student Questions

4.12.5 Scope and Recap

Real-world placement: the same add-a-little-to-everything trick reappears across machine learning under other names — pseudo-counts in Bayesian models, label smoothing in neural networks, and the standard naive-Bayes text classifier's smoothing. Recognizing it in its n-gram form (and knowing when its distortion is acceptable) is the transferable skill here.

4.13 Interpolation and Backoff

4.13.1 Linear Interpolation

To avoid zero probabilities without the heavy manipulation of add-one smoothing, use interpolation or backoff — both are easy and similar to each other. In interpolation, we mix the probabilities of all three orders instead of focusing on only one. If the corpus has trigram counts, use those; if not, use the bigram counts; if not, use the unigram counts. The mixing weights are the lambdas:

$$\hat{P}(w_i\mid w_{i-2},w_{i-1})={\lambda }_{1}P(w_i\mid w_{i-2},w_{i-1})+{\lambda }_{2}P(w_i\mid w_{i-1})+{\lambda }_{3}P(w_i)$$

The lambdas give importance to the trigram, the bigram, and the unigram respectively: ${\lambda }_1$ multiplies the trigram term, ${\lambda }_2$ the bigram, ${\lambda }_3$ the unigram. They must add to one because this is like a probability:

$${\lambda }_1+{\lambda }_2+{\lambda }_3=1$$

4.13.2 Worked Example: A B C D

For a sentence A B C D computed with trigrams, the sentence probability is:

$$P(A\mid ⟨s⟩⟨s⟩)\cdot P(B\mid ⟨s⟩,A)\cdot P(C\mid A,B)\cdot P(D\mid B,C)\cdots$$

The first word needs two start symbols; the second word conditions on one start symbol and A; the third conditions on B and A; and so on. Normally you would calculate each trigram on its own. With interpolation, each of these individual probabilities also adds the bigram and unigram terms — the same three-formula combination for each one.

4.13.3 Stupid Backoff

Backoff is exactly similar, except that instead of taking all three, we take one at a time. If we have the trigram count, we go ahead with that. If we do not have the trigram count, we reduce the weightage by 0.4 — giving 100% weightage to the trigram when available, then 0.4 times weightage for the bigram. If nothing works, we further reduce to the unigram, and the unigram count is divided by $N$ (the total token count), so the unigram gets very little weightage:

$$S(w_i\mid w_{i-2},w_{i-1})=\{\begin{array}{ll}\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})}& \text{if }C(w_{i-2},w_{i-1},w_i)>0\\ 0.4\cdot S(w_i\mid w_{i-1})& \text{otherwise}\end{array}$$

with the final fallback $0.4\times 0.4\times \frac{C(w_i)}{N}$, where $N$ is the total number of tokens in the training corpus.

One honest label: $S$ is a score, not a true probability — it does not sum to 1, so it is good for ranking candidates, not for plugging into probability-based measures.

4.13.4 Comparison: Interpolation versus Backoff

When to pick which: use interpolation when you want a proper probability and can afford tuning the lambdas; use backoff when you only need to rank candidates and want a zero-parameter method — interpolation usually works better, and backoff is the simpler tool.

4.13.5 Student Questions

A follow-up asked whether the zero checks happen at runtime.

The last two questions were about training data and the weights.

4.13.6 Recap, Exam Note, and Real-World Connection

Real-world placement: stupid backoff earned its fame at Google scale — with a billion words of training data, a fast parameter-free ranking method beats a careful method that is too slow to use. The engineering lesson generalizes: a rough tool that scales can win over a precise tool that does not, which is why production systems at web scale still use variants of this "stupid" idea today.

Exam Guidance Summary

What to expect per topic.

• SGNS: expect a numerical similar to the worked spreadsheet example — sigmoid values, errors, gradient updates, and updated embeddings with learning rate 0.05.
• CBOW: no full numerical; a simple addition or a conceptual question, such as the difference between CBOW and skip-gram and when to use which.
• GloVe: no math problems on the final exam for the GloVe vector computation.
• Interpolation: a simple problem is possible; there are no problems on learning the lambda parameters.

Cross-cutting exam rules.

• Evaluation measures (perplexity and the comparison of models) will be covered in the next session, together with applications and the code; the perplexity tool used for research papers builds on this measure.
• When a bigram/trigram count table appears in an exam question, the question will specify which is the next word and which is the earlier word (row versus column).
• Word order in n-gram calculations is extremely important — an exam-ready point repeated many times.
• 4-grams and 5-grams are possible in production and depend on the training corpus, but bigram and trigram are the norm because higher orders are usually missing from the corpus.

Recall from earlier sessions.

• TF-IDF formula variants: some books put the plus one inside the log and some put it outside. Jurafsky's latest version has $1+{\log }_{10}(\text{count})$ with the plus one outside; Manning's has ${\log }_{10}(1+\text{count})$ inside. It doesn't matter much — it just avoids zero. Log base 10 is more used because at web scale the vocabulary is very large; in a toy corpus the calculations are tiny either way. If a formula appears in the exam, it will be specified in the question, or you can write down which formula you are using. The version that divides by the document length is the advanced BM25, not plain TF — plain TF does not need it.

Study advice.

• Revision is important — the material is new, and a recap will cover all the topics again; n-gram models are deliberately kept simpler so that neural language modeling is easier to absorb next session; previous-semester papers will be shared for practice.

Key Industry Applications

Static word embeddings in production.

• Static word embeddings (SGNS, CBOW, GloVe) are low-resource, run on local CPUs, and are used in many production environments; the intuition for contextual embeddings came from them.
• gensim implements all the static word-embedding algorithms, in the style of scikit-learn or TensorFlow; pre-trained embedding models can be downloaded like GPT or LLaMA checkpoints.
• GloVe ships with a t-SNE-based interactive visualization of the vector space, with color-coded clusters of similar words.
• Word-vector arithmetic (king − man + woman ≈ queen) and analogy inference (apple : tree :: grape : wine) are standard tools in embedding quality checks.

Statistical language modeling in products.

• Language modeling powers autocomplete, spell check, grammar check, machine translation, and speech recognition systems like Whisper; statistical language models are widely used in audio analysis.
• Ready-made web-scale n-gram corpora (for example from Google) provide counts and probabilities for generic English; domain-specific corpora (medical, finance) need their own data.
• n-gram models remain a good return-on-investment choice for domain-specific, small-vocabulary products on limited hardware — no high-end servers needed.

Ideas that carried into transformers.

• Transformers use start (CLS) and separator (SEP) tokens, and everywhere — including transformers — log probabilities are used to avoid underflow and to make computation cheaper.
• Google's TPU work separates inference hardware from training hardware, reflecting the offline training versus runtime inference split of these models — the same split that makes pre-trained embeddings cheap to serve.