Perplexity and Neural Language Models

5.1 Language Modeling Recap and the Evaluation Question

5.1.1 The Language Modeling Task

A language model (an LM) is a system that assigns a probability to a sequence of words. The probability is a score between 0 and 1 that says "how much does this string of words sound like the language we trained on." A high score means the sequence reads naturally; a low score means it looks odd. The core task, in one line: given the words seen so far, predict the probability of the next word. This one skill powers machine translation, spelling correction, and speech recognition — all of them pick the most natural-sounding option among candidates.

The previous session built these models the statistical way, with n-grams. Both the n-gram approach and the neural approach need one thing above all: a corpus (a large body of plain text used as training data). The key point is that you do not need labeled data. A language model learns from raw text alone — there are no labels to collect, because the text itself provides the answer: every word is a "next word" you can train on. If you are building a language model for English, any English text works — Wikipedia articles, newspaper articles, anything in textual format. The same training corpus requirement holds for neural language models, coming later in this session.

The n-gram recipe from last time: a bigram probability conditions the next word on the previous word, a trigram on the previous two, and a unigram on nothing at all. In symbols, the bigram probability is $P(w_n\mid w_{n-1})$ and the trigram is $P(w_n\mid w_{n-2},w_{n-1})$, each estimated by counting how often the pattern appears in the corpus and dividing. Unigram is the fallback when no higher-order counts exist.

5.1.2 Zero-Probability Fixes: Laplace Smoothing, Interpolation, Back-Off

Laplace smoothing (add-one smoothing) is the simplest fix: give every unseen event a small non-zero count. The smoothed bigram probability adds 1 to every pair count and adds the vocabulary size $V$ (the number of different words) to the denominator:

$$P^{\ast }(w_i\mid w_{i-1})=\frac{\text{count}(w_{i-1},w_i)+1}{\text{count}(w_{i-1})+V}$$

The catch: smoothing changes the data slightly, which can distort probabilities that are otherwise well estimated. Add-one is a blunt instrument — with a big vocabulary the "+$V$" in the denominator is large, and common pairs lose more probability than feels fair. That is why two other techniques matter.

Interpolation combines all the n-gram orders at once — trigram, bigram, and unigram — each with a weight. The verbal recipe: give more weight to the higher-order n-grams because they capture the context better. The weights ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$ (the Greek letter lambda) are hyperparameters set by the developer, with the single constraint that they sum to one. This matches the standard form in the reference material: ${\lambda }_1$ weights the trigram, ${\lambda }_2$ the bigram, and ${\lambda }_3$ the unigram.

$$\hat{P}(w_n\mid w_{n-2},w_{n-1})={\lambda }_{1}P(w_n\mid w_{n-2},w_{n-1})+{\lambda }_{2}P(w_n\mid w_{n-1})+{\lambda }_{3}P(w_n),{\lambda }_1+{\lambda }_2+{\lambda }_3=1$$

The lambdas can also be learned from held-out cross-validation data rather than hand-set; that was mentioned as possible in real-world use but not required here. Whatever their source, the constraint ${\lambda }_1+{\lambda }_2+{\lambda }_3=1$ is what keeps the blend a proper probability distribution.

Back-off is the alternative: when the count for a higher-order gram is missing, back off to the next lower gram. If the trigram is unseen, fall back to the bigram; if that fails too, the unigram. Think of it as a ladder of context: start at the top rung, and step down only when the current rung has no evidence. Both interpolation and back-off exist to kill the zero-probability problem by borrowing strength from lower-order statistics. The difference between them is when they borrow: back-off switches down only when needed, while interpolation always blends all orders.

Real-world & domain connection. Language models are the hidden judge inside everyday text systems. Spelling checkers use them to choose "minutes" over "minuets" in context, speech recognizers use them to pick "I saw a van" over the identically sounding "eyes awe of an", and machine translation uses them to rank fluent candidates. Modern LLMs are the same idea scaled to billions of parameters, which is why the evaluation question that comes next in this session is asked about them too.

5.2 Intrinsic Versus Extrinsic Evaluation

5.2.1 Two Ways to Evaluate

Any evaluation has two ingredients. First, test on a test dataset: a model may work well on training data, but it must also run well on unseen data. A model that only memorizes its training text is useless. Second, pick an evaluation measure — the number that tells you how well the model ran on that unseen data. The standard measures are likelihood, log likelihood, entropy — and the most popular, perplexity, which the next section builds up.

There are two levels of evaluation, and the contrast is the classic black-box/white-box contrast from software testing. Intrinsic evaluation measures the internal components of the model itself. Perplexity is intrinsic: it scores the language model directly, with no outside task involved. Extrinsic evaluation scores the model by plugging it into a real application — spell checking, speech recognition, or a machine translation (MT) system — and checking whether the end product is good. If the language model produces a good English-to-Hindi translation inside an MT system, that is extrinsic evidence: the whole system works, so the model inside it did its part.

The two levels line up neatly:

To compare two car engines, the true test is a real race (extrinsic), but it takes days to set up; a dynamometer reading in the garage (intrinsic) is fast and a decent hint. Same idea here: run the cheap intrinsic score to filter ideas, then pay for the expensive extrinsic test on the survivors.

Perplexity remains the popular measure even today, for LLMs and transformers everywhere, to check model efficiency or accuracy. When a new model is announced, its perplexity on a standard test set is one of the first numbers reported.

5.2.2 The Perplexity Name

Real-world & domain connection. There is an AI research tool named Perplexity, used for summarizing research papers. The name is likely borrowed from this evaluation measure — a fitting brand choice, since the tool's job is to digest text the way a language model would, and "perplexity" is the very number that measures how well a model understands text. The measure itself also powers the model selection workflow in industry: evaluation tools compute perplexity automatically at scale, and the model with the lower perplexity on your corpus is the one you ship.

5.3 Perplexity: Inverse Probability and the Nth Root

5.3.1 Predicting the Next Word: Why Unigram Fails

A language model predicts the next token given a set of words. Take the phrase "I always order pizza with cheese and —". The probability of the next word is computed given all previous words. Since keeping the entire history is impossible, the model makes a Markov assumption (the session calls it the hidden Markov model assumption): the current word depends on the previous one word, the previous two words, or no words at all.

Unigram is terrible for this task. If you predict the next word without looking at any previous words, "fried rice" is as likely as anything else, because no context constrains it. Unigram is only used when nothing better is available. Bigram looks at the previous word "and" and asks: how often does "mushrooms", "pepperoni", or "fried rice" follow it? Better, but "and" is followed by many words in the corpus, so the model is still unsure. Trigram looks at "cheese and" — now the next word is much more constrained. The idea: the more useful context you feed in, the better the probability you can predict, and the better the probability, the better the language model.

What does "good" mean concretely? A good model assigns high probability to the correct word and, just as important, is not confused. If all candidate words get equal probability, the model cannot decide which to pick. If the correct word gets 0.5, the next gets 0.2, and the third 0.1, the choice is easy. Higher probability for the correct word means a good model.

A second student asked about ties between candidate words, and a third connected the idea to what they already knew from machine learning:

Finally, a student asked about the possible values of the measure itself:

5.3.2 Why Inverse Probability: Length Normalization

Why not just use the sentence probability directly as the score? The sentence probability is a product (joint probability) of the individual word probabilities — "we compute the multiplication joint probabilities". And a prediction can be a single word, a group of words, a sentence, a paragraph, or even a whole document — ChatGPT sometimes returns a PPT or a CSV summary of a document. The same idea applies at every length.

Here is the problem. Suppose one model predicts "I like cats" and another predicts "I like cute cats very much". Both sentences are correct, but the longer sentence gets a lower probability just because it has more words. A longer sentence has more multiplication factors, each below 1, so its product shrinks. That does not mean the second model is worse. Raw probability cannot compare models fairly when the outputs differ in length.

The fix is a length-normalized score: convert the whole-sentence probability into an average per-word confusion measure, and that is perplexity. In words: take the probability of the sequence of words, raise it to the power minus one over N, where N is the length of the sentence — the number of words whose probabilities were multiplied. Since the exponent is $1/N$, it is the Nth root. This is the geometric mean across the N words of the inverse probabilities.

By the chain rule, the sentence probability is the product of per-word probabilities, each conditioned on the words before it:

$$P(w_1,w_2,\dots ,w_N)=\prod _{i=1}^{N}P(w_i\mid w_1,w_2,\dots ,w_{i-1})$$

Perplexity inverts that product and takes the Nth root:

$$PP(W)=P(w_1,w_2,\dots ,w_N{)}^{-1/N}={\left(\frac{1}{P(w_1,w_2,\dots ,w_N)}\right)}^{1/N}={\left(\prod _{i=1}^N\frac{1}{P(w_i\mid w_1,\dots ,w_{i-1})}\right)}^{1/N}$$

With a Markov assumption the conditioning context shrinks — for a bigram model each factor becomes $P(w_i\mid w_{i-1})$, and for a unigram model simply $P(w_i)$. The minus sign in the exponent is what turns "high probability" into "low perplexity"; the $1/N$ averages over the length, so long and short sentences become comparable.

Lowering perplexity is equivalent to maximizing probability. A good model gives high probability to correct words, so a good model has low perplexity, because perplexity is the inverse. Always aim to minimize it. And N is simply the number of words in the sentence.

5.3.3 Reading Perplexity Values

Perplexity makes models comparable even when the sentences differ in length — that is its main advantage.

Now the range question. Probability always lies between 0 and 1, but perplexity does not, because of the inverse and the root. Perplexity can take any whole-number value, depending on the sentence length, the probabilities, and the quality of the model. What does a perplexity of 10 mean? The model cannot decide among 10 different candidate words for the most likely next word. That is the intuitive reading. Ideally you want perplexity of 1 or 2 at most — the model is then deciding between very few words. A reasonable numeric range in practice is 10 to 400 at the maximum; higher than that is a bad model, and lower is always better.

One notation note: writing the exponent $1/N$ is just another way of writing the Nth root. For a number, an exponent of $1/2$ means the square root. So $x^{1/N}$ is the Nth root of $x$.

Visual intuition. Picture perplexity as a bar chart of "effective choices per word." On the horizontal axis sit models of increasing context (unigram, bigram, trigram); on the vertical axis sits perplexity. The bars fall sharply as context grows — on Wall Street Journal text, the classic figures are 962 for a unigram model, 170 for a bigram, and 109 for a trigram. The landmark to notice is that adding one word of context roughly divides the number by two to six; the takeaway is that context is what buys certainty, and the height of the bar is literally "how many equally likely words the model is torn between."

5.3.4 Worked Example 1: "I love NLP models"

Lower perplexity is better, and there is a computational angle too: if the model has to choose among only a few words at each step, the next-word search takes less time.

5.3.5 Worked Example 2: Random Digit Generator

The re-explanation that followed: suppose the model did not give $1/10$ to every digit. Suppose it gave the digit 9 a probability of $7/10$ and spread small values like 0.1 and 0.2 across the rest. Then we would know the model expects 9 next, and it would be a good model. Equal probabilities everywhere means the model has no opinion, and a perplexity of 10 flags exactly that.

5.3.6 More Context, Lower Perplexity

A natural question: how does perplexity relate to the n-gram order? A 15-gram model will have lower perplexity than a lower-order n-gram model. Higher gram, lower perplexity. The reason is context: the more previous words you attend to, the better the model predicts the next word. This idea of attending to more context is exactly how the attention mechanism came into existence — all these ideas were borrowed from the earlier models. (Attention is covered post mid-sem, and the DNN course touched on it.)

But there is a catch. A 10-gram needs counts for 10-word sequences, and those may simply not exist in your training data. That is why the Markov assumption exists in the first place: to make the model computable with the data you actually have. So while higher-order grams almost always — "99%" — give lower perplexity, data sparsity is the price.

5.3.7 Student Questions and Answers

Several students asked follow-up questions about the examples and the measure. The confusion points, deduplicated:

One student worried that the example probabilities were invented; another asked where they come from:

The next question targeted the role of perplexity itself — a misconception that is worth spelling out:

A student asked how to read the "5 equally likely words" phrase; the answer compared it with familiar classifier metrics:

Next came practical questions about using perplexity in the real world and whether ground truth text is needed:

The discussion then turned to the extreme values of the scale, and to the temperature parameter:

Two students asked about comparing commercial models, and about the probabilities being hidden inside them:

A student confirmed the measure's scope, and another pushed on whether higher-order grams always win:

The random digit example drew one more question — why a perplexity of 10 counts as a bad model:

The last question was about length fairness — the very property the Nth root was built for:

Real-world & domain connection. Perplexity is still the headline intrinsic score for modern language models: when a new model or a new LLM is announced, its perplexity on a standard test set is one of the first numbers reported. In production, model selection between commercial models (an OpenAI model versus a Gemini model, say) is done by computing perplexity on your own corpus and picking the lower value — and the evaluation is computed automatically by tools and APIs, never by hand.

5.4 Neural Network Foundations

5.4.1 Neurons, Brains, and the AI Winter

The neural network architecture is based on the neurons of the brain: simple units with interconnections. A memorable analogy: the brain has neuroplasticity, and research shows that age does not automatically reduce memory — what matters is how much stimulation and training you give your brain. The more you train it, the stronger it stays. The same logic explains why today's large language models and agentic AI systems behave intelligently: enormous amounts of training, bombarding them with knowledge, making them learn. Hold the biology lightly, though — a neuron here is just a small function of numbers; the biology is a hint, not the truth.

There is a historical context worth knowing: the field had an AI winter. In the 1990s nothing much was happening in AI — not because the ideas were wrong, but because of two missing ingredients: compute power and data. Now we have data and compute, so these neural algorithms can be implemented efficiently. The core idea itself is simple.

5.4.2 ML Versus DL: Who Finds the Features?

The difference between machine learning and deep learning: in ML you must do feature engineering — hand-craft the features and hardcode them. In neural networks, the model learns its own features from the data.

Two classic examples. To predict the cost of a flat (an Andrew Ng example), you give features: number of rooms, square feet area, location, facilities provided — and the model decides the cost. To classify whether a person plays tennis (classification), you give the weather: is it raining, is it dry, and so on — the model decides play or not. In both, you supply the features. In a neural network, you give an image and it learns all the features automatically; you give text, and through word embedding it learns features and finds context words on its own.

5.4.3 The Neuron as Sum of Products

The simple neural unit looks like linear regression. The input $X$ is not features in the hand-crafted sense — it is your training examples. In word embedding, the individual words of the sentence are the inputs; in language modeling, likewise. The weights are learned by the network, and there is a bias term. The pre-activation output is:

$$z=w_1{x}_1+w_2{x}_2+w_3{x}_3+b$$

Name each symbol: $x_1,x_2,x_3$ are the input values (the features of one example), $w_1,w_2,w_3$ are the weights that say how much each input matters, $b$ is the bias (one extra number added at the end — the unit's baseline leaning), and $z$ is the weighted sum.

This is called SOP — sum of products: products of weights and inputs added together, plus the bias. Note that $z$ is not the final output $y$; it is the intermediate value. The raw sum of products is linear: add more neurons and you still get a sum of products. To get probabilities between 0 and 1, apply the sigmoid at the output side, exactly like logistic regression; and the loss is cross-entropy, as in logistic regression. So the pattern everywhere is: weighted sum of products first, then a nonlinear activation function to produce the final output.

Visual intuition. Picture the unit as a small diagram: three arrows (inputs $x_1,x_2,x_3$) each pass through a weight ($w_1,w_2,w_3$) into a circle marked $\Sigma +b$, which sums them; the result $z$ flows into a curved squashing function, and the output $y$ comes out the right side. Numbers flow left to right: weigh, sum, squash. Without the bend of the activation, stacking units would just give one big straight line — the network would gain nothing from depth.

5.4.4 Sigmoid and the Other Activation Functions

The sigmoid is the most popular nonlinear activation, written with the symbol $\sigma$:

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

Here $e\approx 2.718$ is Euler's number, the base of natural logarithms. Its most interesting property concerns training. When you take the partial derivative with respect to the weights, the derivative of the sigmoid is sigmoid times one minus sigmoid:

$${\sigma }^{\prime }(z)=\sigma (z)\left(1-\sigma (z)\right)$$

That elegant derivative matters because gradient descent computes the derivative of the loss with respect to each and every weight ($w_1$, $w_2$, and so on). A clean derivative makes that work pleasant. A second nice property: the sigmoid's outputs are always between 0 and 1. Sigmoid is used even up to the transformers — transformer blocks contain feedforward networks, and in many hidden-layer places sigmoid is still used.

Other activations exist, and today's LLMs use several. Tanh is used, though not as popular. Instead of the range 0 to 1, tanh maps to values between -1 and +1:

$$\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$$

The most common modern alternative is ReLU and its variant leaky ReLU. For ReLU: if the input is negative, the output is given as zero; for positive values, the output is z itself (the maximum of 0 and z). Leaky ReLU differs on the negative side: instead of exactly zero, it gives a small (lower) slope for negative inputs, so the left side is not completely flat:

$$\mathrm{ReLU}(z)=\max (0,z)$$

$$\mathrm{LeakyReLU}(z)=\{\begin{array}{ll}z& z>0\\ \alpha z& z\le 0\end{array}\alpha \text{ small, e.g. }0.01$$

GELU (Gaussian Error Linear Unit) is another function used in today's LLMs, similar in spirit to ReLU with a smooth variation. Do not memorize the formulas for tanh, leaky ReLU, and GELU — what you should know is softmax, covered next.

In a full architecture: hidden layers typically use sigmoid, and the output layer uses softmax when the problem is multi-class, like predicting the next word. If the output is binary, the output neuron can use sigmoid. Softmax and sigmoid are the most common pair in transformers even today: softmax at the output side, sigmoid in the hidden nodes.

5.4.5 Softmax for Multi-Class Output

Softmax is the workhorse for picking one word out of a large vocabulary:

$$P(y=i)=\frac{e^{z_i}}{{\sum }_{j=1}^V{e}^{z_j}}$$

The numerator exponentiates the raw score $z_i$; the denominator is the sum of exponentiated scores over all classes, so the outputs form a probability distribution that sums to 1. Each $z_i$ is computed as a sum of products for its class. With the normalized values you can tell which class has the highest probability — that one is predicted. In language modeling, the highest-probability word becomes the next token. Here $V$ is the number of classes (the vocabulary size in language modeling).

Why not just use the raw $e^{z_1},e^{z_2},\dots$ without dividing? Because unnormalized values do not let you compare across classes properly; normalizing by the total turns the vector of numbers into a probability distribution and makes the highest one pickable.

Two students probed the normalization idea from opposite directions:

A final question asked what softmax actually changes about the raw scores:

5.4.6 Why Multiple Perceptrons: XOR

Why more than one perceptron? Because some data is not linearly separable. The XOR problem: with two inputs $X_1$ and $X_2$, no single line can separate the outputs. No matter how much you manipulate the weights, a single linear perceptron cannot solve XOR.

The analogy is the SVM kernel trick: kernels transform the data to a high-dimensional space where the two classes become separable. Adding perceptrons does something similar inside the network — more perceptrons give the network the capacity to separate non-linear data.

Concretely: send $X_1$ and $X_2$ to two hidden nodes $H_1$ and $H_2$, not just one. The function becomes more complex, and the network must learn more weights — before, only the weights from $X_1\to H_1$ and $X_2\to H_1$; now also $X_1\to H_2$, $X_2\to H_2$, and $H_2\to Y_1$. Each connection adds a weight, and bias is included at every step. With the extra neuron, the XOR problem becomes solvable.

Many real-world problems cannot be solved by a single perceptron — that is why we build a network of them. Mimicking the human brain: more perceptrons, more complexity, but real problems become solvable. This is why it is called a deep neural network, not a deep neuron.

5.4.7 Architecture Guidelines

A neural network has an input layer, one or more hidden layers, and an output layer. In classic ML algorithms, input connects straight to output; the hidden layers are what neural networks add. Terminology: a perceptron or neuron — the word neuron is typically used when an activation function is added, which is why the field is called neural networks. Hidden neurons live in the hidden layer, so "multi-layer" when there are several.

How many hidden layers, and how many neurons each? Andrew Ng's feedforward-network guidelines, recalled from the DNN course:

• The output layer size is fixed by the problem: one neuron for regression or binary classification, and for multi-class, one neuron per class with softmax. In language modeling, multi-class, the number of output neurons equals the number of words/tokens you want to predict.
• The input layer size equals the number of input tokens/features.
• Hidden layer size: take the number of input neurons and use one or two more — if you have 128 inputs, each hidden layer should have at least 129 or 130 neurons.
• Powers of two (64, 128, 256, ...) are a common and convenient choice for hidden sizes.
• Start small: begin with one hidden layer. If it already gives very good accuracy — 90% or 99% — stop there. More hidden layers mean more complexity, less explainability, and more resources consumed. The universal approximation theorem says even one hidden layer can approximate many functions, but multiple layers are advisable for harder problems.
• The "one or two more than the inputs" rule is a guideline for calculation's sake, not a hard restriction.

5.4.8 Feedforward and Backpropagation

The notation: superscripts indicate the layer number, subscripts indicate the node within a layer. Take a tiny network: 3 input nodes, 3 hidden nodes, 1 output node. For the first node of the second layer (the hidden layer), compute the sum of products over $x_0$ (the bias), $x_1$, $x_2$, each with its weight, then apply the activation $g$ — which can be leaky ReLU, sigmoid, tanh, or any activation function:

$$a_1^{(2)}=g\left(w_{10}^{(1)}{x}_0+w_{11}^{(1)}{x}_1+w_{12}^{(1)}{x}_2\right)$$

Here $a_1^{(2)}$ is the activation of node 1 in layer 2; the weight $w_{10}^{(1)}$ connects the bias input $x_0$ to that node, and $w_{11}^{(1)}$, $w_{12}^{(1)}$ connect the two real inputs. The bias is written as a fake input $x_0$ with a weight attached — nothing changes mathematically, it is just tidier to write. Repeat for all three hidden nodes. Then the output node computes its own sum of products over the hidden outputs with the next layer's weights, applies the activation (sigmoid here), and produces $a_1^{(3)}$. That is the whole forward pass: multiply, sum, activate, move forward. With more hidden layers, the same logic repeats; everything stays densely connected.

A notation note: Andrew Ng uses the symbol $\theta$ (theta) for weights instead of $w$ — the same thing, just a different letter.

In a fully connected network, every node of one layer connects to every node of the next, so the weights form a matrix rather than a vector. Every node in every layer is connected to every node in every next layer — that is why these are called dense networks.

A neat implementation fact: the entire neural network architecture boils down to two operations — multiplication and addition, over and over. Real-world: that repetitive matrix computation is why Google built the TPU (Tensor Processing Unit) as a competitor to GPUs. GPUs serve generic applications; TPUs are specialized, and there are TPUs for training time and TPUs for inference time, because the two phases have different requirements (training needs backpropagation, partial derivatives, and matrix computation).

Backpropagation is where the "real value" lives. It starts with the loss at the final output — say you predict $a^{(4)}$ and compare it with the ground truth; the error, or delta, comes from all the earlier layers and weights. The final loss could be caused by any neuron or weight in any earlier layer, which is why you propagate it backward. Gradient descent then optimizes each weight by taking the partial derivative of the loss with respect to that weight:

$$w_{\text{new}}=w_{\text{old}}-\eta \frac{\partial L}{\partial w}$$

where $\eta$ (the Greek letter eta) is the learning rate — how big each update step is. For binary classification the loss is cross-entropy; taking partial derivatives with respect to each weight gives a chain of derivative terms, which is why the elegant sigmoid derivative $\sigma (z)(1-\sigma (z))$ is so convenient at the hidden layers. Backpropagation through time is the recurrent version, mentioned only in passing here. The session noted that deep learning loss surfaces are non-convex, but did not go further — all the gory partial-derivative mathematics belongs to the DNN course.

Real-world & domain connection. The "multiply and add" insight explains modern hardware: GPUs (and Google's specialized TPUs) are built to accelerate exactly these matrix operations, with separate TPUs for training and inference because the two phases have different compute profiles. Every deep learning framework — and every LLM underneath — runs this same forward-pass/backward-pass loop, scaled up.

5.5 Neural Language Models

5.5.1 The Recipe

A neural language model is the combination of three ideas already covered: (1) the neural network architecture, (2) the concept of computing next-word probabilities from language modeling, and (3) word embeddings. Three together equal neural language modeling. The "Attention Is All You Need" paper — the transformer paper, motivated by machine translation — is fundamentally about neural language modeling.

One honest note: simple feedforward language models work wonderfully for many problems. You do not need a transformer for every task.

And a unifying claim: all of generative AI — LLMs included — boils down to predicting the probability of a set of tokens. LLMs are just larger neural language models: more data, more parameters, bigger architectures.

5.5.2 From Words to Embeddings: A Sentiment Analysis Walkthrough

Take a real-world classification problem: sentiment analysis of a movie review — good, bad, okay, or neutral; for simplicity, binary: the movie is good or not good. Two routes exist. With logistic regression you hand over features: who is the actor, who is the producer, the storyline, and so on. With deep learning you give the review text itself; the model learns the features automatically, using embeddings.

Word embeddings are the bridge: each word becomes a vector of numbers, learned via skip-gram SGNS, TF-IDF, frequency-based counts, and (post mid-sem) contextual embeddings. These vectors are the input to the model instead of hand-made features.

The pipeline for the feedforward version: each word of the review is converted to an embedding vector. The input layer has as many slots as the longest review has words; the hidden layer has one or two more nodes; and because this is binary sentiment classification, the output is a single neuron with sigmoid. The weights are learned during training on reviews labeled positive or negative — that is the backpropagation step. Once the weights are learned, a new review (say, a hotel review: "dessert is good") passes through the network, and the output probability decides: higher probability means positive sentiment.

5.5.3 Sentence Embeddings: Mean and Max Pooling

How do you combine individual word embeddings into a sentence embedding (also called a review embedding or phrase embedding)? Two simple techniques. Worked example with toy two-dimensional embeddings, for the words of "dessert is good" plus "the":

• "the": $[1,0]$
• "dessert": $[0.5,1]$
• "is": $[1.1,0.1]$
• "good": $[0.1,0.9]$

Mean (average) pooling: element-wise average across all words. In general, for word vectors $v_1,\dots ,v_N$:

$$s=\frac{1}{N}\sum _{i=1}^N{v}_i$$

Max pooling: element-wise maximum, where each component $s_j$ of the sentence embedding takes the largest value of that component across all words:

$$s_j=\underset{i}{\max }{v}_{i,j}$$

Both are used; max pooling is more popular, but mean works fine. These toy vectors are two-dimensional only for the demonstration — real embeddings have 768 or 1024 dimensions.

Real-world: Hugging Face already provides a sentence-transformers API that computes sentence embeddings automatically, so you rarely implement pooling yourself.

Students asked three clarifying questions about embeddings and pooling:

5.5.4 Fixed-Length Inputs and Padding

Neural networks need fixed-size inputs, but sentences vary in length. The simple solution: compute the maximum sentence length in your training data and use that as the input length. Shorter sentences get padded with zeros for the remaining slots. (LSTM pipelines use the same padding idea.) On average, the longest English sentences are about 10 to 12 words, so the fixed length stays modest; and if you want to restrict it further, you can split a long sentence into two parts and feed it as two separate inputs.

5.5.5 The Feedforward Language Model

Language modeling as a neural problem: given a set of words as input, predict the most likely next word. The input is a window of words — three words, a window of five, or the whole sentence, depending on the algorithm — converted to word embeddings and fed to a feedforward network. Because the output must pick one word out of the whole vocabulary, the output layer uses softmax, with one output neuron per vocabulary word. The softmax probabilities over the entire vocabulary sum to 1, and one word has the highest value — say the word shown as "9" in the slide's example — so that word is predicted as the next token.

Then the process slides forward: feed the next three words ("all the fish ...") and predict again, repeating until the end of the sentence. This loop is exactly how text generation works at scale: ChatGPT appears to type, but behind the scenes it takes the output of the current step and feeds it back as input for the next step, predicting one token after another, generating a whole sequence.

5.5.6 Weight Matrices W and U

In the feedforward language model, the weights come in two matrices. W is the matrix of weights from the input layer to the hidden layer; U is the matrix of weights from the hidden layer to the output layer.

Students examined the slide's numbers and notation one by one. First, a large number on the slide:

Next, how the matrices are sized, and what the vocabulary count really is:

A student then connected the design back to the n-gram comparison, and another asked about the missing bias:

Finally, a student asked whether a full numeric walkthrough of the network would come:

Real-world & domain connection. This is the seed of the transformer: swap the fixed window for attention, scale the data and parameters, and the same forward pass becomes GPT, Claude, and every modern LLM. Even ChatGPT's conversational output is one token-prediction loop repeated — the output of one step becomes the input of the next.

5.6 N-Gram Versus Neural Language Models

5.6.1 Semantic Generalization: The "Dog Gets Fed" Example

The n-gram model predicts the next word given prior words under the Markov assumption. The feedforward model does the same from a window of word embeddings. The deciding difference: embeddings capture semantics, not just syntax. An n-gram model sees only word identity; it cannot compare words for similarity. To an n-gram model, "cat" and "dog" are as unrelated as "cat" and "Tuesday" — both are just different symbols with no connection.

Beyond that: n-gram models cannot handle very long histories of context, while neural models can; neural models generalize better and are more accurate. The side-by-side comparison:

When to pick which: for a small, fast, interpretable deployment, the n-gram model can still win; for production-grade quality at scale, the neural model is the choice.

5.6.2 Cost, Data, and Interpretability

The neural side pays for its power. Complexity and time: you need GPUs. Real-world price check: an NVIDIA H100 was recently priced around 40 lakh rupees, and an A100 around 10 to 12 lakh rupees. Running LLMs locally needs such hardware; otherwise you call an API and pay as you go — something like 15 USD per month for a subscription.

Environmental cost: LLMs consume a lot of energy and create a large carbon footprint — comparable, in the session's analogy, to a flight. (Flights create a large carbon footprint, but we still need to travel; the same tension applies to LLMs.) The session's speaker mentioned preferring search engines because they cause less carbon footprint than models like ChatGPT or Claude, while admitting to using the models sometimes.

Interpretability: n-gram models are the most interpretable — you can read the counts. LLMs are almost zero-interpretable: the weights do not tell you why the model returned a particular sentence.

Training data: n-gram models need comparatively much smaller data; LLMs need enormous amounts. Context: LLMs handle long contexts, with commercial providers advertising large context-window token limits — one figure mentioned was 500 million tokens. That number is almost certainly mis-stated in the lecture audio: typical commercial context windows are far smaller (tens of thousands to a couple of million tokens). Either way, you pay per token for that context. Quality is better for LLMs. Deployment is easier for n-gram models and expensive and heavy for LLMs.

The summary split: n-gram models are good for small, fast, interpretable deployments; neural/LLMs are for production-ready, large-scale implementations. Which model to use when — and the pros and cons in production — is a next-semester topic in NLP applications courses.

Real-world & domain connection. In industry, the trade-off is a daily decision. Lightweight n-gram-style scoring still appears in small on-device tools and fast fallbacks, while production text systems — search, translation, assistants — run LLMs on GPU clusters (or via API) where quality justifies the cost. The hardware economics shown here, from H100 and A100 price points to pay-as-you-go API subscriptions, are what teams weigh when choosing between local deployment and calling an API.

Exam Guidance Summary

This session's exam-relevant material, gathered in one place:

• Perplexity is the evaluation measure for language models; expect to work the perplexity formula numerically. Know what capital N is (number of words in the sentence), how the Nth root normalizes length, and how to read the result ("5 equally likely words" style interpretations).
• Know softmax: it is used at the output of every multi-class network, including language modeling. Its formula and the fact that its outputs sum to 1 are exam-relevant.
• No backpropagation math problems will be asked in this NLP course — that material belongs to the DNN course. The same holds for the loss computation and weight learning details.
• No numerical example on neural language models is in the exam. The detailed numerical walkthrough of embeddings (with partial derivatives at every step, like the skip-gram negative sampling example) comes post mid-sem with contextual word embeddings.
• You do not need to memorize the tanh, leaky ReLU, and GELU formulas — they were presented as extra information. Sigmoid's properties (range 0 to 1, derivative sigmoid times one minus sigmoid) and softmax are the ones to know.
• Know the practical architecture guidelines: hidden layer neurons one or two more than the inputs, powers of two as convenient sizes, start with one hidden layer, output neurons equal to the vocabulary size for language modeling, single output neuron with sigmoid for binary classification.
• Know the comparison axes between n-gram and neural language models: semantic generalization (the cat/dog "gets fed" example), context length, data size, interpretability, deployment cost, and accuracy.
• There is an NLP quiz — check the calendar along with assignments, quizzes, and webinars.

Key Industry Applications

How this session's ideas show up in real products:

• Perplexity is the standard evaluation measure in industry for LLMs and transformers, from n-grams to GPT-class models. It is computed automatically by evaluation tools and APIs (in the spirit of sklearn's API calls) — never by hand at scale.
• Model selection workflow: to choose between two commercial models (e.g., an OpenAI model versus a Gemini model) for your own task, compute perplexity on your corpus and pick the lower value.
• Temperature parameter in LLMs: controls output variety; explains why production systems do not target perplexity of 1, which would make every word deterministic.
• The Perplexity research tool (AI-powered research-paper summarization) likely borrowed its name from this measure.
• Hugging Face provides a sentence-transformers API that computes sentence embeddings automatically; mean pooling and max pooling are the underlying techniques.
• Google Ngram is the most popular text corpus for n-gram language modeling; many other corpora are used as well.
• "Attention Is All You Need" (the transformer paper, motivated by machine translation) is built on neural language modeling.
• NVIDIA H100 (about 40 lakh INR) and A100 (about 10–12 lakh INR) GPUs are the hardware of local LLM deployment; API access is the pay-as-you-go alternative at roughly 15 USD per month.