Deep Neural Networks

Recurrent Neural Networks — Foundations and Architecture

Published: 2026-07-15

Level: postgraduate

Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Convolutional Neural Networks — covered in Lectures 9 & 10
Activation Functions (ReLU, tanh, sigmoid) — covered in Lectures 6 & 7
Dropout Regularization — covered in Lecture 8
Transfer Learning — covered in Lecture 8
IID Data and Sequential Data — covered in Lecture 2
Vanishing and Exploding Gradients — covered in Lectures 6 & 7

Recurrent Neural Networks — Foundations and Architecture

12.1 CNN Architecture Review and Transfer Learning

12.1.1 CNN Architecture Review — MNIST Digit Classification

Hook. You have 70,000 handwritten digits. Your job: build a machine that looks at a picture of a squiggly "3" and says "that is a 3". Its accuracy must exceed a human's. A feedforward network would need millions of parameters just to handle the raw pixels. How does a CNN do it with far fewer?

Intuition. Think of a CNN like a detective examining a crime scene photo. The detective does not stare at every pixel at once. They scan for small local clues — a fingerprint here, a footprint there. Then they combine these clues into bigger patterns: "a scuffle happened in this corner." A CNN works the same way. Its early layers spot tiny patterns like edges and corners. Its deeper layers combine those into shapes. Its final layers decide: "this is a '3'."

The MNIST dataset contains handwritten grayscale images of digits 0 through 9. The task is to classify each image into one of ten classes. Each image is a $28 \times 28$ pixel grayscale picture. So the input dimension is $28 \times 28 \times 1$ (height, width, channels). Since there is only one channel, the last dimension is 1, corresponding to grayscale.

Two competing architectures were discussed.

Architecture 1 — Flatten then Dense. After convolutions and pooling, flatten the feature maps. Feed them into a fully connected feedforward neural network with several hidden layers. The hidden layers extract more complex patterns from the feature maps. Then classify.

Architecture 2 — Global Average Pooling (GAP). After convolutions and pooling, do not flatten. Instead, apply global average pooling. This consolidates the information per local region. The result feeds directly into one dense layer that maps to the output. The only weights beyond the convolution layers lie between the GAP layer and the single dense layer. This architecture assumes the CNN has already extracted enough hierarchical features. No more deep fully-connected layers are needed.

The Python code shown in class uses Architecture 2.

Network structure (Architecture 2 — GAP):

- Input: $28 \times 28 \times 1$ grayscale images

- First convolution + pooling block:

- 32 kernels, each of size $3 \times 3$

- ReLU activation

- Max pooling with a $2 \times 2$ window

- Second convolution + pooling block:

- 64 kernels, each of size $3 \times 3$

- ReLU activation

- Max pooling with a $2 \times 2$ window

- Global Average Pooling (GAP)

- Dense layer with 10 neurons (output layer)

- Softmax activation for multi-class classification

- Loss function: categorical cross-entropy

The kernel size stays the same ( $3 \times 3$ ) in both blocks. What changes is the number of kernels. They expand from 32 to 64, increasing the feature extraction capacity.

Pooling layers learn zero parameters. There is no weight multiplication in pooling — it is a simple aggregation (max or average). So the parameter count in pooling layers is 0. Convolution layers do have learnable weights (the kernel values plus biases).

Parameter counting. A convolution layer with $K$ kernels of size $F \times F$ applied to an input with $C$ channels has:

$ Params = K \times (F \times F \times C) + K $

The extra $K$ accounts for one bias per kernel. Pooling layers contribute 0 parameters. A dense layer with $N$ inputs and $M$ outputs contributes:

$ Params = N \times M + M $

The extra $M$ is the bias terms.

Worked example — dense layer parameter count. Consider a dense layer with 128 neurons feeding into an output layer with 10 neurons:

$ Params = 128 \times 10 + 10 = 1290 $

The 128 × 10 = 1280 are the weight connections. The 10 are the bias terms, one per output neuron. The same logic applies to all layer pairs.

Normalization: Pixel values range from 0 to 255. Scale them to $[0, 1]$ by dividing by 255. The relative patterns and variance between pixel values are preserved even after rescaling. This normalization helps the training process.

One-hot encoding: The labels (0–9) are converted to one-hot vectors. For digit 9, the one-hot vector has size 10 with zeros everywhere except a 1 at the last position.

Optimizer: SGD, Adam, or RMSprop (user's choice)
Loss: categorical cross-entropy (multi-class)
Metric: accuracy

Training: Run for a fixed number of epochs (e.g., 10) with a batch size of 128. A validation split (e.g., 0.1) monitors progress. The validation set helps with hyperparameter tuning during training but is not the final test set.

Confusion matrix — how to read it. The diagonal entries are correct classifications. Off-diagonal entries are misclassifications. For example, if 3 images of digit 3 are classified as digit 5, the value at row 3, column 5 is 3. The "support" in classification reports is the number of true instances per class. For MNIST, digit 0 has about 980 images.

Visualizing learned filters: After training, the 32 learned $3 \times 3$ kernels can be visualized. Unlike hand-crafted filters such as Gabor filters, CNN-learned filters are not easily interpretable. They do not form clean horizontal or vertical lines. They capture subtle patterns instead. With more training epochs and data, some kernels may develop recognizable patterns — arcs, diagonal lines, or horizontal edges. In early training the patterns may be noisy.

Pitfalls — common traps when building CNNs:

1. Forgetting to normalize. Feeding raw pixel values (0–255) into a network without scaling to $[0, 1]$ can cause unstable gradients. Always normalize.

2. Confusing pooling parameters with learnable parameters. Pooling has zero weights. Do not count it in your parameter budget.

3. Using too many dense layers after convolutions. If the convolutional base already extracts good features, more dense layers may be unnecessary. A single dense output layer (Architecture 2) works better than stacking multiple ones. Extra dense layers add parameters and risk overfitting.

4. Misreading kernel size vs. number of kernels. Kernel size (e.g., $3 \times 3$ ) controls the receptive field. Number of kernels (e.g., 32) controls how many different feature maps you extract. Do not mix them up.

A CNN extracts hierarchical features: early layers find edges, later layers find shapes, and the output layer decides the class. Architecture 2 (GAP) preserves spatial information better than flattening by feeding each pooled region as a single summary value to the classifier. The next subsection explains exactly why.

Real-World & Domain Connection. CNNs power image classification in production systems everywhere. Examples include Facebook's photo tagging, Google Photos search, medical imaging for tumor detection, and quality inspection on factory assembly lines. The MNIST architecture, though simple, is the blueprint that scales to these production systems.

12.1.2 GAP vs Flatten — Why It Matters

Hook. Flattening a feature map destroys something valuable — the spatial layout. Two pixels that are neighbors in the image become strangers in the flat vector. Does the network still "know" they belong together?

Intuition. Imagine you have four houses on a street block. Flattening is like demolishing the block and listing each house on a separate page of a spreadsheet. You lose the fact that House A is next to House B. GAP is like taking one aerial photo that captures the whole block in a single shot. The spatial relationship is preserved.

Consider a section of the feature map after pooling:

$[0.5 0.1 0.6 0.7]$

Global Average Pooling: Take the average of these four values. Pass this single consolidated value directly to the dense layer. The neighborhood pattern is preserved — the machine "knows" these four cells are spatially adjacent.

Flattening: Each of the four values becomes a separate input to the feedforward network. The machine loses the spatial relationship. It cannot tell that 0.5, 0.6, 0.1, and 0.7 belong to the same $2 \times 2$ region. The local pattern is destroyed.

GAP retains spatial neighborhood information that flattening discards. When your convolutional layers have already extracted good features, GAP + one dense layer is often enough. Flattening makes sense when you plan to do further nonlinear processing with more dense layers.

Real-World & Domain Connection. GAP was popularized by the Network in Network paper and later adopted by ResNet and GoogLeNet. It is the standard way to bridge convolutional backbones to classifier heads in modern architectures. It also acts as a structural regularizer — forcing each feature map to summarize a single global concept reduces overfitting.

12.1.3 Activation Functions in CNNs

Hook. A convolution is just a dot product with a sliding window. Stack 100 convolution layers with no activation function, and the whole thing collapses into a single linear operation. Depth becomes useless. How does one simple function — ReLU — save the entire architecture?

Why ReLU after convolutions? After convolving a kernel over an image, some regions get high scores. Others get low or negative scores. High scores mean the pattern is present; low/negative scores mean it is absent. ReLU does two things:

Introduces nonlinearity, which is essential for learning complex, non-linear decision boundaries.
Filters out negative values. Negative scores mean the pattern is absent here. In most cases, we only need to pass forward where patterns ARE present. ReLU naturally zeros out negative activations.

Without nonlinearity, stacking convolution layers would still produce a linear function. No matter how deep you go, the whole network stays equivalent to a single linear operation — a matrix multiplication. ReLU breaks this linearity. Every layer can now learn a different, non-linear transformation of the data.

Q: Why do we add ReLU activation in the first and second convolution layers?

A: ReLU introduces nonlinearity into the system. Without it, stacking convolution layers would still produce a linear function. No matter how deep you go, the whole network stays equivalent to a single linear operation. Additionally, ReLU filters negative values: after convolution, regions where a pattern is absent get low/negative scores. Since we care about where patterns are present (not absent), ReLU zeros out those negative activations, passing only positive information upstream.

Pitfalls:

1. Dying ReLU. If a neuron receives consistently negative inputs, its gradient becomes zero and it stops learning. This is the "dying ReLU" problem. Using LeakyReLU or a small learning rate can help.

2. "More depth = always better" is false — without nonlinearity. A 100-layer network with no activations equals a 1-layer network. Activation functions are what make depth useful in the first place.

ReLU does two jobs at once. First, it adds nonlinearity, which makes depth useful. Second, it filters out noise by passing only positive feature detections upstream. It is the default activation in nearly every modern CNN.

12.1.4 Dropout Regularization

Hook. Your model does brilliantly on the training set — 99% accuracy. Then you test it on new images and it drops to 72%. The network learned to cheat. It memorized specific pixel combinations instead of learning general patterns. How do you force it to learn patterns that generalize?

Intuition. Think of dropout like a study group where half the members are randomly absent each session. No single person can become irreplaceable — everyone has to learn the full material. The group becomes more resilient. Dropout does the same to neurons: in each training step, it randomly knocks out half the neurons. No neuron can rely on another being present. The network must learn strong features that work under partial failure.

Dropout is a technique to prevent overfitting. In each training step, a fraction of neurons (e.g., 50%) in a layer are randomly switched off. Their connections do not participate in the forward pass, and gradients do not flow through them during backpropagation. In the next training step, a different random subset of neurons is dropped.

This forces the network to avoid co-adaptation. No single neuron can rely on specific other neurons always being present. The result is a stronger model.

In the MNIST example, dropout with a rate of 0.5 is applied after the 128-neuron hidden layer.

Pitfalls:

1. Dropout is only active during training. At test time, all neurons are active. The outputs are scaled down (by multiplying by the keep probability $1 - p$ ) to match the expected magnitude seen during training.

2. Too high a dropout rate kills learning. If you drop 90% of neurons, the remaining few may not carry enough signal. The network may fail to learn anything. Start at 0.2–0.5 and tune.

3. Dropout slows convergence. Since a different random subset is dropped each step, the effective capacity varies. More epochs may be needed.

Dropout is a simple but powerful regularizer. Randomly silencing neurons during training forces the network to build redundancy. Every feature must be learnable even when some neurons are missing. At test time, all neurons contribute, producing a stronger ensemble effect.

Real-World & Domain Connection. Dropout was introduced by Hinton et al. (2012) and Srivastava et al. (2014). It remains one of the most widely used regularization techniques across all neural network architectures — CNNs, RNNs, and Transformers alike. In modern practice, batch normalization has partially reduced the need for dropout, but dropout is still commonly applied to fully-connected layers.

12.1.5 Transfer Learning with EfficientNet

Hook. Training a CNN from scratch on ImageNet takes weeks on a GPU cluster and needs 1.2 million labeled images. But you only have 5,000 images of skin lesions. You need a classifier tomorrow. Can you borrow someone else's already-trained network?

Intuition. Think of transfer learning like hiring a master chef who trained in French cuisine to run your Italian kitchen. The chef already knows knife skills, heat control, and plating — the fundamentals. You only need to teach them the specifics of Italian sauces and pasta. You do not retrain them from scratch on how to boil water. Transfer learning does the same: the early layers of a pre-trained CNN have already learned to detect edges, textures, and shapes. You only fine-tune the later layers for your specific task.

What is transfer learning? Take a model pre-trained on one dataset. Reuse its learned weights for a different but related task. This avoids training from scratch. It is especially useful when the target dataset is small.

EfficientNet is a pre-trained CNN architecture whose weights were learned on the ImageNet dataset. To apply it to CIFAR-10 (32 × 32 color images across 10 classes), the following steps are taken:

The transfer learning pipeline (procedural):

Step 1 — Load the base model. Take the EfficientNet architecture with ImageNet weights. Set include_top=False to remove the output layer (the last dense layer that classified into 1000 ImageNet classes). The remaining layers become the "base model."

Step 2 — Freeze early layers. Not all layers need retraining. Set layer.trainable = False for the first 50 layers. These weights are "frozen." Their values remain as learned from ImageNet. During backpropagation, gradients do not flow through frozen layers.

How many layers to freeze is an empirical decision. There is no fixed rule. The decision depends on task similarity and data size. The more similar the tasks and the less target data you have, the more layers you should freeze.

Step 3 — Add custom top layers. Build a sequential model starting with the frozen base model, then add:

- Global Average Pooling

- Dense layer (with optional dropout)

- Dense output layer with 10 neurons (for CIFAR-10 classes)

Step 4 — Compile and train. Set up the optimizer, loss function, and metrics as usual. Training may use early stopping (e.g., patience=3 — stop if validation loss does not improve for 3 consecutive epochs).

CIFAR-10 and CIFAR-100: CIFAR-10 contains 32 × 32 color images across 10 classes. The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Each image typically features a single prominent object. CIFAR-100 has 100 fine-grained classes. Both datasets are commonly used for benchmarking image classification models.

Q: How do we know how many layers EfficientNet has so we can decide how many to freeze?

A: The EfficientNet architecture is documented in its research paper. You can look up the full architecture diagram to see the total number of layers. The decision of how many layers to freeze (say, 50) is empirical. There is no fixed formula. It depends on two things: how similar are the source and target datasets, and how much target data do you have?

Pitfalls:

1. Freezing too few layers with small data. If you have only 500 images and you leave most layers trainable, the network overfits. Freeze more when data is scarce.

2. Freezing too many layers when tasks are very different. Suppose your source task is ImageNet and your target is X-ray anomaly detection. Early-layer features (edges, textures) may transfer well. But mid-layer features (object parts) may not. Experiment with freezing different numbers of layers.

3. Not matching input preprocessing. ImageNet models expect a specific normalization (subtract mean, divide by standard deviation per channel). If you skip this step, the pre-trained weights will not work properly on your data.

Transfer learning is the industry standard. You rarely train a CNN from scratch. Load a pre-trained backbone (EfficientNet, ResNet), freeze early layers, replace the classification head, and fine-tune. Section 12.2 will now pivot from spatial patterns in images to temporal patterns in sequences. In this new setting, the IID assumption no longer holds.

Real-World & Domain Connection. Transfer learning with EfficientNet and ResNet backbones is the default approach in many domains. These include medical imaging (detecting tumors in CT scans), satellite imagery (classifying land use), and retail (product recognition). The practice of fine-tuning pre-trained models has become so universal that PyTorch and TensorFlow both provide pre-trained model zoos with one-line loading. EfficientNet specifically is notable for its compound scaling method. It scales depth, width, and resolution together. This achieves state-of-the-art accuracy with fewer parameters than previous architectures.

12.2 Motivation for Sequential Models

12.2.1 The IID Assumption and Its Limits

Hook. Every model you have built so far — linear regression, logistic regression, feedforward networks, CNNs — rests on a hidden assumption. The assumption: each data point is a fresh roll of the dice, unrelated to all others. But what happens when today's stock price directly determines tomorrow's? The dice are now loaded. Every model you know breaks.

Intuition. Think of IID data like a bag of marbles where each draw is independent and the bag's contents never change. Now think of sequential data like a conversation. Each thing you say depends on what was said before. You cannot shuffle the sentences and expect the conversation to still make sense. That is the difference between IID and sequential data.

In traditional machine learning and deep neural networks, data instances are assumed to be Independent and Identically Distributed (IID).

Independent: The data for one instance does not determine the data for another. Given a dataset of car features and prices, car 1's price does not influence car 2's price. The instances are unrelated to each other. This independence is key in MLE and MAP derivations. The joint probability factorizes as a product of individual probabilities.

Identical: All instances are drawn from the same underlying distribution. A model trained on car prices from the NCR region will not generalize to Bangalore. The data distributions differ — they are not identical.

Under the IID assumption, the joint probability of a dataset ${(x_{1}, y_{1}), \dots, (x_{n}, y_{n})}$ factorizes cleanly:

$ P (all data) = i = 1 \prod n P (x_{i}, y_{i}) $

Each term is independent of the others. This factorization makes MLE and gradient-based training work so well. The total log-likelihood is just a sum of per-example terms. No coupling between examples.

These assumptions hold for cross-sectional data. Examples include car prices, customer churn, and image classification. But they break down for sequential data where there is autocorrelation.

Scope: When the IID assumption fails.

- Time series data (stock prices, weather, sensor readings): Today's value depends on yesterday's. The joint probability does not factorize into independent terms.

- Text/NLP: Each word depends on the words before it. "The cat sat on the" constrains the next word heavily.

- Speech/audio: Sound at time $t + 1$ is highly correlated with sound at time $t$ .

- Video: Frame 100 depends on frame 99.

When the IID assumption fails, standard feedforward networks and gradient-based training still work. But the model architecture must account for the dependencies. This is the motivation for recurrent neural networks.

The IID assumption is the silent partner behind every standard ML algorithm. For cross-sectional data (house prices, customer churn, image classification) it holds. For sequential data (time series, text, speech) it fails. The next section shows exactly why.

12.2.2 Sequential Data and Autocorrelation

Hook. If you know today's stock price, can you predict tomorrow's? Not perfectly — but you can do better than random guessing. That is because stock prices have *autocorrelation*: each value is correlated with its own past values. This property is what makes sequence prediction possible — and it is exactly what breaks the IID assumption.

Intuition. Think of a line of dominoes. Tipping the first one topples the second, which topples the third. Each domino's fall is caused by the previous one. This is autocorrelation in the physical world. In data, autocorrelation means the value at time $t$ is correlated with the value at $t - 1$ , $t - 2$ , and so on.

Consider stock price prediction. The target variable $Y$ is the stock price:

$Y_{1}$ — price at day 1
$Y_{2}$ — price at day 2
$Y_{3}$ — price at day 3

Here, $Y_{1}$ influences $Y_{2}$ , and $Y_{2}$ influences $Y_{3}$ . There is autocorrelation — values at one time step are correlated with values at previous time steps. The IID assumption does not apply.

This same property exists in natural language. The words "the cat sat on the" determine the likelihood of the next word being "mat". The earlier words constrain the later words. If you change the order, the meaning changes entirely. The model's predictions would also change.

Sequential data is characterized by:

- Order matters. Changing the order changes the pattern.

- Dependencies span across time. Day 1 influences day 2, which influences day 3 — so day 1 indirectly influences day 3.

- Periodicity, cyclicity, and seasonality may be present (in time series).

Examples of sequential data:

Time series: stock prices, temperature readings, weather patterns
Natural language: sentences, paragraphs, documents
Speech: audio waveforms
Any data where instances have a meaningful ordering

Pitfall: Assuming independence when there is autocorrelation. Train a standard feedforward network on sequential data without accounting for temporal dependencies. The model will treat each time step as an independent sample. It may still learn something. Highly correlated features can appear predictive. But its predictions will be fragile. It cannot use the pattern of change across time.

Autocorrelation is both the problem (it breaks IID) and the opportunity (it is the signal we want to learn). A good sequence model must capture these dependencies. But what exactly must a sequence model handle? The next section lays out the constraints.

12.2.3 Constraints for Sequence Models

Hook. A feedforward network expects a fixed-size input vector. Give it a 5-word sentence and a 50-word sentence and it panics. It does not know what to do with variable-length data. How do you build a model that gracefully handles a 3-word tweet and a 300-page novel with the same architecture?

A good sequence model must handle:

Variable-length sequences. Sentences can have 3 words or 30 words. The model cannot assume a fixed input size.

Long-range dependencies. The word "mat" depends not just on the previous word "the" but potentially on "cat" which appeared several words earlier. The model must retain information across many time steps.

Order preservation. The model must be sensitive to the position of each element. Permuting the input should change the output.

A standard feedforward neural network cannot satisfy these constraints. If you jumble the words and feed them into a DNN, the network processes them independently. It cannot distinguish "the cat sat" from "sat cat the". A specialized architecture is needed — one that processes elements in order, remembers the past, and handles variable lengths.

Pitfalls — what naive approaches get wrong:

1. Padding/truncation hack. You can pad short sequences and truncate long ones to a fixed size. But padding wastes computation, and truncation throws away information. This is a band-aid, not a solution.

2. Sliding windows without state. A window of k previous time steps cannot capture dependencies longer than k. Choose k too small and you miss long-range patterns; choose k too large and your model explodes.

3. Positional encoding alone. Adding a position index as a feature tells the network *where* each element is. But it does not give the network *memory* of what came before. You need something that carries information forward.

Three hard constraints define the sequence modeling problem: variable-length inputs, long-range dependencies, and order sensitivity. Feedforward networks fail on all three. Section 12.3 explores the traditional approaches that people tried before RNNs became the standard solution.

Real-World & Domain Connection. The constraints for sequence models are not abstract — they drive real engineering decisions. Google's search autocomplete handles queries of 2 to 20+ words with the same model. Netflix's recommendation engine tracks viewing history of wildly different lengths per user. Apple's Siri transcribes speech ranging from "hey" to multi-minute dictations. Every production sequence model must handle variable lengths.

12.3 Traditional Approaches to Sequence Modeling

12.3.1 Bag-of-Words and Hand-Engineered Features

Hook. Before neural networks could read text, data scientists had to convert sentences into numbers by hand. A 10-word vocabulary meant 10 features. A 50,000-word vocabulary meant 50,000 features — most of them zero. And the sentence "not good" produced the same feature vector as "good." How did anyone build usable text models this way?

Intuition. Think of bag-of-words like counting ingredients in a recipe without noting the order. You know the dish has tomatoes, pasta, and basil. But you have no idea what the dish actually is. It could be pasta with tomato sauce or a tomato salad with pasta on the side. The counts are there, but the structure is lost.

Before neural sequence models, text was converted to tabular features manually. For sentiment classification (positive/negative/neutral), a data scientist would:

Identify a vocabulary of sentiment-bearing words (e.g., "good", "bad", "great", "terrible").
Create binary features: does this word appear in the sentence? (1 for yes, 0 for no)
Alternatively, use count-based features: how many times does the word appear?
Optionally, apply TF-IDF or other weighting schemes.

Problems with bag-of-words:

- Loss of word order. "Not good" and "good" produce identical feature vectors. But they mean opposite things.

- Vocabulary explosion. Every unique word becomes a feature. Dimensionality grows with vocabulary size.

- Semantic grouping is manual. The data scientist must decide that "good", "great", and "excellent" are semantically similar and must be treated similarly. This does not scale.

- Variable-length sentences are hard to represent in a fixed-size feature vector. Truncation or padding is needed.

Bag-of-words treats text as a bag of independent tokens. It loses order, it explodes dimensionality, and it forces manual feature engineering. Word embeddings solve the dimensionality and semantics problems — but order remains the unsolved challenge.

12.3.2 Word Embeddings

Hook. What if every word in the English language could be represented as just 100 to 300 floating-point numbers? Words with similar meanings would end up with similar numbers. That is the promise of word embeddings. No manual grouping. No vocabulary explosion. Just a dense vector.

Intuition. Think of word embeddings like placing words on a giant map. Words that are used in similar contexts ("king" and "queen") end up near each other. Words that are unrelated ("king" and "banana") end up far apart. The map is learned automatically from text — the computer figures out the layout without being told any rules.

Word embeddings solve the semantic similarity problem by converting each word into a dense, low-dimensional vector of real numbers. For example, a word might be represented as:

$embedding ("good") = [0.1, 2.5, - 1.5, 3.2]$

These numbers are not directly interpretable. They do not correspond to "presence of X" or "frequency of Y." Instead, they capture semantic relationships. Words with similar meanings have similar vectors. The embedding is learned from data — or can be pre-trained from large corpora using methods like Word2Vec, GloVe, or FastText.

Embeddings solve the vocabulary explosion problem. They map thousands of words into a compact vector space (typically 100–300 dimensions). Instead of a 50,000-dimensional sparse one-hot vector, each word becomes a dense 300-dimensional vector. The embedding matrix $E \in R^{∣ V ∣ \times d}$ maps each word index to its $d$ -dimensional vector.

However, a simple embedding lookup still does not capture word order — each word is processed independently. The sentence "dog bites man" and "man bites dog" produce the same set of embeddings, just in a different order. Without a mechanism to process the sequence, the meaning is still lost.

Pitfalls:

1. Embeddings need large corpora to be good. Word2Vec trained on a small dataset (e.g., 100K sentences) will not capture nuanced semantic relationships. Pre-trained embeddings (trained on billions of words) are usually better than training from scratch.

2. Out-of-vocabulary words. Every embedding vocabulary has a fixed size. Words not in the vocabulary get mapped to a special <UNK> token, which loses all semantic information about that word.

3. Embeddings without sequence context are blind to word order. An embedding tells you what a word *means* but not how it relates to other words in the current sentence. You need a sequence model (like an RNN) on top.

Word embeddings transform sparse, high-dimensional word identifiers into dense, semantically meaningful vectors. They solve vocabulary explosion and semantic grouping — but they still don't capture word order. For that, we need to model sequences. N-gram models were the first systematic attempt.

12.3.3 Markov Assumption and N-Gram Models

Hook. To predict the next word in a sentence, do you really need to remember every word since the beginning? Or is the last word — or the last two — usually enough? The Markov assumption says: keep only a small window. It makes the math tractable. But what do you lose when you throw away the past?

Intuition. Think of a Markov model like a board game. Your next move depends only on the square you are standing on — not on how you got there. A bigram model (first-order Markov) looks at just the previous word. A trigram model looks at the previous two. It is like reading with tunnel vision: you see only the last one or two words and guess the next.

To make sequence modeling computationally feasible, the Markov assumption limits the context window.

First-order Markov model (bigram): The current state depends only on the immediately previous state.

$ P (x_{t} ∣ x_{t - 1}, x_{t - 2}, \dots, x_{1}) \approx P (x_{t} ∣ x_{t - 1}) $

In NLP, this is a bigram model — pairs of consecutive words are analyzed:

- (token, the), (the, cat), (cat, sat), (sat, on), (on, the)

Each pair is treated as an independent observation. The probability of a sentence is the product of these bigram probabilities.

Second-order Markov model (trigram): The current state depends on the two previous states.

$ P (x_{t} ∣ x_{t - 1}, x_{t - 2}, \dots, x_{1}) \approx P (x_{t} ∣ x_{t - 1}, x_{t - 2}) $

In NLP, this is a trigram model:

- (token, the, cat), (the, cat, sat), (cat, sat, on), ...

An N-gram model generalizes to any fixed window size of $N - 1$ previous elements.

Worked example — joint probability under bigram. Suppose we want the probability of "the cat sat" under a bigram model:

$ P ("the", "cat", "sat") = P ("the") \cdot P ("cat" ∣ "the") \cdot P ("sat" ∣ "cat") $

If from training data: $P ("cat" ∣ "the") = 0.02$ and $P ("sat" ∣ "cat") = 0.01$ , then:

$ P \approx 0.05 \cdot 0.02 \cdot 0.01 = 1 0^{- 5} $

That is one in 100,000 — about right for a specific three-word phrase in a large corpus. The probability of the reversed order "sat cat the" would be much lower because $P ("cat" ∣ "sat")$ and $P ("the" ∣ "cat")$ are smaller numbers.

Small window (bigram): Computationally fast, but cannot capture long-range dependencies. The model cannot learn that "mat" depends on "cat" if they are separated by several words.
Large window (trigram, 4-gram, ...): Better at capturing dependencies, but the number of possible N-gram combinations grows as $V^{N}$ . Data sparsity becomes a severe problem — most N-grams never appear in the training data.

Pitfalls:

1. Data sparsity. For a vocabulary of 50K words, a trigram has $50, 00 0^{3} = 1.25 \times 1 0^{14}$ possible combinations. Even a billion-word corpus covers a tiny fraction of these. Most trigrams get a probability of zero, which is unrealistic.

2. Fixed window is inherently limited. Consider: "The cat, which was black and white and very fluffy, sat on the mat". The word "sat" depends on "cat" — which is 10 words back. No trigram can capture this.

3. Smoothing is mandatory but arbitrary. To handle unseen n-grams, you need smoothing (add-1, Kneser-Ney, etc.). The choice of smoothing method changes your results but has no principled justification.

N-gram models make sequence prediction tractable by limiting the context window. They work well enough for keyboard autocomplete and baseline benchmarks. But their fixed window is a hard ceiling. A vocabulary of $V$ words and an $N$ -gram window means $V^{N}$ possible patterns — most unseen. Section 12.4 introduces the latent variable model, which escapes the fixed window entirely.

Real-World & Domain Connection. N-gram models are the technology behind the predictive text on your phone's keyboard. When you type "I am on my," the keyboard suggests "way" because "on my way" is a high-frequency trigram. Google's original PageRank paper (1998) used bigram language models for spell correction. N-grams are also the baseline that every modern NLP paper must beat. If your fancy Transformer cannot outperform a simple trigram model on perplexity, you have a problem.

Q: In what real-life case would bigram be a better approach than trigram?

A: It depends on the trade-off between accuracy and computational complexity. If you are willing to accept less expressiveness for faster computation, use bigram. If the use case requires richer context and you have enough data, use trigram. The decision is empirical, based on the dataset and the prediction task. Modern NLP has mostly moved beyond N-gram models. RNNs and Transformers are preferred because fixed-window models are too constrained.

The fundamental limitation of N-gram models is the fixed window size. To overcome this, we need a model that can capture dependencies across an arbitrary number of time steps. And it must do so without an exponential explosion of parameters.

12.4 Latent Variable Models

12.4.1 The Hidden State Concept

Hook. An N-gram model must store explicit counts for every possible word sequence. With 50,000 words and a 3-word window, that is 125 trillion entries — most of which never appear. What if instead of storing every possible past, you stored just one compressed summary? A vector of, say, 256 numbers that captures the "essence" of everything you have seen so far?

Intuition. Think of the hidden state like a running summary in your head during a long conversation. You do not memorize every word verbatim. You compress the key points into an "essence." When the next sentence arrives, you update your mental summary. A latent variable model works the same way: it maintains a hidden vector that is continuously updated as new information arrives.

Instead of manually choosing a fixed window size, the model should automatically extract and propagate an essence. This essence is a compressed summary of all past information. The model uses it at each time step.

This is the latent variable model (also called a hidden variable model). The term "latent" means hidden. The hidden variable encodes information about the past. This information is not directly observable from the current input alone.

How it works:

At time step $t$ , the model receives:

1. The current input $x_{t}$

2. A hidden state $h_{t - 1}$ that summarizes all past observations $x_{1}, x_{2}, \dots, x_{t - 1}$

The model produces:

1. An output $y_{t}$ (prediction for the current step)

2. An updated hidden state $h_{t}$ that now summarizes $x_{1}, x_{2}, \dots, x_{t}$

This hidden state is passed forward to the next time step. Whether the sentence has 2 words or 1000 words, the model always receives a single hidden state vector. This is how variable-length sequences are handled.

Worked example — temperature prediction with hidden state.

| Time Step | Input (Temp) | Hidden State Captures |

|-----------|-------------|----------------------|

| Day 1 | 20°C | $h_{1}$ : starting point — 20°C |

| Day 2 | 22°C | $h_{2}$ : warming trend 20→22 |

| Day 3 | 24°C | $h_{3}$ : strong warming continues |

| Day 4 | ? | Predict next temperature from $h_{3}$ |

By Day 3, $h_{3}$ encodes not just "it is 24°C" but "it has been rising 2°C per day for 3 days". A simple feedforward network given only the current temperature (24°C) would predict 24°C again. The hidden state gives the model memory of the trend.

Sense-check: If the temperature had been 24, 22, 20 (falling), $h_{3}$ would capture a cooling trend instead. The hidden state adapts to the pattern, not just the value.

At each step, a function $G$ consolidates the previous hidden state and current input into the new hidden state:

$h_{t} = G (h_{t - 1}, x_{t})$

This function $G$ is what the model must learn. The hidden state is the foundation of recurrent neural networks.

Scope: What the hidden state can and cannot do.

- Can: Capture patterns learned from training data (trends, cycles, seasonal effects).

- Can: Adapt to variable-length sequences — one fixed-size vector handles any length.

- Cannot: Store perfectly lossless memory. The hidden state has fixed capacity (e.g., 256 floats). For very long sequences, older information gets compressed or forgotten. This is the *vanishing gradient* problem in RNNs — covered in Section 12.6.

The hidden state is the core idea behind all recurrent architectures. Instead of memorizing every past observation, the model learns to compress the relevant history into one fixed-size vector. The function $G$ — how to update this vector — is what the RNN learns during training.

12.4.2 Why the Hidden State Approach Beats N-Grams

Hook. An N-gram model stores $V^{N}$ possible sequences. A hidden state model stores exactly one vector of $H$ numbers — no matter how long the sequence. For $V = 50, 000$ , $N = 3$ , and $H = 256$ : N-gram needs 125 trillion entries. The hidden state needs 256 floats. That is a compression ratio of roughly 500 billion to 1.

An N-gram model stores explicit counts of specific word sequences. With a vocabulary of $V$ words and an $N$ -gram window, there are $V^{N}$ possible sequences. Most of these combinations are never seen in training.

A latent variable model stores a compressed vector $h_{t}$ of fixed dimensionality, regardless of sequence length. The capacity is bounded by the vector size, not by the vocabulary size raised to a power. The model learns which aspects of the past are relevant to keep and which to forget.

Comparison — N-gram vs. Hidden State:

| Dimension | N-Gram | Hidden State (RNN) |

|-----------|--------|-------------------|

| Memory | Explicit counts of word sequences | Compressed vector $h$ |

| Capacity | $V^{N}$ — grows explosively | $H$ — fixed, independent of $V$ |

| Unseen patterns | Zero probability (smoothing needed) | Can generalize via learned features |

| Context window | Fixed at $N - 1$ | Theoretically unlimited |

| Training | Count-based (MLE) | Gradient-based (SGD) |

| When to use | Fast, simple, interpretable baselines | Complex sequential patterns, variable-length data |

Q: What exactly does the hidden state store? Is it like Bayes' theorem?

A: The hidden state stores the essence as a set of weights. A single neuron does not correspond to a single word. In CNNs, a single pixel is not tied to a single weight. The same is true in RNNs — the weights learn complex patterns across the sequence. The hidden state is a dense vector capturing hierarchical features. The model's depth and width allow it to extract increasingly complex sequential patterns. It is not like Bayes' theorem — it is a learned, distributed representation rather than a probabilistic update.

Pitfalls:

1. The hidden state is lossy. You are compressing an arbitrary-length sequence into a fixed-size vector. Information will be lost. For very long sequences (hundreds to thousands of time steps), the hidden state may lose track of early information.

2. Training is harder than n-grams. N-gram models are count-based — fast and deterministic. RNNs need gradient descent, careful hyperparameter tuning, and can suffer from vanishing/exploding gradients.

3. Interpretability disappears. An n-gram model tells you exactly which word sequences drive predictions. A hidden state vector of 256 floats gives you no such transparency.

The hidden state compresses the entire past into one vector. It trades the combinatorial explosion of n-grams for a fixed-capacity bottleneck. The model learns what to remember and what to forget. Section 12.5 now shows exactly how an RNN implements this hidden state with neural network weights. The same architecture is applied at every time step.

Real-World & Domain Connection. The hidden state concept powers every major sequence modeling architecture — RNNs, LSTMs, GRUs, and even the encoder in Transformers. Google's Smart Compose in Gmail uses a hidden-state model to suggest email completions as you type. Speech recognition systems use hidden states to track phoneme context across time. The core idea is the same: compress the past into a vector and update it at each step.

12.5 Recurrent Neural Network Architecture

12.5.1 Core Idea — Same Network, Applied Sequentially

Hook. What if you could build one network with one set of weights? And apply it over and over to every element in a sequence? The same weights process word 1, then word 2, then word 3. The only thing that changes is a "memory vector" that gets passed along with each step. That is an RNN. One network. Infinite patience.

Intuition. Think of an RNN like a factory worker on an assembly line. A conveyor belt brings items one at a time. The worker processes each item using the same set of tools (weights). But the worker also carries a notebook (hidden state) where they jot down notes about what they have seen so far. When the next item arrives, the worker glances at the notebook. They see: "ah, I have seen three warm days in a row." Then they update their notes accordingly. Same worker, same tools, same notebook. Only the items and the notes change.

An RNN is a neural network with recurrent connections in its hidden layers. The key architectural idea: the same network is applied at every time step. It uses the same weights throughout. But at each step it receives two inputs: the current data and the hidden state from the previous step.

For a time series prediction task (predicting tomorrow's temperature from today's temperature, humidity, and pressure):

Input $x_{1}$ at day 1: a vector of features $[T_{1}, H_{1}, P_{1}]$ — three input neurons
Hidden layer: can have any number of neurons (e.g., 2, 4, 100). This hidden layer has TWO jobs:

Process the current input $x_{t}$
Receive and integrate the hidden state $h_{t - 1}$ from the previous time step

Output $y_{1}$ : prediction for day 2

At day 2, the same network (same weights) processes $x_{2}$ and $h_{1}$ to produce $y_{2}$ . At day 3, the same network processes $x_{3}$ and $h_{2}$ to produce $y_{3}$ . And so on.

The RNN's three weight matrices are shared across all time steps:

- $W_{x h}$ — weights from input to hidden layer

- $W_{hh}$ — recurrent weights from previous hidden state to current hidden state

- $W_{h y}$ — weights from hidden layer to output layer

At time $t$ , the hidden state update is:

$ h_{t} = tanh (W_{x h} x_{t} + W_{hh} h_{t - 1} + b_{h}) $

The output is:

$ y_{t} = W_{h y} h_{t} + b_{y} $

The term $W_{hh} h_{t - 1}$ is the recurrent contribution — it injects the memory of the past into the current computation. This is what makes it "recurrent."

An RNN is not multiple networks — it is one network, reused. Only the hidden state vector changes from step to step. This parameter sharing is what lets the RNN generalize to sequences of any length.

12.5.2 Unfolding the RNN Through Time

Intuition. An RNN with a cycle in its wiring diagram is hard to reason about. But if you "unroll" it — draw a copy of the network for each time step — the cycle disappears. You get a chain of identical networks. Time becomes a spatial dimension. This unfolded view is how we compute, train, and understand RNNs.

The recurrent architecture can be visualized by "unrolling" or "unfolding" the single network across time steps:

Time step 1:    x₁ → [Hidden] → y₁
                     ↓ (recurrent weight W_hh)
Time step 2:    x₂ → [Hidden] → y₂
                     ↓
Time step 3:    x₃ → [Hidden] → y₃

Each box labeled [Hidden] is the same network — same input-to-hidden weights, same hidden-to-output weights, and same recurrent weights. The arrow between hidden states represents the recurrent weight matrix $W_{hh}$ that captures temporal dependencies between successive time steps.

What do the recurrent connections capture? The hidden layer at time $t - 1$ feeds information to the hidden layer at time $t$ through a dense connection. If the hidden layer has $H$ neurons, the recurrent weight matrix $W_{hh}$ has dimensions $H \times H$ . Every neuron in the previous time step connects to every neuron in the current time step. After unfolding, the computational graph becomes a deep feedforward network. The depth equals the sequence length. And all layers share the same weights.

Pitfalls:

1. Unfolding is conceptual, not literal. You do not actually create separate copies of the network in memory. The unfolded view is a mental model for understanding the computation. In code, you use a loop: for t in range(T): h = tanh(W_xh @ x[t] + W_hh @ h + b_h).

2. Long sequences ⇒ very deep unfolded graph. A 1000-word sentence produces a 1000-layer unfolded network. Gradients must flow across all 1000 layers. This causes the vanishing/exploding gradient problem (Section 12.6).

Unfolding converts a recurrent computation into a chain of identical layers. This view is essential for understanding both forward propagation and backpropagation. Each layer processes one time step. In backpropagation, gradients flow backward through the chain, accumulating across time steps.

Hook. A feedforward network needs separate weights for position 1, position 2, and position 3. An RNN reuses the same weights everywhere. For a 100-word sentence, the feedforward network needs 100 independent sets of parameters. The RNN uses exactly three weight matrices — no matter the sentence length. How is this possible?

All weights are shared across time steps. The same weight matrices are used at $t = 1$ , $t = 2$ , $t = 3$ , and every subsequent step:

$W_{x h}$ — weights from input to hidden layer (same for all $t$ )
$W_{hh}$ — recurrent weights from previous hidden state to current hidden state (same for all $t$ )
$W_{h y}$ — weights from hidden layer to output layer (same for all $t$ )
Biases $b_{h}$ and $b_{y}$ — also shared

During training, these shared weights are updated using gradients accumulated across all time steps. Parameter sharing means the model learns a single set of temporal patterns that apply throughout the entire sequence, regardless of sequence length.

Parameter count for an RNN:

| Component | Dimensions | Count |

|-----------|-----------|-------|

| Input → Hidden ( $W_{x h}$ ) | $D$ inputs, $H$ hidden | $D \times H$ |

| Hidden → Output ( $W_{h y}$ ) | $H$ hidden, $V$ outputs | $H \times V$ |

| Recurrent ( $W_{hh}$ ) | $H \times H$ | $H \times H$ |

| Biases ( $b_{h}$ , $b_{y}$ ) | $H$ and $V$ | $H + V$ |

Worked example — RNN parameter count.

For $D = 10, 000$ vocabulary size, $H = 256$ hidden units, and $V = 10, 000$ output vocabulary:

$ W_{x h} W_{h y} W_{hh} Biases = 10, 000 \times 256 = 2, 560, 000 = 256 \times 10, 000 = 2, 560, 000 = 256 \times 256 = 65, 536 = 256 + 10, 000 = 10, 256 $

$ Total = 2, 560, 000 + 2, 560, 000 + 65, 536 + 10, 256 = 5, 185, 792 $

The recurrent weights $W_{hh}$ (65,536) are the only addition compared to a standard feedforward network with the same input/output dimensions. The rest — $W_{x h}$ and $W_{h y}$ — exist in any network. Sense-check: For a feedforward network processing 10,000-dimensional one-hot word vectors, you would need similar $W_{x h}$ and $W_{h y}$ matrices anyway. The RNN's extra cost is $W_{hh}$ — roughly 1.3% of the total in this example.

Parameter sharing is how RNNs handle variable-length sequences. You pay for three weight matrices regardless of sequence length. Compare this to an N-gram model that needs $V^{N}$ parameters for an $N$ -word context. The RNN's parameter count is independent of how far back it looks.

12.5.4 Forward Propagation Equations

At each time step $t$ , the forward pass computes:

Hidden state:

$ h_{t} = tanh (W_{x h} x_{t} + W_{hh} h_{t - 1} + b_{h}) $

Three components are added:

1. $W_{x h} x_{t}$ — contribution from the current input

2. $W_{hh} h_{t - 1}$ — contribution from the previous hidden state

3. $b_{h}$ — bias

A nonlinear activation (typically $tanh$ ) is then applied. The first two components distinguish an RNN from a standard feedforward network. A standard network uses only $W_{x h} x_{t} + b_{h}$ . The RNN adds the recurrent contribution $W_{hh} h_{t - 1}$ .

Output:

$ y_{t} = W_{h y} h_{t} + b_{y} $

If classification is required, softmax is applied:

$ y^_{t} = softmax (W_{h y} h_{t} + b_{y}) $

Initial hidden state: At $t = 0$ (before any input), the hidden state $h_{0}$ is typically initialized to a vector of zeros. This is the starting point from which the first time step begins building its memory.

Why $tanh$ ? The tanh activation maps values to $(- 1, 1)$ , which is better than sigmoid's $(0, 1)$ for hidden states. Centering around zero helps gradients flow in both directions during backpropagation. ReLU is less common in basic RNNs. Its unbounded positive outputs can cause the hidden state to explode over many time steps. Modern architectures (LSTM, GRU) use gating mechanisms that work better with sigmoid and tanh combinations.

Worked numerical example — simple RNN forward pass.

Assume:

- Input dimension: 2 (features per time step)

- Hidden units: 3

- Output units: 2

First time step input: $x_{1} = [1.0, 0.5]$

Initial hidden state: $h_{0} = [0, 0, 0]$ (zero initialization)

Weights:

$ W_{x h} = [0.1 0.4 0.2 0.5 0.3 0.6] W_{hh} = 0.01 0.04 0.07 0.02 0.05 0.08 0.03 0.06 0.09 W_{h y} = 0.7 0.9 1.1 0.8 1.0 1.2 $

Step 1 — Compute hidden activations (before activation):

Contribution from input:

$ [1.0 \times 0.1 + 0.5 \times 0.4, 1.0 \times 0.2 + 0.5 \times 0.5, 1.0 \times 0.3 + 0.5 \times 0.6] = [0.3, 0.45, 0.6] $

Since $h_{0} = [0, 0, 0]$ , $W_{hh} h_{0} = [0, 0, 0]$ .

After adding bias $b_{h} = [0, 0, 0]$ :

$ z_{h} = [0.3, 0.45, 0.6] $

Step 2 — Apply tanh activation:

$ h_{1} = tanh (z_{h}) = [tanh (0.3), tanh (0.45), tanh (0.6)] \approx [0.291, 0.422, 0.537] $

Step 3 — Compute output:

$ W_{h y} h_{1} = [0.291 \times 0.7 + 0.422 \times 0.8 + 0.537 \times 1.1, 0.291 \times 0.9 + 0.422 \times 1.0 + 0.537 \times 1.2] = [0.204 + 0.338 + 0.591, 0.262 + 0.422 + 0.644] = [1.133, 1.328] $

After adding bias and applying softmax (if needed), this produces the final prediction $y_{1}$ .

Sense-check: The values $0.291, 0.422, 0.537$ are all in $(- 1, 1)$ as expected from tanh. The output values $1.133$ and $1.328$ are raw logits — softmax would convert them to probabilities summing to 1. At the next time step, the same weight matrices are reused with $h_{1}$ as the recurrent input.

Pitfalls:

1. Zero initialization of $h_{0}$ is the default but not always best. For some tasks, a learned initial state can help. The zero vector gives the network a "blank slate" at the start.

2. $tanh$ saturation. For inputs far from zero, $tanh$ saturates (gradient ≈ 0). If $W_{x h} x_{t}$ or $W_{hh} h_{t - 1}$ produce large values, the gradient signal dies. This is one cause of the vanishing gradient problem.

3. Dimension confusion. $W_{x h}$ has shape $(H, D)$ if written in code as h = tanh(W_xh @ x + W_hh @ h_prev + b_h). Always check: input $x_{t}$ has shape $(D,)$ , hidden state has shape $(H,)$ , so $W_{x h}$ must map $D \to H$ , giving shape $(H, D)$ . Different textbooks transpose this — the math works either way as long as you are consistent.

The forward pass of an RNN is simple: three matrix multiplications (input-to-hidden, hidden-to-hidden, hidden-to-output), one tanh, and one optional softmax. The real complexity lies in training. Specifically, backpropagation through time must navigate the unfolded chain to assign credit across all time steps. That is the topic of Section 12.6.

Real-World & Domain Connection. The three-matrix RNN architecture (sometimes called the Elman RNN, after Jeffrey Elman who proposed it in 1990) is the simplest recurrent architecture. It underlies everything from early speech recognition systems to the first successful machine translation models. While modern systems use LSTMs and Transformers, the Elman RNN's forward pass equations remain the conceptual foundation for understanding how recurrence works.

12.6 Training Recurrent Neural Networks

12.6.1 Loss Computation Across Time

Hook. A feedforward network makes one prediction and gets one loss value. An RNN makes a prediction at every single time step. For a 100-word sentence, that is 100 predictions and 100 losses. How do you combine all those losses into a single number that drives training?

Unlike a feedforward network, an RNN produces an output at every time step. This means there is a loss at every time step too.

For a sequence of length $T$ , the total sequence loss is the sum of losses at all time steps:

$ L_{total} = t = 1 \sum T L_{t} $

The loss reported during training (the average per-time-step loss) is:

$ L ˉ = \frac{1}{T} t = 1 \sum T L_{t} $

In NLP contexts, the time step is called a token and the reported loss is the per-token average loss. For time series, it is the per-time-step average loss.

Worked example — loss across time. Consider a next-word prediction task with a 3-word input "the cat sat" and vocabulary of 10,000 words. The target sequence is "cat sat mat".

At $t = 1$ , input "the", the model outputs a probability distribution over 10,000 words. The correct next word is "cat". Cross-entropy loss at $t = 1$ : $L_{1} = - lo g P ("cat" ∣ "the")$ .

At $t = 2$ , input "cat", target "sat". $L_{2} = - lo g P ("sat" ∣ h_{2})$ .

At $t = 3$ , input "sat", target "mat". $L_{3} = - lo g P ("mat" ∣ h_{3})$ .

Total loss: $L_{total} = L_{1} + L_{2} + L_{3}$ . Average: $L ˉ = (L_{1} + L_{2} + L_{3}) /3$ .

Sense-check: If the model assigns probability 0.5 to the correct word each time, $L_{t} = - lo g (0.5) \approx 0.693$ and $L ˉ \approx 0.693$ . Better predictions give lower loss.

Pitfall: Confusing average loss with final-step loss. The reported training loss is almost always the average across all time steps, not just the last step. If you only compute loss at the final time step, the model ignores intermediate predictions. That may be fine for some tasks like sentiment classification at the end. But it fails for tasks like language modeling where every next-word prediction matters.

The training loss for an RNN is the sum (or average) of per-step losses. This makes sense: the model is rewarded for making good predictions at every time step, not just at the end. But totalling losses across time also means gradients must flow backward through every time step. This brings us to Backpropagation Through Time.

12.6.2 Backpropagation Through Time — Introduction

Hook. In a feedforward network, gradients flow backward through layers — layer 3 → layer 2 → layer 1. In an RNN, gradients must flow backward through layers AND backward through time — $t = 3 \to t = 2 \to t = 1$ . This dual path through time is what makes RNN training so much harder. It is also what gives RNNs their memory.

Training an RNN uses Backpropagation Through Time (BPTT). Unlike standard backpropagation, the gradient must flow through two paths:

Spatial path (through layers): From the output layer backward through the hidden layer to the input layer — same as standard backpropagation. This captures errors due to hierarchical feature extraction.

Temporal path (through time): From a later time step backward through the recurrent connections. This captures how errors at time $t$ depend on $h_{t - 1}$ , which in turn depends on $h_{t - 2}$ , all the way back to $h_{1}$ .

The total gradient is the sum of gradients from both paths. BPTT unrolls the network through time, computes gradients at each unrolled copy, and sums them to update the shared weights. Because all time steps share the same $W_{hh}$ , the gradient update for $W_{hh}$ gets contributions from every single transition $h_{t - 1} \to h_{t}$ . These contributions come from across the entire sequence.

BPTT is more complex than standard backpropagation. Gradients must flow backward through an arbitrary number of time steps. This causes two problems:

Vanishing gradient problem: Gradients become exponentially small over many time steps. Early parts of the sequence receive effectively zero gradient and cannot be learned from.
Exploding gradient problem: Gradients become exponentially large over many time steps. Weight updates become huge and training destabilizes.

Scope: When vanishing/exploding gradients hit hardest.

- Vanishing: Long sequences (100+ time steps), especially with saturating activation functions like tanh/sigmoid. The recurrent weight matrix $W_{hh}$ is multiplied at every step. If its eigenvalues are less than 1, gradients shrink. If greater than 1, they blow up.

- Exploding: Unstable architectures, high learning rates, sequences with large variations. Gradient clipping (capping gradient norms) is the standard fix for exploding gradients.

- Deep feedforward networks face the same issues with many layers. In RNNs, these problems happen over time steps instead of layers. But the math is the same. It is repeated multiplication of a weight matrix during backpropagation.

BPTT is the training algorithm for RNNs. It computes gradients by unrolling the network through time. Errors are propagated backward across both paths — through layers and through time steps. The resulting vanishing and exploding gradient problems are the central challenge of RNN training. Solutions (LSTM, GRU, gradient clipping) are covered in the next lecture modules.

Real-World & Domain Connection. BPTT was formalized by Werbos (1990) and popularized by Rumelhart, Hinton, and Williams. The vanishing gradient problem was analyzed in detail by Hochreiter (1991) and Bengio et al. (1994) — work that directly motivated the invention of Long Short-Term Memory (LSTM) networks. Today, gradient clipping (limiting the norm of the gradient vector) is a standard line in every RNN training script.

12.7 Student Questions and Answers

12.7.1 Key Clarifications

Q: Do we send each word in a sequence as one row of data? How is a sentence processed in an RNN?

A: Yes — at each time step, one element is passed as input to the network. It could be a word or a time point. The input at time $t$ is a vector $x_{t}$ . For NLP, each word is first converted to an embedding vector. All embeddings have the same fixed dimensionality (e.g., 300). For a sentence with 5 words, there are 5 time steps. The sequence is processed one step at a time. The hidden state carries forward the context.

Follow-up detail: The embedding lookup is the very first operation. You take a word index (an integer), look up its row in the embedding matrix $E$ , and get back a dense vector. This vector — not the raw index — is $x_{t}$ , the input to the RNN.

Q: How does the hidden state get updated? Does it have to wait for all previous time steps to complete?

A: The hidden state at time $t$ ( $h_{t}$ ) is computed from two things. These are the current input $x_{t}$ and the previous hidden state $h_{t - 1}$ . It does NOT wait for all previous time steps to be processed individually — $h_{t - 1}$ already encapsulates all prior information. However, in a standard RNN, each time step is processed sequentially. You must compute $h_{1}$ before $h_{2}$ , and so on. This sequential dependency is a fundamental limitation of RNNs: you cannot parallelize across time steps. Transformers solve this through parallel attention mechanisms — they process all positions simultaneously. That is covered in the next module.

Common confusion point (several students asked variations of this): The hidden state vector $h_{t}$ is NOT a lookup table. It does not map each past word to a weight. It is a dense, distributed representation. No single number in $h_{t}$ corresponds to "the word cat was seen 3 steps ago". Instead, the entire vector collectively encodes the sequence context. This is exactly like CNN feature maps — no single pixel in the feature map is tied to one weight. The representation is holistic.

Exam Guidance Summary

Exam note: The exam focuses on conceptual understanding and architecture comparisons, not on coding RNNs from scratch. Expect questions that ask you to explain *why* something works a certain way, or to compare two approaches.

CNN architecture differences (GAP vs flatten). A frequently tested conceptual question. Be able to explain what information each approach preserves or discards, and when to use which.

Parameter counting. You must be able to count parameters for convolution layers, dense layers, and pooling layers. Remember: convolution layers have $K \times F \times F \times C + K$ parameters. Pooling layers have zero learnable parameters. Dense layers have $N \times M + M$ parameters.

ReLU in CNNs. Know its two purposes. It introduces nonlinearity (making depth useful) and filters negative activations (passing only positive feature detections upstream).

Dropout. Understand it as a regularization technique that randomly switches off neurons during training. Know that it prevents co-adaptation and that it is only active during training, not inference.

Transfer learning pipeline. Be able to describe all four steps. Load base model with include_top=False. Freeze early layers. Add custom top layers (GAP + dense + output). Compile and train. Understand the trade-off in deciding how many layers to freeze.

IID assumption and its failure for sequential data. Know what "independent" and "identically distributed" mean. Be able to give examples where IID holds (cross-sectional data) and where it fails (time series, text, speech).

Markov assumption and N-gram models. Know bigram (first-order Markov: $P (x_{t} ∣ x_{t - 1})$ ) and trigram (second-order: $P (x_{t} ∣ x_{t - 1}, x_{t - 2})$ ). Understand the fixed-window limitation and the data sparsity problem ( $V^{N}$ possible combinations).

Latent variable model and hidden state. Understand that $h_{t}$ compresses all past $x_{1}, \dots, x_{t - 1}$ into one fixed-size vector. Be able to explain how this handles variable-length sequences.

RNN architecture. Core exam content includes several key concepts: the three weight matrices ( $W_{x h}$ , $W_{hh}$ , $W_{h y}$ ). Parameter sharing across time. The forward pass equations $h_{t} = tanh (W_{x h} x_{t} + W_{hh} h_{t - 1} + b_{h})$ and $y_{t} = W_{h y} h_{t} + b_{y}$ . And unfolding.

RNN loss and training. Loss is the sum (or average) of per-time-step losses. BPTT propagates gradients through both layers and time steps. Be aware of vanishing/exploding gradients and that solutions (LSTM, GRU) are covered in the next module.

CIFAR-10 and CIFAR-100. Know these are 32×32 color image datasets with 10 and 100 classes respectively. They are commonly used for benchmarking image classification.

Key Industry Applications

CNN architectures. CNNs power image classification, object detection, and visual recognition in production systems worldwide. They are deployed in healthcare imaging (tumor detection in CT/MRI scans). They are also used in autonomous vehicles (pedestrian and sign detection). Other applications include retail (automated checkout, visual search) and manufacturing (defect inspection on assembly lines).

Transfer Learning. Fine-tuning pre-trained models like EfficientNet or ResNet on domain-specific data is now the industry standard. It avoids training from scratch — a critical advantage when labeled data is scarce (medical imaging, satellite imagery, industrial inspection). Every major cloud platform (AWS SageMaker, Google Vertex AI, Azure ML) offers pre-trained model zoos with one-line loading.

EfficientNet. A family of models that scale efficiently across depth, width, and resolution using a compound scaling method. EfficientNet variants (B0 through B7) provide a range of accuracy-efficiency trade-offs. They are widely used as backbones in computer vision pipelines for transfer learning.

Recurrent Neural Networks. RNNs power time series forecasting for stock prices, weather prediction, and energy demand. They are used in speech recognition (converting audio to text) and machine translation (converting one language to another). They also handle anomaly detection (identifying unusual patterns in sensor streams or server logs). While Transformers have supplanted RNNs in many NLP tasks, RNNs remain strong in certain areas. These include low-latency streaming applications and on-device models where computational resources are constrained.

Word Embeddings. Word embeddings (Word2Vec, GloVe, FastText) are foundational building blocks for modern NLP. They are used in search engines (understanding query intent) and recommendation systems (matching item descriptions to user preferences). They also power chatbots (representing conversational context). Additionally, they enable sentiment analysis (detecting emotional tone in reviews and social media). Pre-trained embeddings trained on billions of words are the default starting point — training embeddings from scratch is rare in industry.

N-gram Models. N-gram models remain relevant in lightweight, low-latency applications. They power keyboard text prediction on mobile devices (suggesting the next word as you type). They also serve as fast baseline models in NLP benchmarks. When computational resources are minimal and 90% accuracy is enough, a well-tuned trigram model with smoothing can still be the right tool.

DNN Lecture 12 notes · Recurrent Neural Networks — Foundations and Architecture

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

1CNN Architecture Review and Transfer Learning

Reviews CNN architecture for MNIST digit classification, GAP vs flatten, activation functions, dropout regularization, and transfer learning with EfficientNet.

2Motivation for Sequential Models

Explains the IID assumption and its failure for sequential data, autocorrelation in time series, and constraints for sequence models.

3Traditional Approaches to Sequence Modeling

Covers bag-of-words, word embeddings, Markov assumption, and N-gram models for sequence prediction.

4Latent Variable Models

Introduces the hidden state concept and explains why latent variable models overcome N-gram limitations.

5Recurrent Neural Network Architecture

Core RNN architecture, unfolding through time, parameter sharing, and forward propagation equations.

6Training Recurrent Neural Networks

Loss computation across time steps, backpropagation through time, and vanishing/exploding gradients.

7Student Questions and Answers

Clarifications on word-level RNN processing, hidden state updates, and distributed representations.

8Exam Guidance Summary

Exam-focused revision topics covering CNN architecture, RNN fundamentals, and key comparisons.

9Key Industry Applications

Real-world applications of CNNs, transfer learning, RNNs, word embeddings, and N-gram models.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

CNN Architecture (GAP vs Flatten)

Must-know: Global Average Pooling preserves spatial neighborhood information by consolidating each feature map into a single average value. Flattening destroys spatial relationships by converting the feature map into a flat vector. GAP + one dense layer is sufficient when the convolutional base has already extracted good features.

$Params = K \times (F \times F \times C) + K$ for a convolution layer with $K$ kernels of size $F \times F$ and $C$ input channels.

Top pitfall: Confusing pooling parameters with learnable parameters. Pooling layers have zero learnable weights.

Self-check: A 2x2 feature map has values 0.5, 0.6, 0.1, 0.7. What does GAP produce and what does flattening produce?

Connects to: Activation Functions in CNNs, Dropout Regularization.

Transfer Learning Pipeline

Must-know: Four steps: (1) load pre-trained base model with include_top=False, (2) freeze early layers, (3) add custom top layers (GAP + dense + output), (4) compile and train. Number of frozen layers depends on task similarity and data size.

Top pitfall: Freezing too few layers with small data leads to overfitting. Freezing too many layers when tasks are very different prevents the model from adapting.

Self-check: You have 500 medical images and a pre-trained EfficientNet. How many layers do you freeze and why?

Connects to: CNN Architecture Review.

IID Assumption and Its Limits

Must-know: IID means each data point is Independent (unrelated to others) and Identically distributed (drawn from the same distribution). The joint probability factorizes as a product. This holds for cross-sectional data but fails for sequential data where autocorrelation exists.

$P (all data) = i = 1 \prod n P (x_{i}, y_{i})$

Top pitfall: Training a standard feedforward network on sequential data without accounting for temporal dependencies. The model may appear to learn but its predictions will be fragile.

Self-check: Give three examples where the IID assumption fails and explain why.

Connects to: Sequential Data and Autocorrelation.

Markov Assumption and N-Gram Models

Must-know: The Markov assumption limits context to a fixed window. Bigram (first-order): P(x_t | x_{t-1}). Trigram (second-order): P(x_t | x_{t-1}, x_{t-2}). The number of possible N-gram combinations grows as V^N, causing severe data sparsity.

$P (x_{t} ∣ x_{t - 1}, x_{t - 2}, \dots, x_{1}) \approx P (x_{t} ∣ x_{t - 1})$ for a bigram model.

Top pitfall: Fixed window is inherently limited. The word 'sat' may depend on 'cat' which is 10 words back. No trigram captures this.

Self-check: With a 50,000-word vocabulary, how many possible trigram combinations exist? Why is this a problem?

Connects to: Latent Variable Models.

Hidden State (Latent Variable Model)

Must-know: The hidden state h_t compresses all past observations into one fixed-size vector. At time t, the model receives input x_t and previous hidden state h_{t-1}, produces output y_t and updated hidden state h_t. This handles variable-length sequences with a single fixed-capacity vector.

$h_{t} = G (h_{t - 1}, x_{t})$

Top pitfall: The hidden state is lossy. Compressing an arbitrary-length sequence into a fixed-size vector means older information may be forgotten (vanishing gradient problem).

Self-check: How does the hidden state handle a 2-word sentence vs a 1000-word sentence with the same architecture?

Connects to: RNN Architecture.

RNN Architecture and Forward Propagation

Must-know: An RNN has three shared weight matrices: W_xh (input to hidden), W_hh (hidden to hidden, recurrent), W_hy (hidden to output). The same weights are reused at every time step. Parameter sharing lets RNNs handle sequences of any length.

$h_{t} = tanh (W_{x h} x_{t} + W_{hh} h_{t - 1} + b_{h})$ $y_{t} = W_{h y} h_{t} + b_{y}$

Top pitfall: Tanh saturation causes vanishing gradients. For large inputs, tanh gradients approach zero. Dimension confusion: W_xh maps from input dimension D to hidden size H, giving shape (H, D).

Self-check: A vocabulary has 10,000 words and you use 256 hidden units. Calculate the total number of parameters in the RNN.

Connects to: BPTT, Hidden State.

Backpropagation Through Time

Must-know: BPTT propagates gradients through two paths: spatial (through layers, standard backprop) and temporal (through time steps via recurrent connections). The gradient update for W_hh gets contributions from every time step transition.

$L_{total} = t = 1 \sum T L_{t}$

Top pitfall: Vanishing gradients: repeated multiplication of W_hh during backpropagation causes gradients to shrink exponentially for long sequences. Exploding gradients: when eigenvalues of W_hh exceed 1, gradients blow up.

Self-check: Why does a 1000-word sentence cause vanishing gradients in a basic RNN but not in a 10-word sentence?

Connects to: RNN Architecture.

Previous LectureLecture 11 · CNN Architectures and Applications

Next LectureLecture 13 · Recurrent Neural Networks — Advanced Architectures

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Recurrent Neural Networks — Foundations and Architecture

12.1 CNN Architecture Review and Transfer Learning

12.1.1 CNN Architecture Review — MNIST Digit Classification

12.1.2 GAP vs Flatten — Why It Matters

12.1.3 Activation Functions in CNNs

12.1.4 Dropout Regularization

12.1.5 Transfer Learning with EfficientNet

12.2 Motivation for Sequential Models

12.2.1 The IID Assumption and Its Limits

12.2.2 Sequential Data and Autocorrelation

12.2.3 Constraints for Sequence Models

12.3 Traditional Approaches to Sequence Modeling

12.3.1 Bag-of-Words and Hand-Engineered Features

12.3.2 Word Embeddings

12.3.3 Markov Assumption and N-Gram Models

12.4 Latent Variable Models

12.4.1 The Hidden State Concept

12.4.2 Why the Hidden State Approach Beats N-Grams

12.5 Recurrent Neural Network Architecture

12.5.1 Core Idea — Same Network, Applied Sequentially

12.5.2 Unfolding the RNN Through Time

12.5.3 Parameter Sharing — The Critical Property

12.5.4 Forward Propagation Equations

12.6 Training Recurrent Neural Networks

12.6.1 Loss Computation Across Time

12.6.2 Backpropagation Through Time — Introduction

12.7 Student Questions and Answers

12.7.1 Key Clarifications

Exam Guidance Summary

Key Industry Applications

DNN Lecture 12 notes · Recurrent Neural Networks — Foundations and Architecture

Sections Breakdown

Exam Revision Notes

CNN Architecture (GAP vs Flatten)

Transfer Learning Pipeline

IID Assumption and Its Limits

Markov Assumption and N-Gram Models

Hidden State (Latent Variable Model)

RNN Architecture and Forward Propagation

Backpropagation Through Time

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat