Deep Neural Networks

Recurrent Neural Networks — Advanced Architectures

Published: 2026-07-15

Level: postgraduate

Audience: Postgraduate students in Deep Neural Networks

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Activation Function, Classification — covered in Lecture 2: Deep Neural Network Components and Perceptron
Classification — covered in Lecture 3: Perceptron Learning and Introduction to Regression
Activation Function — covered in Lecture 4: Linear Neural Networks for Regression
Softmax, Sigmoid, Activation Function, Classification — covered in Lecture 5: Gradient Descent Variants, Classification, and Evaluation
Sigmoid, Vanishing Gradient, Exploding Gradient, Activation Function, Classification — covered in Lecture 6: Deep Feedforward Neural Networks
Sigmoid, Vanishing Gradient, Exploding Gradient, Activation Function — covered in Lecture 7: Deep Feed-Forward Neural Network Architecture Design and Training
Softmax — covered in Lecture 8: Exam Revision and Introduction to Convolutional Neural Networks
RNN, Recurrent, Hidden State, Backpropagation Through Time, Activation Function, Classification — covered in Lecture 12: Recurrent Neural Networks — Foundations and Architecture

Recurrent Neural Networks — Advanced Architectures

13.1 RNN Recap and Backward Propagation Through Time

13.1.1 Sequential Data and the Need for RNN

Hook. Feedforward networks treat every input as independent. But most real-world data — stock prices, speech, text — has order. Can a model that sees every moment in isolation ever learn that "yesterday's rain makes today's flood"?

Intuition + Analogy. A weather forecaster who only looks at today's barometer has no context. A good forecaster remembers the pressure trend over the past week. An RNN works the same way. At each new time step, it receives the current input and a summary of everything it has seen so far. This summary is called the hidden state. This summary flows forward like a running note you update as you read a paragraph. Each new sentence gets written alongside what you already understood from earlier sentences.

Formalize — RNN Forward Equations.

An RNN uses shared weights across every time step. That is what makes it different from a feedforward net that would need separate parameters at each position. At time step $t$ :

$ a_{t} = W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h} $

$ h_{t} = tanh (a_{t}) $

$ o_{t} = W_{h y} h_{t} + b_{y} $

$ y^_{t} = softmax (o_{t}) $

Where:

$x_{t}$ — input vector at time $t$ (e.g., a word embedding)
$h_{t - 1}$ — hidden state from the previous step (the running summary)
$W_{x h}$ — input-to-hidden weight matrix (same for every $t$ )
$W_{hh}$ — hidden-to-hidden (recurrent) weight matrix (same for every $t$ )
$W_{h y}$ — hidden-to-output weight matrix (same for every $t$ )
$b_{h}, b_{y}$ — bias vectors
$tanh$ — squashes the activation between -1 and 1

The key insight: the same $W_{x h}$ , $W_{hh}$ , and $W_{h y}$ serve every time step. This is parameter sharing — unfolding the RNN reveals a deep structure where weights are tied across all temporal layers. The hidden state $h_{t}$ is a lossy summary. It compresses the entire past $x_{1}, \dots, x_{t}$ . The result is a single fixed-length vector.

Worked Example — Toy Sequence of Length 3.

Suppose input dimension $d = 2$ , hidden dimension $h = 3$ , and:

$ x_{1} = [10], x_{2} = [01], x_{3} = [11] $

The weight matrices (all shared across steps):

$ W_{x h} = [0.5 0.3 0.2 0.4 0.1 0.6], W_{hh} = 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.1 $

Initial state $h_{0} = 000$ . Biases are zero for simplicity.

Step $t = 1$ :

$ W_{x h} x_{1} = [0.5 0.3 0.2 0.4 0.1 0.6]^{T} [10] $

Actually, let's align the dimensions. $W_{x h}$ is $h \times d = 3 \times 2$ , so:

$ W_{x h} x_{1} = 0.5 0.2 0.1 0.3 0.4 0.6 [10] = 0.5 0.2 0.1 $

$ W_{hh} h_{0} = 000 $

$ a_{1} = 0.5 0.2 0.1, h_{1} = tanh (a_{1}) = 0.462 0.197 0.100 $

Step $t = 2$ :

$ W_{x h} x_{2} = 0.3 0.4 0.6, W_{hh} h_{1} = 0.046 0.020 0.010 $

$ a_{2} = 0.346 0.420 0.610, h_{2} = tanh (a_{2}) = 0.333 0.397 0.545 $

Step $t = 3$ :

$ W_{x h} x_{3} = 0.8 0.6 0.7, W_{hh} h_{2} = 0.033 0.040 0.055 $

$ a_{3} = 0.833 0.640 0.755, h_{3} = tanh (a_{3}) = 0.682 0.565 0.639 $

Notice: the same $W_{x h}$ and $W_{hh}$ are reused at every step. The hidden state grows richer as it absorbs more history.

Assumptions & Scope.

Scope: RNNs assume the data has sequential/temporal structure where order matters. The hidden state dimension $h$ is fixed and chosen by the designer. The sequence length $T$ is known at training time. Weights are shared across time, so the model cannot allocate more parameters for longer sequences.

When it applies: time series forecasting, speech, text, video, any data with autocorrelation across steps.

When it breaks: if data points are independent (e.g., batch of unrelated images), RNN adds unnecessary complexity. If sequences are very long (hundreds of steps), simple RNNs struggle due to vanishing gradients. This motivates GRU/LSTM (covered in later sections).

Visual Intuition. Unroll the RNN along the time axis. The horizontal axis is time: $t = 1, 2, \dots, T$ . Each column is a copy of the same cell. The vertical axis shows inputs entering from the bottom. A hidden state box sits at center. Outputs emerge at the top. A horizontal arrow connects each hidden state to the next — this is the recurrent edge carrying $W_{hh}$ . Key landmark: every hidden state sends information forward to the next state and upward to the output. The unrolled graph looks like a chain, not a tree. Takeaway: an RNN with $T$ time steps is effectively a $T$ -layer network with tied weights.

Pitfalls.

Treating sequential data as i.i.d. Feeding a shuffled sequence to a feedforward net throws away the order information that holds the signal.
Confusing sequence length with batch size. The sequence length $T$ unfolds the network in time; the batch dimension is orthogonal.
Forgetting that weights are shared. When gradients are computed, a single weight receives contributions from every time step. Summing them is essential.
Using tanh without understanding its range. $tanh$ outputs values in $(- 1, 1)$ . This bounds the hidden state. It also means gradients saturate for large inputs.

Recap + Bridge. RNN = feedforward DNN + a time dimension with shared weights. The hidden state is a compressed memory of the past. Unfolding reveals a deep computational graph where the same parameters serve every position. This sharing is what makes BPTT non-trivial — because one weight matrix gets gradient signals from every time step. That brings us to the two-fold gradient.

Real-World & Domain Connection. RNNs power real-time speech recognition on phones (voice-to-text) and machine translation (Google Translate). They also power stock price forecasting for algorithmic trading. In NLP, the hidden state at the final time step becomes the "meaning vector" for sentence-level sentiment classification. The same mechanism powered early product review analyzers at Amazon and Yelp.

13.1.2 Symbol Registry

The table below lists every symbol used in this section. Keep it handy when tracing BPTT — getting the index direction right is the most common exam mistake.

Symbol	Meaning	LaTeX	Type	Shape Hint
$t$	Time step index	$t$	integer	$1 \leq t \leq T$
$T$	Total sequence length	$T$	integer	e.g., 10 words
$x_{t}$	Input at time $t$	$x_{t}$	vector	$d \times 1$
$h_{t}$	Hidden state at time $t$ (lossy summary of past)	$h_{t}$	vector	$h \times 1$
$a_{t}$	Pre-activation (before tanh) at time $t$	$a_{t}$	vector	$h \times 1$
$o_{t}$	Logits (pre-softmax output) at time $t$	$o_{t}$	vector	$v \times 1$
$y^_{t}$	Predicted output probability distribution	$y^_{t}$	vector	$v \times 1$
$y_{t}$	Ground-truth output at time $t$	$y_{t}$	vector	$v \times 1$
$L_{t}$	Loss at time $t$	$L_{t}$	scalar	cross-entropy
$L$	Total sequence loss	$L$	scalar	$\sum_{t = 1}^{T} L_{t}$
$W_{x h}$	Input-to-hidden weight matrix	$W_{x h}$	matrix	$h \times d$
$W_{hh}$	Hidden-to-hidden (recurrent) weight matrix	$W_{hh}$	matrix	$h \times h$
$W_{h y}$	Hidden-to-output weight matrix	$W_{h y}$	matrix	$v \times h$
$b_{h}$	Hidden bias vector	$b_{h}$	vector	$h \times 1$
$b_{y}$	Output bias vector	$b_{y}$	vector	$v \times 1$
$d$	Input dimension (e.g., embedding size)	$d$	integer	—
$h$	Hidden state dimension (capacity of the "memory")	$h$	integer	—
$v$	Output dimension (e.g., vocabulary size)	$v$	integer	—

13.1.3 Two-Fold Gradient in BPTT

Hook. When you backpropagate through a feedforward net, each weight gets one gradient signal — from the layer above. But in an RNN, the hidden state at time $t$ gets two gradient signals. Where does the second one come from, and what happens if you forget to add it?

Intuition + Analogy. Imagine a relay race. Runner 3 gets direct feedback from her own performance (time split at her leg). But she also gets feedback from runner 4, whose time depends partly on how well runner 3 handed off the baton. The coach gives runner 3 both pieces of feedback: her own split and the downstream effect she had on runner 4. In an RNN, $h_{t}$ is runner 3. The loss at time $t$ is her own split. The loss at time $t + 1$ (and beyond) is the downstream baton effect. Both gradients land on $h_{t}$ and must be added.

Formalize — The Two-Fold Gradient Sum.

At any hidden state $h_{t}$ , the total gradient is the sum of two contributions:

$ \frac{\partial L}{\partial h _{t}} = Path 1: vertical, from current output \frac{\partial L _{t}}{\partial h _{t}} + Path 2: horizontal, from future time steps \frac{\partial L}{\partial h _{t + 1}} \cdot \frac{\partial h _{t + 1}}{\partial h _{t}} $

Path 1 goes through the output layer:

$ \frac{\partial L _{t}}{\partial h _{t}} = W_{h y}^{T} \cdot (y^_{t} - y_{t}) $

(assuming cross-entropy loss with softmax output — the softmax+cross-entropy derivative simplifies to prediction minus target.)

Path 2 goes through the recurrent connection. Since $h_{t + 1} = tanh (W_{hh} h_{t} + W_{x h} x_{t + 1} + b_{h})$ , the Jacobian is:

$ \frac{\partial h _{t + 1}}{\partial h _{t}} = W_{hh}^{T} \cdot diag (1 - (h_{t + 1})^{2}) $

The term $diag (1 - (h_{t + 1})^{2})$ is the derivative of tanh, applied element-wise. The identity $tanh^{'} (z) = 1 - tanh^{2} (z) = 1 - h_{t + 1}^{2}$ holds. Each hidden state element contributes its own derivative on the diagonal.

The full recurrence for the gradient runs backward through time. It goes from $t = T$ down to $t = 1$ :

$ \frac{\partial L}{\partial h _{t}} = W_{h y}^{T} (y^_{t} - y_{t}) + W_{hh}^{T} \cdot diag (1 - h_{t + 1}^{2}) \cdot \frac{\partial L}{\partial h _{t + 1}} $

The BPTT algorithm runs in $O (T)$ time and $O (T)$ memory. It must store every intermediate hidden state from the forward pass.

Worked Example — Scalar Gradient Tracing (Simplified).

Suppose a scalar RNN with no biases and 1D hidden state. Let $w_{hh} = 0.5$ , $w_{x h} = 0.8$ , $w_{h y} = 1.0$ . Sequence: $x_{1} = 1$ , $x_{2} = 2$ , target $y_{1} = 0.3$ , $y_{2} = 0.7$ . Initial $h_{0} = 0$ .

Forward pass:

$t = 1$ : $a_{1} = w_{hh} \cdot 0 + w_{x h} \cdot 1 = 0.8$ . Then $h_{1} = tanh (0.8) = 0.664$ .

$t = 2$ : $a_{2} = w_{hh} \cdot 0.664 + w_{x h} \cdot 2 = 0.332 + 1.6 = 1.932$ . Then $h_{2} = tanh (1.932) = 0.959$ .

Outputs use mean squared error instead of softmax for simplicity. We get $y^_{1} = w_{h y} \cdot h_{1} = 0.664$ . And $y^_{2} = 0.959$ .

Losses: $L_{1} = \frac{1}{2} (0.664 - 0.3)^{2} = 0.066$ , $L_{2} = \frac{1}{2} (0.959 - 0.7)^{2} = 0.034$ . Total $L = 0.100$ .

Backward pass — two-fold gradient at $h_{1}$ :

Path 1 (from $L_{1}$ ): $\frac{\partial L _{1}}{\partial h _{1}} = (y^_{1} - y_{1}) \cdot w_{h y}$ . This evaluates to $(0.664 - 0.3) \cdot 1.0 = 0.364$ .

Path 2 (from $L_{2}$ , through $h_{2}$ ): First, $\frac{\partial L _{2}}{\partial h _{2}} = (y^_{2} - y_{2}) \cdot w_{h y}$ . This works out to $(0.959 - 0.7) \cdot 1.0 = 0.259$ .

Then through the recurrent connection: $\frac{\partial h _{2}}{\partial h _{1}} = w_{hh} \cdot (1 - h_{2}^{2})$ . We compute $0.5 \cdot (1 - 0.920) = 0.5 \cdot 0.080 = 0.040$ .

So Path 2 contribution: $0.259 \cdot 0.040 = 0.010$

Total gradient at $h_{1}$ : $0.364 + 0.010 = 0.374$

The gradient from the future (0.010) is smaller because it passed through tanh, which squashes the signal. This foreshadows the vanishing gradient problem.

Assumptions & Scope.

Scope: The two-fold gradient decomposition holds for any RNN with a loss computed per time step. The tanh derivative formula $diag (1 - h_{t + 1}^{2})$ assumes the activation function is tanh. If a different activation (ReLU, sigmoid) is used, replace the derivative accordingly.

When it applies: During training of simple RNNs, GRUs, and LSTMs, all use the same principle. They sum gradients from current and future losses.

When it breaks: If the loss is computed only at the final time step, you get $L = L_{T}$ . Path 1 then vanishes for all $t < T$ . Only Path 2 contributes. This does not break BPTT — it just simplifies the gradient computation.

Visual Intuition. Draw the unrolled RNN with 3 time steps. For the gradient at $h_{2}$ , draw one arrow coming down from $L_{2}$ (vertical). Draw another arrow coming left from $h_{3}$ (horizontal). At $h_{1}$ , the horizontal arrow starts at $h_{2}$ . It passes through the tanh "bottleneck", multiplied by $1 - h_{2}^{2}$ . The vertical arrow comes from $L_{1}$ . The gradient at the earliest hidden state $h_{1}$ is the dimmest. It has passed through the most tanh bottlenecks. Takeaway: information from $L_{t}$ flows backward like a river that forks. One branch goes up to the output weights. The other goes left to earlier time steps through a narrowing channel.

Pitfalls.

Forgetting to add Path 1 and Path 2. The gradient at $h_{t}$ is the sum, not either alone. Omitting one path gives wrong gradients and prevents learning.
Confusing the direction of recurrence. The recurrent weight multiplies $h_{t - 1}$ in the forward pass. In the backward pass, the gradient flows from $h_{t + 1}$ to $h_{t}$ . The direction is opposite but the weight matrix is the same, only transposed.
Ignoring the tanh derivative. The diagonal Jacobian $diag (1 - h^{2})$ approaches zero when $∣ h ∣$ nears 1. This is the mathematical root of vanishing gradients.
Mixing up transposes. In forward: $W_{hh} h_{t - 1}$ . In backward: $W_{hh}^{T} \cdot \frac{\partial L}{\partial h _{t + 1}}$ . Not transposing gives dimension mismatches.

Recap + Bridge. BPTT is not a new algorithm — it is backprop applied to the unrolled RNN graph. The "through time" part means gradients travel horizontally across time steps via $W_{hh}$ and the tanh Jacobian. The sum-of-two-gradients is the defining feature. This mechanism also explains why gradients vanish: repeated multiplication by $W_{hh}$ and tanh derivatives shrinks the signal. Understanding this two-fold gradient sets up the next section. There we walk through a full numerical BPTT example with matrices.

Real-World & Domain Connection. The two-fold gradient is a special case of the more general backpropagation through structure. This technique is used in recursive neural networks for parsing constituency trees. It also appears in graph neural networks. The same principle — summing gradients from multiple downstream consumers of a node — appears whenever a computation graph has branching paths. In PyTorch and TensorFlow, the autograd engine handles this sum automatically. Still, understanding it is essential for debugging gradient issues in production NLP pipelines.

13.1.4 Sequence Loss and Return Sequences

Hook. When you feed "The movie was surprisingly good" to an RNN, should it output one verdict or one tag per word? The answer changes everything about your architecture.

Intuition + Analogy. Think of reading a paragraph out loud to answer two different questions. Question 1: "What is the main topic?" — you only speak after reading the whole paragraph. Question 2: "Read each word and tell me if it is a noun or verb." — you speak after every single word. The RNN's return_sequences parameter is the switch between these two modes.

Formalize — Sequence-Level Loss.

The total loss over a sequence of length $T$ sums the per-step losses:

$ L = t = 1 \sum T L_{t} = t = 1 \sum T Loss (y^_{t}, y_{t}) $

Each $L_{t}$ uses cross-entropy (classification) or mean squared error (regression) depending on the task. The sum runs over all time steps where a target label exists.

The return_sequences parameter determines which hidden states produce output:

return_sequences=True: $y^_{1}, y^_{2}, \dots, y^_{T}$ are all returned. The output shape is $(T, v)$ .
return_sequences=False: Only $y^_{T}$ (from the final hidden state $h_{T}$ ) is returned. The output shape is $(v,)$ .

Under the hood, return_sequences=False discards $W_{h y} h_{t}$ for all $t < T$ . Only $W_{h y} h_{T}$ reaches the loss function.

Comparison — When to Pick Which.

Criterion	`return_sequences=True`	`return_sequences=False`
Output shape	$(T, v)$ — one prediction per time step	$(v,)$ — one prediction for the whole sequence
Use case	POS tagging, NER, frame-level speech recognition	Sentiment classification, topic classification
Loss	Summed over all $T$ steps	Computed only at the final step
Gradient flow	Every hidden state gets a Path 1 gradient	Only $h_{T}$ gets a Path 1 gradient; earlier states get only Path 2
Downstream	Feeds into another RNN or CRF layer	Feeds into a Dense/softmax classifier
Example	"The/DET cat/NOUN sat/VERB"	"This review is positive"
Memory	Higher — stores all intermediate outputs	Lower — only the final output is kept

Worked Example — Same Sequence, Two Tasks.

Sequence: "I love this movie" (3 words, each embedded as a 4D vector).

Task A — Sentiment (return_sequences=False):

RNN processes all 3 words. Only $h_{3}$ is used.
Output: a 2D vector $y^_{3} = [0.12, 0.88]$ → positive with 88% confidence.
Loss: $L = L_{3}$ only. Intermediate hidden states $h_{1}, h_{2}$ receive gradient only through the recurrent path.

Task B — POS Tagging (return_sequences=True):

RNN outputs at every step: $y^_{1}, y^_{2}, y^_{3}$ , each a 5D vector (5 possible tags).
Output:
"I" → $[0.7, 0.1, 0.1, 0.05, 0.05]$ → PRON
"love" → $[0.1, 0.1, 0.65, 0.1, 0.05]$ → VERB
"this" → $[0.8, 0.05, 0.05, 0.05, 0.05]$ → DET
"movie" → $[0.05, 0.05, 0.1, 0.75, 0.05]$ → NOUN
Loss: $L = L_{1} + L_{2} + L_{3} + L_{4}$ (sum of 4 cross-entropies).

Assumptions & Scope.

Scope: The return_sequences choice applies to SimpleRNN, GRU, and LSTM layers equally. It controls whether the layer returns the full sequence of hidden states (or outputs) or just the last one. When stacking RNN layers, the intermediate layers typically use return_sequences=True. This ensures the next recurrent layer receives a sequence, not a single vector.

When it applies: Every time you use an RNN layer in Keras/PyTorch, you must make this choice.

When it breaks: Using return_sequences=False for a task that needs per-step predictions will silently give wrong results. The model compiles and runs but learns nothing useful. The loss is computed only from the last step.

Pitfalls.

Silent mismatch. Using return_sequences=False for word-level tagging compiles fine but trains on the wrong objective.
Stacking confusion. When you stack two RNN layers, the first must have return_sequences=True. Otherwise the second receives only a single vector instead of a sequence.
Shape errors in downstream layers. A Dense(v) layer expecting input shape (T, h) gets (h,) when return_sequences=False. This causes a dimension mismatch at the first batch.
Forgetting that loss is summed. If you scale the per-step loss without accounting for sequence length, longer sequences dominate the total loss. Consider averaging instead.

Recap + Bridge. The return_sequences flag is the architectural switch between "one answer per sequence" and "one answer per token". This choice determines the loss computation. It also determines the gradient flow. It also determines what downstream layers expect. Picking correctly is the first design decision in any RNN project. This connects directly to the architecture variants (bidirectional, deep RNN, encoder-decoder) covered in later sections.

Real-World & Domain Connection. This decision appears in every production NLP system. Google's BERT uses per-token outputs for question answering to extract answer spans. It uses a single pooled output for sentence classification. Speech recognition systems like Alexa and Siri use per-frame outputs at the acoustic modeling layer. They produce phoneme probabilities every 10ms frame. These are then fed into a decoder that produces the final text output.

13.1.5 Student Questions and Answers

Q: I need a recap of RNN concepts — what are the essentials?

A: Whenever your application has sequential or temporal data, use a recurrent neural network. The reason is autocorrelation among successive time steps. The key difference from a regular DNN is backward propagation through time. Information propagates across time steps. Gradients flow from future time steps back to distant past ones. Understand the need, the architectural differences, and the training requirements. Then all the variants — GRU, LSTM, bidirectional, deep RNN — become easier to grasp.

Q: Why does the RNN use the same weights at every time step instead of learning separate weights?

A: Parameter sharing is the core idea. If each time step had its own weights, the model would need $T \times (params per step)$ parameters. This is impossible for long sequences. Sharing also means the model learns a single pattern that applies at any position in the sequence. For example, the RNN may learn that "not" before an adjective flips sentiment. It can apply that rule at position 2, position 15, or position 50. It uses the same $W_{hh}$ each time. A network without weight sharing would need to re-learn the rule at every position.

Q: How do I know whether return_sequences should be True or False?

A: Ask: does my task need a prediction for every time step, or just for the whole sequence? If the answer is "every word needs a tag" → True. If the answer is "I need one label for the whole sentence" → False. When stacking RNN layers, layers before the last one almost always need return_sequences=True.

Q: What happens to the gradient at $h_{1}$ if the sequence is very long?

A: The gradient at $h_{1}$ must travel backward through the recurrent connection $T - 1$ times. Each trip multiplies by $W_{hh}^{T}$ and the tanh Jacobian $diag (1 - h^{2})$ . If the eigenvalues of $W_{hh}$ are less than 1, the gradient shrinks exponentially (vanishing gradient). If they are greater than 1, it grows exponentially (exploding gradient). This is why simple RNNs struggle with sequences longer than about 10-20 steps, and why GRU/LSTM were invented.

Q: Is the hidden state $h_{t}$ a perfect summary of everything up to time $t$ ?

A: No — it is a lossy summary. The hidden state is a fixed-length vector. All of $x_{1}, \dots, x_{t}$ must be compressed into $h$ numbers. Information from early time steps gets overwritten as new inputs arrive. This is the fundamental bottleneck of simple RNNs. GRU and LSTM add extra memory states (cell state in LSTM, gated update in GRU) to reduce this information loss.

Exam Guidance.

Exam note — Parameter counting. A simple RNN's total parameters: $h (d + h + 1) + v (h + 1)$ . The input dimension is $d$ and hidden is $h$ . The output dimension is $v$ . Know the breakdown: $h d$ comes from $W_{x h}$ . Also $h^{2}$ from $W_{hh}$ . And $h$ from $b_{h}$ . Then $v h$ from $W_{h y}$ and $v$ from $b_{y}$ .

Exam note — BPTT gradient: You must be able to trace the gradient from $L_{t}$ to $h_{t}$ . Also from $h_{t + 1}$ to $h_{t}$ . Then sum them. Show the tanh derivative explicitly as a diagonal matrix. Its entries are $1 - h_{j}^{2}$ for each hidden unit $j$ . Graders look for the presence of both paths.

Exam note — return_sequences: Be precise. False → only $y^_{T}$ . True → $y^_{1}, \dots, y^_{T}$ . The wrong answer here cascades into wrong loss computation and wrong gradient flow.

Exam note — Notation: Use the symbols in the registry table consistently. Mixing $W_{x h}$ and $U$ (the textbook notation) on the same page confuses graders. Pick one convention and stick with it.

13.2 BPTT Numerical Example

Hook: Training a vanilla feedforward network is like grading a stack of unrelated homework — each sheet gets independent feedback. But an RNN is like grading a multi-page essay where each sentence builds on the last. You cannot grade page 2 without knowing what page 1 said. Page 1's "credit" depends partly on whether page 2's argument holds. BPTT is the mechanism that distributes blame across all time steps.

13.2.1 Intuition: The Two-Source Gradient

At any hidden state $h_{t}$ , gradient information arrives from two directions:

1. Vertical (from the output): The loss at time $t$ sends a gradient downward through $W_{h y}$ . This is identical to backprop in a standard feedforward net.

2. Horizontal (from the future): The loss at time $t + 1$ (and $t + 2$ , $t + 3$ , etc.) sends gradient backward through the recurrent weight $W_{hh}$ . It also sends gradient through the tanh nonlinearity.

Analogy: Each runner in a relay race receives two types of feedback. First, her own split time — the vertical gradient from $y^_{t}$ . Second, feedback from downstream runners whose final leg depends on how smoothly she handed off the baton. This is the horizontal gradient from $h_{t + 1}$ . The total adjustment a runner makes considers both.

13.2.2 Formalizing BPTT: The Equations

Forward pass at time step $t$ :

$ h_{t} = tanh (W_{x h} x_{t} + W_{hh} h_{t - 1}) y^_{t} = softmax (W_{h y} h_{t}) $

Backward pass for hidden state $h_{t}$ — two-source gradient:

$ \nabla_{h_{t}} L = vertical: from output loss W_{h y}^{⊤} (y^_{t} - y_{t}) + horizontal: from future time steps W_{hh}^{⊤} (\nabla_{h_{t + 1}} L ⊙ (1 - h_{t + 1}^{2})) $

Where:

$y^_{t} - y_{t}$ is the gradient of cross-entropy loss combined with softmax. Since $y_{t}$ is one-hot encoded, this is simply the predicted probability vector minus the one-hot label vector.
$⊙$ denotes element-wise (Hadamard) multiplication.
$1 - h_{t + 1}^{2}$ is the derivative of $tanh$ , applied element-wise. The full Jacobian of $tanh$ is $diag (1 - h_{t + 1}^{2})$ .
$W_{hh}^{⊤}$ is the transpose of the recurrent weight matrix — needed because gradients flow backward, transposing the forward multiplication.

For the final time step $T$ , there is no future time step, so only the vertical term applies:

$ \nabla_{h_{T}} L = W_{h y}^{⊤} (y^_{T} - y_{T}) $

Gradients for weight matrices are accumulated over all time steps:

$ \frac{\partial L}{\partial W _{h y}} = t = 1 \sum T \frac{\partial L _{t}}{\partial y ^ _{t}} h_{t}^{⊤} $

$ \frac{\partial L}{\partial W _{hh}} = t = 1 \sum T - 1 (\nabla_{h_{t}} L ⊙ (1 - h_{t}^{2})) h_{t - 1}^{⊤} $

$ \frac{\partial L}{\partial W _{x h}} = t = 1 \sum T (\nabla_{h_{t}} L ⊙ (1 - h_{t}^{2})) x_{t}^{⊤} $

Cost: BPTT requires storing every intermediate activation $h_{1}, h_{2}, \dots, h_{T}$ in memory. Both the time and memory complexity are $O (τ)$ , where $τ$ is the sequence length. For long sequences, this becomes the bottleneck. It motivates truncated BPTT and architectures like LSTMs.

13.2.3 Step-by-Step Numerical Trace

This section traces BPTT through a concrete small network. The lecture did not specify the numeric values of the weight matrices. So the illustration focuses on the process and dimensions rather than producing a final scalar loss value.

Setup:

Component	Value
Sequence length $T$	2
Input dimension	2
Hidden dimension	2
Output dimension	2
$W_{x h}$	$2 \times 2$
$W_{hh}$	$2 \times 2$
$W_{h y}$	$2 \times 2$
Biases	All zero
Activation	$tanh$
Loss	Cross-entropy with softmax

Inputs:

$ x_{1} = [10], x_{2} = [01] $

True labels (one-hot):

$ y_{1} = [10], y_{2} = [01] $

---

Step 1: Forward pass at $t = 1$ .

$ h_{1} = tanh (W_{x h} x_{1} + W_{hh} h_{0}) $

Assuming $h_{0} = [0, 0]^{⊤}$ (zero initial state), this reduces to $tanh (W_{x h} x_{1})$ . The $2 \times 2$ matrix $W_{x h}$ multiplies $[1, 0]^{⊤}$ , producing a $2 \times 1$ vector. The $tanh$ squashes each component into $(- 1, 1)$ .

$ y^_{1} = softmax (W_{h y} h_{1}) $

The output $W_{h y} h_{1}$ is a 2-vector. Softmax turns it into a probability vector.

---

Step 2: Forward pass at $t = 2$ .

$ h_{2} = tanh (W_{x h} x_{2} + W_{hh} h_{1}) $

Now both the current input $x_{2} = [0, 1]^{⊤}$ and the previous hidden state $h_{1}$ contribute. The term $W_{hh} h_{1}$ is the memory the network carries from the first time step.

$ y^_{2} = softmax (W_{h y} h_{2}) $

---

Step 3: Backward pass at $t = 2$ (final time step).

Since $t = 2$ is the last step, there is no gradient from any $h_{3}$ . The vertical gradient alone drives the update:

$ \nabla_{h_{2}} L = W_{h y}^{⊤} (y^_{2} - y_{2}) $

Assume the network's softmax output at $t = 2$ was, for illustration, $y^_{2} = [0.3, 0.7]$ . The true label was $y_{2} = [0, 1]$ . Then:

$ y^_{2} - y_{2} = [0.3 0.7] - [01] = [0.3 - 0.3] $

Multiplying by $W_{h y}^{⊤}$ (a $2 \times 2$ matrix) transforms this 2-vector into the 2-vector $\nabla_{h_{2}} L$ .

---

Step 4: Backward pass at $t = 1$ .

Now $h_{1}$ receives gradients from two sources:

$ \nabla_{h_{1}} L = vertical W_{h y}^{⊤} (y^_{1} - y_{1}) + horizontal — through time W_{hh}^{⊤} (\nabla_{h_{2}} L ⊙ (1 - h_{2}^{2})) $

The horizontal term first applies the $tanh$ derivative $(1 - h_{2}^{2})$ element-wise to the gradient from $h_{2}$ . This gates the backward signal. If any element of $h_{2}$ is near $\pm 1$ (saturated $tanh$ ), the gate is near zero, and that gradient pathway is shut. Then $W_{hh}^{⊤}$ routes this gated signal back to $h_{1}$ .

The two terms are added. If the horizontal signal is small, the vertical term dominates. This happens with saturated $tanh$ or small $W_{hh}$ eigenvalues. The network then fails to learn long-range dependencies.

---

Step 5: Accumulate weight gradients.

Each time step contributes its local gradient to the shared weight matrices. For example, $\frac{\partial L}{\partial W _{hh}}$ sums the outer products from $t = 1$ :

$ \frac{\partial L}{\partial W _{hh}} = (\nabla_{h_{1}} L ⊙ (1 - h_{1}^{2})) h_{0}^{⊤} $

Even with only two time steps, the role of the $tanh$ derivative as a gradient gate is visible. Wherever $h_{t}$ saturates, the update to $W_{hh}$ shrinks.

13.2.4 Assumptions and Scope

Scope: This trace assumes a vanilla RNN with $tanh$ activation and zero biases. It uses two time steps and identical dimensions (2) for all layers. In practice:

Weights are initialized randomly (e.g., Xavier/Glorot), not all-zeros.
Biases are trained and add degrees of freedom.
Sequence lengths often exceed hundreds, making BPTT memory-prohibitive.
Truncated BPTT (unrolling only the last $k$ steps) is a common substitute for full BPTT.
The softmax + cross-entropy gradient $y^- y$ is derived assuming the loss is computed independently at each time step (summed over $T$ ). If predictions at some time steps are masked, those terms are omitted.
This example uses a many-to-many RNN layout (output at every time step). Other layouts modify which time steps contribute to the loss. For many-to-one or one-to-many layouts, different vertical gradient terms appear.

13.2.5 Visualizing Gradient Flow Through Time

Picture the unrolled RNN as a horizontal chain of identical feedforward blocks, one per time step. Down each block, the vertical gradient arrow points from the softmax down to the hidden state. Between adjacent blocks, a horizontal arrow carries the $tanh$ -gated gradient backward through the recurrent weight.

At the final block ( $t = T$ ), only the vertical arrow feeds $h_{T}$ . Then at $t = T - 1$ , an arrow from the right arrives. It carries the vertical signal from $t = T$ . That signal has passed through one $W_{hh}^{⊤}$ multiplication and one $(1 - h^{2})$ gate. At $t = T - 2$ , the signal has passed through two such gates and two weight multiplications.

Each step backward through time multiplies the gradient signal by two factors. One is a weight matrix, which may amplify or dampen the signal. The other is a $tanh$ gate, always in $(0, 1]$ . The product of these gates over many steps determines whether the gradient survives or vanishes. When $W_{hh}$ has eigenvalues greater than 1 and the $tanh$ is in its linear regime, the product grows exponentially. That is the exploding gradient. When eigenvalues are less than 1 or the $tanh$ saturates, the product shrinks exponentially. That is the vanishing gradient.

13.2.6 Pitfalls of BPTT

O(τ) Memory: Every hidden state $h_{1} \dots h_{T}$ must be stored for the backward pass. A sequence of 10,000 tokens with 512-dimensional hidden states requires storing ~5 million scalars for one sample. Multiply by batch size and this becomes the dominant memory cost.

Vanishing Gradients: The $tanh$ derivative $1 - h^{2}$ is always in $(0, 1]$ . Multiplied over $T$ steps with $W_{hh}^{⊤}$ whose spectral radius is less than 1, the gradient decays to zero. Early time steps receive negligible updates — they cannot learn long-range patterns.

Exploding Gradients: If the eigenvalues of $W_{hh}$ exceed 1, repeated multiplication causes the gradient norm to blow up toward infinity. Parameter updates become erratic, and training destabilizes. Gradient clipping (capping the norm) is a common mitigation.

Saturation Dead-Zones: When $tanh$ outputs near $\pm 1$ , its derivative nears 0. That time step's contribution to the recurrent gradient is blocked — the network in practice learns nothing from it.

Parallelization Barrier: The forward pass through time is inherently sequential — $h_{t}$ depends on $h_{t - 1}$ . Unlike CNNs or Transformers, BPTT cannot compute all time steps in parallel, limiting training throughput on modern hardware.

Student Q&A:

Q: "Do we have the chain rule as well? The vanishing gradient and exploding gradient problem will still exist in this."

A: Yes, exactly. RNNs inherit the same gradient problems from feedforward deep networks. But the problems are greatly amplified. The gradient flows through time in addition to flowing through layers. Each extra time step adds another matrix multiplication and another $tanh$ gate in the chain. The effective depth in an RNN is the sequence length, which can be hundreds or thousands. This is the core motivation for architectures like LSTM and GRU — discussed in the next section.

13.2.7 Recap and Bridge to LSTMs

BPTT unrolls the RNN through time, turning the recurrence into a deep feedforward network with shared weights. The gradient at each hidden state $h_{t}$ is the sum of two terms. Term (a) is the direct loss contribution from $y^_{t}$ propagated through $W_{h y}$ . Term (b) is the gated, weight-transformed signal from all future hidden states. It propagates through $W_{hh}$ and the $tanh$ Jacobian $diag (1 - h^{2})$ . The softmax + cross-entropy gradient simplifies to $y^- y$ . BPTT costs $O (τ)$ in both time and memory. Repeated multiplication by $W_{hh}$ makes gradient vanishing and exploding exponentially worse than in feedforward nets. The next section introduces LSTMs. They are designed to sidestep these problems by replacing the multiplicative tanh gate with an additive cell state.

BPTT on a 2-step, 2-dimensional RNN might seem straightforward. But scale this same math to a sequence of 500 words with a 256-dimensional hidden state. The memory footprint and gradient instability become real engineering constraints. Every language model trained before 1997 wrestled with exactly this. LSTMs were published that year. Modern sequence models either use LSTMs or GRUs. These change the recurrence to avoid repeated multiplicative gating. Others abandon recurrence entirely in favor of attention via Transformers. Transformers trade the sequential forward-pass bottleneck for quadratic memory in sequence length.

13.3 Output Activation: Softmax vs Sigmoid

13.3.1 The Decision Framework

You are building a classifier. The hidden layers are done. Now you stare at the output layer. One question blocks you: softmax or sigmoid?

Get this wrong, and your network learns nonsense. Probabilities may sum to 2.3 across classes. Or a multi-label detector may insist only one label can ever be active.

Analogy: Restaurant Menu vs. Medical Checklist

Imagine two forms you fill out at a clinic:

- Menu (softmax): You order exactly one main dish. Pick chicken, fish, or vegetarian. Choosing one rules out the others. The probabilities across dishes must sum to 1. This is multi-class classification.

- Checklist (sigmoid): The doctor asks, "Do you have a headache? A fever? A cough?" You answer yes or no to each independently. Having a headache does not prevent also having a fever. Each checkbox is a separate binary decision. This is multi-label classification.

The same distinction drives the output layer. If every input gets exactly one label, use softmax. If each label is an independent yes/no question, use sigmoid — one node per label.

Exam note: The question "softmax or sigmoid?" is never about counting output nodes. It is about whether the classes are mutually exclusive. Two output nodes can serve multi-class (softmax, 2 classes) or multi-label (sigmoid, 2 independent labels). Read the problem statement for guiding phrases. "Assign one tag per word" means multi-class → softmax. "Identify all objects present" means multi-label → sigmoid each.

13.3.2 Formal Definitions and Worked Example

Softmax

For an output layer with $K$ classes, the softmax function takes raw scores. These are a vector of logits $z = [z_{1}, z_{2}, \dots, z_{K}]$ . It produces a probability distribution:

$y^_{i} = \frac{e ^{z_{i}}}{\sum _{j = 1}^{K} e ^{z_{j}}}, i = 1, 2, \dots, K$

Every $y^_{i}$ lies in $(0, 1)$ and $\sum_{i = 1}^{K} y^_{i} = 1$ . The highest-scoring class wins, but every class gets a slice of the probability pie. The denominator forces competition — a spike in one logit suppresses all others.

Sigmoid (Binary)

For a single binary decision, sigmoid squashes one logit into $(0, 1)$ :

$y^= \frac{1}{1 + e ^{- z}}$

For multi-label with $K$ independent labels, apply sigmoid to each of the $K$ logits independently. Each output is a standalone probability. No competition.

Worked POS Tagging Example

Take the sentence: "the cat sat". The task is part-of-speech tagging — assign one tag per word from the set {preposition, noun, verb}. Three classes, mutually exclusive. Softmax.

Suppose the RNN processes "cat" at time step $t = 2$ and produces the logit vector:

$ z = [0.8, 2.1, - 0.3] $

Step through softmax:

Class	Logit $z_{i}$	$e^{z_{i}}$	Probability $y^_{i}$
Preposition	0.8	2.23	2.23 ÷ 11.14 = 0.20
Noun	2.1	8.17	8.17 ÷ 11.14 = 0.73
Verb	-0.3	0.74	0.74 ÷ 11.14 = 0.07

Sum = 11.14 | Sum of probabilities = 1.00

The model is 73% confident that "cat" is a noun. The ground truth label for "cat" is $y = [0, 1, 0]$ (one-hot). The cross-entropy loss for this time step is:

$ L_{2} = - i = 1 \sum 3 y_{i} ln (y^_{i}) = - ln (0.73) \approx 0.31 $

Contrast with sigmoid: if you used three independent sigmoids for this task, the three outputs could sum to 1.4 or 0.6. The network would never learn the exclusivity constraint. It could output 0.9 for both "preposition" and "noun" simultaneously, which is wrong for POS tagging.

13.3.3 Scope, Pitfalls, and Student Q&A

When Softmax

- Multi-class classification with $K \geq 2$ mutually exclusive classes.

- Examples: POS tagging ( $K$ = number of tags), digit recognition (0–9, $K = 10$ ), language identification.

- Loss partner: categorical cross-entropy.

When Sigmoid

- Single binary classification (one output node).

- Multi-label classification (multiple independent binary outputs, one sigmoid per label).

- Examples: disease diagnosis (multiple conditions can co-occur), object detection (multiple objects in one image), sentiment with multiple emotions.

- Loss partner: binary cross-entropy applied independently to each node.

The 2-Class Edge Case

A 2-class problem is the most confusing. Both work:

Approach	Output Nodes	Activation	When to Use
Binary	1	Sigmoid	Classes are `0` vs `1`; single probability of class 1
Multi-class	2	Softmax	Two mutually exclusive categories; two probabilities summing to 1

Both are valid. The context decides. If the labels are one-hot encoded as [1,0] / [0,1], the problem is framed as multi-class → softmax. If labels are 0 and 1, binary → sigmoid.

Pitfalls

1. Softmax for multi-label: Probabilities are forced to sum to 1. The model cannot assign high probability to two labels simultaneously. Use sigmoid instead.

2. Sigmoid for multi-class: Nothing enforces exclusivity. The model may output [0.9, 0.8, 0.1] for a single-class task. Use softmax to inject competition.

3. Overthinking 2 outputs: Two output nodes do not automatically mean softmax. Ask: are the two labels mutually exclusive or independent?

4. Confusing classification with regression: Two output values could be continuous predictions (e.g., predicting price and weight). No activation (linear) or a bounded activation like ReLU may apply. Read the task description.

Visual Comparison

Property	Softmax	Sigmoid
Output range	(0, 1) each	(0, 1)
Sum constraint	Forces sum = 1	No constraint
Competition	Yes — high score suppresses others	No — each node is independent
Use case	Exactly one label per input	Zero, one, or many labels
Typical loss	Categorical cross-entropy	Binary cross-entropy (per node)
$K = 1$	Nonsensical	Valid binary classifier

Recap

- Softmax = mutually exclusive classes, probabilities sum to 1, multi-class.

- Sigmoid = independent binary decisions, no sum constraint, binary or multi-label.

- The number of output nodes does not dictate the activation; the relationship between classes does.

- In RNNs for sequence tagging (like POS), each time step is a multi-class decision → softmax at every time step.

Real-World

- POS tagging (NLP): Every word gets one tag. Softmax output layer with $K$ = vocabulary of tags.

- Named Entity Recognition (NER): Each token gets one entity label (person, location, organization, or none). Softmax.

- Medical coding: A patient record can have multiple diagnosis codes simultaneously. Sigmoid per code (multi-label).

- Image tagging: An image can contain "cat" AND "sofa" AND "plant". Sigmoid per tag.

- Sentiment analysis: If you allow only one sentiment (positive/negative/neutral), use softmax. If you allow mixed emotions (happy + surprised), use sigmoid.

Student Q&A

Q: How do I decide between softmax and sigmoid when I see two output nodes in an exam question?

A: Look at the labels. If the labels are one-hot pairs like [1,0] or [0,1], each sample has exactly one active class. Those two classes are mutually exclusive. Use softmax. If the labels look like [0,1], [1,0], or [1,1], both can be active together. The two nodes answer independent yes/no questions. Use sigmoid on each. The task description will always clarify whether classes compete or coexist.

Q: Could two output values be continuous rather than class probabilities?

A: Yes. If the task is regression, the output layer may use no activation. That is the linear case. Or it may use a bounded activation like ReLU. Two output values could represent predicted temperature and humidity, for example. Classification always involves discrete labels. Regression always involves continuous targets. The problem statement will tell you.

Q: Why not always use softmax for everything?

A: Because the sum-to-1 constraint is wrong for multi-label tasks. If an image contains both a cat and a dog, softmax forces a trade-off. It raises the cat probability only by lowering the dog probability. Sigmoid lets both be high. Forcing softmax on a multi-label problem degrades accuracy. The model structurally cannot express co-occurrence.

Exam note: The most common trap is seeing two output nodes and assuming sigmoid. Binary classification with $K = 2$ classes works with either approach. Softmax uses 2 nodes for mutually exclusive classes. Sigmoid uses 1 node for the probability of class 1. Both are correct when matched to the label format. The key question: Can multiple labels be true at the same time? If yes → sigmoid (multi-label). If no → softmax (multi-class).

13.4 Vanishing and Exploding Gradients in RNN

Hook: You train an RNN to classify a 20-word sentence. The word at position 1 sets the tone for the whole meaning. By the time the loss from position 20 backpropagates to position 1, does the gradient still carry any useful signal? In most vanilla RNNs, the answer is no — the gradient has shrunk to near zero. The network simply cannot learn that position 1 matters for position 20.

13.4.1 Revisiting the Gradient Problem

In a feedforward network, the gradient travels backward through layers — depth $D$ . In an RNN, the gradient travels backward through time as an extra channel — sequence length $T$ . The effective depth is $T$ (or $T \times L$ if you stack $L$ layers). A 20-step single-layer RNN has effective depth 20. A 50-step, 4-stacked-layer RNN has effective depth 200. Each step backward multiplies the gradient by $W_{hh}^{⊤}$ and a tanh Jacobian term.

From section 13.1, the horizontal gradient at time $t$ is:

$\nabla_{h_{t}}^{(horiz)} L = W_{hh}^{⊤} ((1 - h_{t + 1}^{2}) ⊙ \nabla_{h_{t + 1}} L)$

Unroll this recurrence over $k$ steps from time $T$ back to $T - k$ . The gradient is multiplied $k$ times by $W_{hh}^{⊤}$ . It is also gated $k$ times by the tanh derivative. In simplified form (suppressing the element-wise tanh gating):

$\nabla_{h_{T - k}} L \approx (W_{hh}^{⊤})^{k} \cdot \nabla_{h_{T}} L$

Analogy — The Whisper Game (Telephone): Twenty people sit in a line. Person 1 whispers a sentence to person 2, who whispers to person 3, and so on. By the time person 20 speaks the sentence out loud, it bears no resemblance to the original. Each retransmission introduces a small distortion. In an RNN, each time step is one "retransmission." The gradient is the sentence. The distortion comes from multiplying by $W_{hh}^{⊤}$ . If the per-step distortion factor is less than 1, the signal vanishes. If greater than 1, it explodes into noise.

Why Eigenvalues Drive the Behavior:

Repeated multiplication $(W_{hh}^{⊤})^{k}$ is governed by the eigenvalues of $W_{hh}$ . Let $λ_{m a x}$ be the spectral radius (largest absolute eigenvalue):

Condition	Behavior	Gradient Over $k$ Steps
$λ_{m a x} < 1$	Vanishing	Magnitude $\propto (λ_{m a x})^{k} \to 0$
$λ_{m a x} = 1$	Stable (fragile)	Magnitude stays roughly constant
$λ_{m a x} > 1$	Exploding	Magnitude $\propto (λ_{m a x})^{k} \to \infty$

The tanh derivative $1 - h^{2}$ is always in $(0, 1]$ . It acts as an extra shrinking factor at every step — it can only make vanishing worse, never better. When the hidden state saturates near $\pm 1$ , the tanh gate is near 0, and that gradient pathway closes entirely.

In practice, $λ_{m a x} < 1$ is the common case after training begins. Weights tend to shrink. So vanishing gradients are the default problem. Exploding gradients happen early in training if initial weights push eigenvalues above 1. But they are easier to detect because the loss hits NaN. They are also easier to fix with gradient clipping.

Worked Example: Gradient Decay Over 20 Steps

Setup: Single-layer RNN, hidden dimension 128, 20-step sequence. After some training, $W_{hh}$ has spectral radius $λ_{m a x} \approx 0.85$ . The average tanh derivative across hidden units is $\sim 0.5$ . This is a realistic middle case — neither fully saturated nor fully in the linear regime.

Effective decay per time step backward:

$ decay per step \approx λ_{m a x} \times (avg tanh gate) = 0.85 \times 0.5 = 0.425 $

Trace the gradient from step 20 back to step 1:

Steps Back ( $k$ )	Target Time Step	Multiplier $(0.425)^{k}$	Signal Status
0	$t = 20$	$1.0$	Full signal (final loss)
5	$t = 15$	$0.42 5^{5} \approx 1.4 \times 1 0^{- 2}$	~1.4% remains — diminishing but usable
10	$t = 10$	$0.42 5^{10} \approx 1.9 \times 1 0^{- 4}$	~0.02% remains — barely any signal
15	$t = 5$	$0.42 5^{15} \approx 2.7 \times 1 0^{- 6}$	Negligible
20	$t = 1$	$0.42 5^{20} \approx 3.7 \times 1 0^{- 8}$	Effectively zero

By step 1, the gradient has decayed by a factor of nearly $1 0^{- 8}$ . The weights $W_{x h}$ and $W_{hh}$ receive almost no signal from the final loss to update how they handle the first input. The network cannot learn that $x_{1}$ was important for the output at $y_{20}$ .

Exploding counterexample: If $λ_{m a x} = 1.5$ and the tanh gate is near 1 (linear regime), then $1. 5^{20} \approx 3325$ . The gradient at $t = 1$ is thousands of times larger than at $t = 20$ , and parameter updates become erratic. But this requires both large eigenvalues and unsaturated tanh — vanishing is far more common.

13.4.2 Long-Term Dependencies Lost

Visual: Gradient Magnitude vs Time Step

Picture a plot with 20 time steps on the x-axis (running left to right, from $t = 1$ to $t = 20$ ). Gradient magnitude is on the y-axis (log scale). The gradient from the loss at $t = 20$ propagates leftward:

- Vanishing curve: Starts at full height at $t = 20$ , then drops sharply in a near-perfect exponential decay. By $t = 15$ , the curve is below 2% of the original. By $t = 10$ , it hugs the x-axis, indistinguishable from zero. The first 5–10 time steps receive effectively no gradient.

- Exploding curve: Starts at $t = 20$ and climbs steeply leftward. At $t = 1$ , it is orders of magnitude above the right side. The gradient at early steps dominates the loss, making training unstable.

Neither curve is usable for learning relationships that span the full sequence. The network has a gradient horizon — only the most recent ~5 to 10 time steps send meaningful gradient signals. Anything beyond that is forgotten.

Scope: Where This Hurts

Applications where input-output relationships span many time steps are most affected:

Sentiment analysis on long reviews: A word at the start may say "masterpiece". A twist at the end may say "complete disappointment". The RNN must connect position 1 to the final label. That spans dozens of steps. A vanilla RNN cannot.
Machine translation: In German, the main verb often appears at position 2, but the English translation places it at the end. The gradient must cross 15+ time steps to tune the encoding of that verb.
Time series forecasting with long seasonality: A pattern repeats every 100 observations. The gradient from step 100 must influence how step 1's pattern is encoded. Vanilla RNNs miss it.
Speech recognition: A phoneme spoken at second 1 can change the interpretation of a word at second 10.

In all these cases, a vanilla RNN with a gradient horizon of ~10 steps fails to capture the dependency.

Pitfalls

Stacking layers multiplies the problem. The effective depth is $T \times L$ (time steps × stacked layers). A 50-step sequence with 4 hidden layers forces the gradient through ~200 multiplicative steps. At that scale, even modest per-step decay makes the gradient vanish. The earliest time step of the bottom layer hits exactly zero in floating-point arithmetic. In practice, stacking more than about 4 hidden layers in a vanilla RNN guarantees complete gradient vanishing. This is called diminishing returns. Adding more layers stops helping because nothing reaches the early layers to adjust them.

Longer sequences mean more forgetting, not more learning. Adding time steps beyond the gradient horizon only dilutes the gradient. It adds no useful signal for the earliest steps.

Weight initialization cannot solve this. Careful initialization (Xavier, orthogonal) can set $λ_{m a x} \approx 1$ at the start. But as training proceeds and weights shift, the spectral radius drifts. Maintaining $λ_{m a x} \approx 1$ throughout training for a vanilla RNN is practically impossible.

Truncated BPTT is a workaround, not a solution. Unrolling only the last $k$ steps (e.g., $k = 20$ ) saves memory and limits decay. But it also means the network literally cannot learn dependencies longer than $k$ steps. Those earlier steps are never included in the backward pass.

Recap: Vanishing and exploding gradients are worse in RNNs because the recurrence multiplies the gradient by $W_{hh}^{⊤}$ at every time step. The effective depth equals the sequence length. With $λ_{m a x} < 1$ and a tanh gate less than 1, the gradient decays exponentially — typically vanishing within ~10 steps. Long-term dependencies cannot be learned. Stacking more layers compounds the issue.

This is the central motivation for LSTM and GRU. Instead of repeatedly multiplying the gradient by a weight matrix, these architectures introduce an additive update path. The cell state in LSTM and the update gate in GRU let the gradient flow backward through time without repeated multiplicative decay. Sections 13.11 through 13.14 develop this solution.

Real-World

Before 2017, virtually all production NLP systems used LSTMs, not vanilla RNNs. Vanilla RNNs could not handle the 50–200 token sequences typical of sentences. Google's Neural Machine Translation system (GNMT, 2016) used 8 stacked LSTM layers with residual connections. The skip connections were explicitly added to fight gradient vanishing across those 8 layers. Without them, the bottom layers received zero gradient and contributed nothing to the translation.

Today, Transformers have largely replaced RNNs for NLP. But the analysis remains identical. Deeper networks and longer sequences demand architectures that avoid repeated multiplicative gradient flow. The LSTM/GRU insight was born directly from studying the vanishing gradient problem described here. It is to replace repeated multiplication with additive updates. This remains one of the most important architectural innovations in deep learning history.

13.5 Memory Metaphor for Specialized RNN Architectures

13.5.1 Human Memory as an Analogy

Close your eyes and try to recall everything you ate for lunch three weeks ago. Nothing? Now try to remember your best friend's name. Instantly there. Your brain does not treat all memories equally. It keeps some ready at the surface. It archives others deep in the back. This is not a bug. It is the most efficient design nature ever stumbled upon. It is exactly what specialized RNNs imitate.

Imagine you attend scattered class sessions across a semester — session 1, session 3, session 6, session 9, session 12. Between sessions, life happens. You forget details. When session 9 begins and the professor asks a question from session 1, you fumble. "I remember the topic," you think, "but I cannot reconstruct the exact answer."

Now imagine you keep a diary. After every session, you write down everything — every formula, every diagram, every intuition. When exam week arrives, you do not need to rely on your fragile brain. The diary has it all. You read it. You refresh.

But notice something subtle: the diary alone is not enough. If you had to flip through a thousand pages mid-exam to answer a quick question, you would fail. You need a working surface — a scratchpad — that holds the currently relevant piece of the diary right now. That scratchpad is your short-term memory.

So you have three things interacting. First, the raw input arriving right now — the professor's question. Second, your scratchpad that holds what you just processed — working memory. Third, your diary sitting on the shelf holding everything you have ever written — long-term storage. The scratchpad decides what to throw away from the diary. It also decides what new things to write into it. This three-way dance is the core metaphor.

Why does this matter? A plain RNN has only a scratchpad. It overwrites it at every step. If you try to remember session 1 during session 9, the plain RNN has already scribbled over it. Sessions 2, 3, 4 — gone. This is the vanishing gradient, dressed in human terms. The diary solves this: you need a stable conveyor belt of memory that glides untouched through time. Write to it occasionally. Read from it when needed. Never let the mere passage of time erase it. That conveyor belt is the cell state.

13.5.2 Cell State and Hidden State

LSTM splits memory into two tracks:

- The diary — cell state $C_{t}$ . This is the long-term conveyor belt. It runs horizontally across the entire sequence, from the first token to the last. Nothing directly overwrites it. Information is added or removed only through carefully gated operations (explored in later sections). A value placed on this belt at time step 1 can travel untouched all the way to time step 100. It is the network's institutional knowledge.

- The scratchpad — hidden state $h_{t}$ . This is the working memory, computed fresh at every time step. It is a filtered, focused extract of the cell state, mixed with the current input $x_{t}$ . It answers a specific question. Given everything I know from the diary and what I am seeing right now, what is the most useful summary? What can I hand to the next time step? It drives predictions, generates the output, and controls reading from and writing to the cell state.

The relationship is asymmetric. $h_{t}$ depends on $C_{t}$ — the scratchpad is a filtered view of the diary. But $C_{t}$ does not depend on $h_{t}$ for its survival. The belt rolls forward independently. This asymmetry is the whole point.

Scope. This two-track design answers a simple question. How do we let a neural network remember something from 200 steps ago without 199 nonlinear squashing operations? The answer: give it a highway that bypasses the nonlinearities. The cell state is that highway. Every specialized RNN — LSTM, GRU, Peephole LSTM, and variants — is a different answer to the same question. That question is: how should we manage the reading and writing on this highway?

Visual. Picture two horizontal rails running left to right across time:

Time:      t=1      t=2      t=3      t=4      ...      t=T

Cell      [C₁] --- [C₂] --- [C₃] --- [C₄] --- ... --- [C_T]
state     (diary — stable, linear highway, rarely overwritten)

Hidden    [h₁] --- [h₂] --- [h₃] --- [h₄] --- ... --- [h_T]
state     (scratchpad — recomputed every step, volatile)

Input:     x₁       x₂       x₃       x₄                x_T

The top rail (cell state) flows with minimal resistance — mostly linear operations, no squashing. The bottom rail (hidden state) is noisy, nonlinear, reactive. It recomputes itself at every step based on the current input and the top rail's content. Arrows go from input up to the cell state for writing. Arrows go from cell state down to the hidden state for reading. Arrows go from hidden state to the output for predicting. The cell state has a self-loop that lets it carry identity forward. This is the secret sauce that defeats vanishing gradients.

At any given moment, the hidden state is like a spotlight scanning a long shelf of books. It illuminates only the page that matters right now. The shelf — the cell state — preserves everything else in the dark. Each page waits to be illuminated when its moment comes.

Pitfalls. A common misunderstanding is that the cell state is "memory" and the hidden state is "output." This is too coarse. Both are memory; they differ in durability. The cell state is persistent memory (write rarely, read when needed). The hidden state is transient memory (rewritten every step and immediately consumed). Another pitfall: thinking the cell state is unbounded and magical. It is not — it is a fixed-size vector. It must learn what to keep and what to discard. If the sequence is too long, even the cell state fills up and must overwrite older entries. The diary has finite pages.

A third pitfall: assuming GRU uses exactly the same two-track design. It does not. GRU merges both tracks into a single hidden state and controls memory through reset and update gates. It is a lighter, more compact design — closer to a sticky note than a diary-plus-scratchpad system. The GRU's hidden state tries to be both persistent and reactive at once.

Recap. RNNs fail because they have only one track — the hidden state — and it gets noise-corrupted at every step. LSTMs add a second track, the cell state, which is a stable memory highway insulated from that per-step noise. This dual-track design lets the network carry signals over arbitrarily long gaps. The cell state preserves. The hidden state predicts. Sections 13.6 through 13.8 will unpack how LSTMs manage this two-track system with forget, input, and output gates. They will also show how GRUs achieve a similar effect with a leaner, single-track design.

Real-World. Every time your phone's keyboard guesses the next word after a long sentence, an LSTM cell state carries the context. It carries the opening clause to the cursor. Every time a voice assistant parses "remind me to buy milk when I get home," the LSTM cell state preserves the dependency. It preserves it across words. It understands that "when I get home" modifies the reminder condition, not the milk. The two-track memory metaphor is not just a teaching trick. It is architecturally present in models that millions of people use every day.

13.6 Bidirectional RNN

13.6.1 Why Bidirectional Context Matters

Imagine reading this sentence: "the cat spoke."

When you first encounter the word "cat", you are unsure. Is it an animal? Or a nickname for Caitlin? You keep reading. You see "spoke." Now you know — it is probably a person. The future word changed how you understand the past word.

A normal RNN reads only left to right. It sees "cat" first. It has no idea that "spoke" is coming. It cannot revise its understanding. The damage is already done.

What if you read the sentence backwards too? You would see "spoke" first. That context would flow backward to "cat." The ambiguity dissolves. This is the core idea behind a bidirectional RNN.

Exam note: Bidirectional RNNs solve the "cat spoke" ambiguity problem. A unidirectional RNN cannot use future context to disambiguate a word. Bidirectional RNNs resolve this by processing the sequence in both directions.

13.6.2 Architecture and Formal Definition

Think of two readers sitting side by side over the same sentence. One reads from start to end. The other reads from end to start. Each takes notes after every word. At the end, they compare notes. Each word position now has two sets of notes — one from forward reading, one from backward reading. These are combined into a single understanding.

This is exactly how a bidirectional RNN works. It runs two separate RNNs in parallel on the same sequence. One sweeps forward. The other sweeps backward. Their hidden states are concatenated at every time step.

Forward RNN: processes $x_{1} \to x_{2} \to \dots \to x_{T}$ , producing hidden states $h_{1}, h_{2}, \dots, h_{T}$

Backward RNN: processes $x_{T} \to x_{T - 1} \to \dots \to x_{1}$ , producing hidden states $h_{1}, h_{2}, \dots, h_{T}$

At each time step $t$ , the two hidden states are concatenated:

$h_{t}^{bi} = [h_{t}; h_{t}]$

The notation is simple: $h$ means forward, $h$ means backward. All formulas and calculations inside each RNN stay identical to the standard RNN. There are just two copies.

What this looks like: Picture two parallel RNN cells sweeping in opposite directions across a sentence. The forward RNN starts at the first word and moves right. The backward RNN starts at the last word and moves left. At the middle word, both meet. Their hidden states at that position are then joined together. The result is a hidden representation that knows the full left and right context around every word.

13.6.3 Worked Parameter Count

Since a bidirectional RNN is two independent RNNs plus an output layer, the total parameter count is simple. Let us work through a concrete example.

Setup: input dimension $d = 2$ , hidden dimension $h = 2$ .

For a single RNN, the hidden state update needs three weight matrices and one bias vector:

- Input-to-hidden weights: $d \times h = 2 \times 2 = 4$ parameters

- Hidden-to-hidden weights: $h \times h = 2 \times 2 = 4$ parameters

- Bias: $h = 2$ parameters

So one RNN has $4 + 4 + 2 = 10$ trainable parameters.

A bidirectional RNN has two such RNNs: $10 + 10 = 20$ parameters before the output layer.

For the output layer, the concatenated hidden vector has size $2 h = 4$ . If we do a simple binary classification at each time step ( $o = 1$ ), the output layer contributes:

- Weights: $(2 h) \times o = 4 \times 1 = 4$ parameters

- Bias: $o = 1$ parameter

Total bidirectional RNN parameters: $20 + 4 + 1 = 25$ .

Compare this to a single unidirectional RNN with the same output layer. It costs $h \times o + o = 2 + 1 = 3$ . Total: $10 + 3 = 13$ .

The bidirectional version uses roughly double the parameters. This is the memory cost of looking both ways.

Exam note: The number of parameters in a bidirectional RNN is about $2 (d h + h^{2} + h)$ . Add the output layer cost of $(2 h) o + o$ . Double the hidden weights means double the computation.

When this matters: The entire sequence must be fully available before processing can begin. You cannot start the backward pass until you have read the last input. This makes bidirectional RNNs unsuitable for streaming or real-time applications. Data arrives one step at a time in those settings. They are designed for offline, batch-style processing of complete sequences.

13.6.4 When to Use, When to Avoid

Use a bidirectional RNN	Do not use a bidirectional RNN
The full sequence is available before prediction	Data arrives as a real-time stream
You have a document, webpage, or textbook to process	You need a prediction after every incoming token
Your task is tagging each position in a fixed-length input	Latency matters more than accuracy
Speech recognition on a recorded utterance	Real-time text output during a conversation
Handwriting recognition on a scanned page	Auto-complete while the user is still typing

Pitfall — doubled computation: Bidirectional RNNs double both the parameter count and the training time. Every forward pass runs two RNNs. Every backward pass for backpropagation also runs through two RNNs. If your sequence is already short or your accuracy is already high, adding bidirectionality may not be worth the extra cost.

Pitfall — streaming impossibility: The backward RNN starts from the last time step. So you must wait for the entire sequence to arrive. You cannot predict anything until both directions have completed. In a live system, this means unacceptable latency. For real-time use cases, stick with unidirectional RNNs and consider using attention mechanisms later.

Real-world applications where bidirectional RNNs excel:

- Speech recognition: A recorded audio clip is fully available. The biRNN sees left context (previous phonemes) and right context (upcoming phonemes) around every frame. This dramatically improves phoneme disambiguation.

- Handwriting recognition: A scanned page of cursive writing is a static image. Processing stroke sequences in both directions helps resolve ambiguous letter boundaries.

- Part-of-speech tagging: Given a fixed corpus of text, tag every word with its grammatical role. Words before and after the target word inform the tag decision.

- Named entity recognition: Is "Apple" a fruit or a company? The words "announced" and "stock" to the right give you the answer. Bidirectional context is essential.

A bidirectional RNN provides context, not action. It is an encoder. It compresses the full input into context-rich hidden states. To generate text, classify sequences, or build a search engine, you need an extra network (a decoder or classifier) on top. The next section introduces the encoder-decoder pattern that pairs bidirectional encoders with unidirectional decoders.

13.6.5 Student Questions and Answers

Q: "In what cases would we have all the sequence provided to us before prediction?"

A: Whenever you are working with a static dataset. Think of a textbook, a webpage, a saved email, or a recorded audio file. The full text or audio is already stored on disk. You read it in, process it with a biRNN, and tag each word or phoneme. This is the standard setup for NLP tasks like POS tagging on a given corpus.

Q: "Can we use a bidirectional RNN for summarization?"

A: Not by itself. Summarization is a sequence-to-sequence problem. The input length (e.g., 100 words) differs from the output length (e.g., 20 words). A bidirectional RNN alone cannot shrink or expand the sequence length. What it can do is act as an encoder. It reads the entire input from both directions and compresses the full context into a dense hidden state. That state is then passed to a decoder RNN, which generates the summary one word at a time. The bidirectional part handles understanding. The decoder handles generation. Together they form an encoder-decoder architecture — the topic of the next lecture.

Q: "Wouldn't a bidirectional RNN help summarize because it captures essence from both directions?"

A: Absolutely. The bidirectional RNN excels at this. For example, it processes forward and backward through a document. The hidden state at the first time step now carries context from both ends of the text. It holds a compressed, context-aware representation of the entire document. This is powerful for downstream tasks. But the bidirectional RNN only encodes — actual classification, prediction, or generation requires another network downstream. The encoder provides the rich representation. The decoder or classifier uses it.

Exam note: The bidirectional RNN is always an encoder, never a standalone decoder or classifier. If a question asks about tasks requiring output of variable length, consider summarization, translation, or captioning. The bidirectional RNN can only serve as the encoder half of an encoder-decoder pair.

13.6.6 Recap and Bridge

A bidirectional RNN runs two RNNs in opposite directions. Their hidden states are concatenated at every time step. This gives each position access to the full left and right context. The trade-off is roughly double the parameters and computation. The reward is significantly better accuracy on tasks where future context matters.

The bidirectional RNN is purely an encoder. It reads the sequence and produces context-aware representations. But many real-world tasks need to generate output — a summary, a translation, a caption. That requires a decoder.

This sets up the encoder-decoder architecture, which we study next. The encoder (bidirectional RNN) reads the input. The decoder (unidirectional RNN) generates the output step by step. Together they solve sequence-to-sequence problems.

13.7 Python Implementation Notes

13.7.1 SimpleRNN with Return Sequences

Hook. You have built an RNN. It runs over a sequence. But when you call it, does it give you just the final answer? Or an answer at every step? The difference is one parameter: return_sequences.

Analogy. Think of watching a movie. When return_sequences=False, you ask "Did you like the film?" — you only get the final verdict. When return_sequences=True, you ask "What were you feeling at each scene?" — you get a reaction after every moment. The same RNN watches the same sequence. Only the output behavior changes.

Formalize. In Python, a typical SimpleRNN stack looks like this:

model = Sequential([
    Embedding(vocab_size, 128),          # convert words to vectors
    SimpleRNN(64, return_sequences=False),  # RNN layer
    Dense(num_classes, activation='softmax')   # output
])

The embedding layer turns each word token into a dense vector of dimension 128. The SimpleRNN layer has 64 hidden units. It reads the embedded sequence and produces output. If return_sequences=False (the default), the RNN gives output only after the last time step. If return_sequences=True, the RNN produces an output vector at every time step — one per input word.

Worked Example. Consider a sentence of 5 words: "The movie was very good."

- With SimpleRNN(64, return_sequences=False): output shape is (batch_size, 64). You get one 64-dimensional vector summarizing the whole sentence. Feed that into a single Dense(2, activation='softmax') for a Sentiment: Positive or Negative.

- With SimpleRNN(64, return_sequences=True): output shape is (batch_size, 5, 64). You get five 64-dimensional vectors. Each corresponds to one word's context after seeing everything before it. Feed each through TimeDistributed(Dense(num_tags, activation='softmax')) to tag every word as Noun, Verb, Adjective, etc.

- With Bidirectional(SimpleRNN(64, return_sequences=True)): output shape is (batch_size, 5, 128). The 128 comes from concatenating the 64 forward units and 64 backward units. Each word now gets forward and backward context.

Scope. Use return_sequences=False for many-to-one tasks: sentiment classification, spam detection, document categorization. Use return_sequences=True for many-to-many tasks: POS tagging, NER, speech frame labeling. If you stack multiple RNN layers, all but the last must use return_sequences=True so the next layer receives a sequence.

Pitfalls. A common mistake: using return_sequences=True on the last RNN layer when you intend to do sequence classification. The Dense layer will receive a 3D tensor, causing a shape error. Another mistake: using return_sequences=False on an intermediate layer when stacking RNNs. The next RNN expects a sequence but gets a single vector.

Recap. One parameter controls output shape. False gives the ending; True gives the full movie.

Real-World. Sentiment classification of movie reviews uses return_sequences=False. POS tagging on a textbook corpus uses return_sequences=True.

13.7.2 Bidirectional Wrapper

Hook. An RNN reading left-to-right only knows the past. But sometimes the future also matters. How do you give an RNN awareness of what comes next?

Analogy. Imagine two proofreaders. One reads the document front to back. The other reads it back to front. Then they sit together and compare notes. Each word now has context from both sides. That is the bidirectional wrapper.

Formalize. The bidirectional wrapper is an inbuilt function that takes any RNN layer and runs it in both directions:

from tensorflow.keras.layers import Bidirectional, SimpleRNN

model = Sequential([
    Embedding(vocab_size, 128),
    Bidirectional(SimpleRNN(64, return_sequences=True)),
    TimeDistributed(Dense(num_tags, activation='softmax'))
])

What happens internally: two independent copies of SimpleRNN(64) are created. One processes the sequence from left to right. The other processes the same sequence from right to left. At each time step, their hidden states are concatenated. So 64 becomes 128 per time step. The weights are not shared. Forward and backward RNNs learn their own parameters.

Worked Example. Start with SimpleRNN(64). Its parameter count for input dimension d and hidden dimension h=64 is:

d × 64 + 64 × 64 + 64  = 64d + 4160

When you wrap it: Bidirectional(SimpleRNN(64)). The parameter count doubles:

2 × (64d + 4160) = 128d + 8320

The output layer now receives 2h = 128 dimensions per time step. The number of parameters in the output layer also doubles because its input size doubles.

Scope. Use bidirectional RNNs when the full sequence is available before prediction. NLP tasks on static corpora (POS tagging, NER on documents) are perfect candidates. Do not use bidirectional RNNs for streaming data (voice assistants, real-time output, stock tickers). You would be forced to wait for the entire sequence before the backward pass can begin.

Pitfalls. Students often miss that Bidirectional wraps around any RNN type: SimpleRNN, LSTM, GRU. The wrapper does not know or care about the inner mechanics. It just creates two copies. Also note that if you stack layers and wrap each one with Bidirectional, the parameter count multiplies further.

Recap. Bidirectional(SimpleRNN(64)) creates two RNNs, processes the sequence in opposite directions, and concatenates their outputs. Double the computation, double the parameters, but richer per-word context.

Real-World. Modern NLP pipelines for POS tagging use Bidirectional(LSTM(128)) as the standard encoder. Named Entity Recognition systems rely on bidirectional context to disambiguate words like "Apple" (company versus fruit).

13.7.3 Data Splitting Caution

Hook. You split your dataset into training and test sets. In nearly every machine learning lecture, you are told: shuffle the data. Randomize the rows. It prevents ordering bias. But here, that advice is dangerous.

Analogy. Imagine a novel. You tear out all the pages, shuffle them, and hand them to someone. Can they understand the story? No. The meaning lives in the sequence. The same is true for RNN data.

Formalize. Data may consist of independent, complete sequences — sentences in a corpus, separate audio clips, or individual video clips. For such data, shuffling at the instance level is fine. The temporal order within each instance stays intact. Only the order of instances changes.

Data may be one long continuous sequence chopped into segments. Examples include a single EEG recording partitioned into 100-sample windows, or a stock price stream sliced into fixed chunks. Shuffling the segments destroys the continuity. The RNN trains on segment 47, then segment 12, then segment 89. The hidden state cannot carry information across these artificial boundaries.

Worked Example. Consider two scenarios:

- Scenario A: You have 10,000 sentences. Each is a self-contained sequence. You randomly shuffle the sentences. Train on 8,000, test on 2,000. This is correct. Each sentence's internal word order is preserved.

- Scenario B: You have one EEG recording of 100,000 time points. You chop it into windows of 100 points each, giving 1,000 windows. If you randomly shuffle these 1,000 windows and split 80-20, your RNN trains on disjoint, out-of-order fragments. The hidden state from window 47 does not naturally connect to window 48. The RNN learns to predict from scrambled history. This is incorrect.

Scope. If your instances are independent sequences, standard random shuffle is safe. If your instances are segments of a longer continuous stream, split chronologically: train on earlier segments, test on later segments. No random shuffle.

Pitfalls. The most dangerous case is subtle. It is a dataset where sentences are independent but come from the same source (e.g., paragraphs of a book). Shuffling sentences still works for sentence-level tasks. But if your task requires paragraph-level coherence, even shuffling sentences within a paragraph can break the signal. Always ask: does my task need cross-instance temporal order?

Another pitfall: using train_test_split with shuffle=True (the default in sklearn) on segmented continuous data. Always set shuffle=False when splitting chronologically ordered segments.

Recap. Random shuffle breaks temporal order. Independent sentences can be shuffled. Continuous segments cannot.

Real-World. EEG seizure detection pipelines use chronological train-test splits: train on the first 70% of the recording, test on the last 30%. Speech recognition on independent utterances uses standard random shuffle. Stock prediction always uses chronological splits because the future must remain unseen during training.

Exam note: return_sequences=False gives output at the last time step only (many-to-one). return_sequences=True gives output at every time step (many-to-many). Remember: if you stack RNN layers, set return_sequences=True on all except the last.

Exam note: Bidirectional(SimpleRNN(64)) doubles the hidden size at each time step because forward and backward hidden states are concatenated, not averaged. Parameter count also roughly doubles.

Exam note: Random shuffle is correct for independent sequences like separate sentences. Random shuffle is incorrect for continuous time series chopped into segments — use chronological splitting instead.

13.8 Deep (Stacked) RNN

13.8.1 Hierarchical Feature Learning

Hook. A single hidden layer can learn simple patterns — like "ed" signals past tense. But can a deeper RNN understand that a paragraph is about finance, even when no single word says "money"? Deeper layers build coarser, more abstract features on top of earlier ones. This is what makes a deep RNN more expressive.

Intuition + Analogy — The Assembly Line. Picture a car assembly line. Station 1 welds the chassis. Station 2 bolts on the doors. Station 3 installs the engine. Each station receives the partially refined product from the previous one and adds its own transformation. No station can skip ahead; each depends on the output of all earlier stations. In a stacked RNN, layer 1 processes raw inputs, layer 2 refines what layer 1 produced, and layer 3 refines further. Each layer is a station on the assembly line, and the product becomes increasingly abstract as it moves upward.

Formalize — Stacked RNN Layer Equations.

A deep RNN with $L$ hidden layers has $L$ separate hidden states at every time step. Each layer $ℓ$ maintains its own recurrent weight matrix $W_{hh}^{(ℓ)}$ and input-to-hidden weight matrix $W_{x h}^{(ℓ)}$ . For layer $ℓ = 1$ (the bottom layer, closest to the input):

$ a_{t}^{(1)} = W_{hh}^{(1)} h_{t - 1}^{(1)} + W_{x h}^{(1)} x_{t} + b_{h}^{(1)} $

$ h_{t}^{(1)} = tanh (a_{t}^{(1)}) $

For layer $ℓ > 1$ , the input is the hidden state of the layer below at the same time step:

$ a_{t}^{(ℓ)} = W_{hh}^{(ℓ)} h_{t - 1}^{(ℓ)} + W_{x h}^{(ℓ)} h_{t}^{(ℓ - 1)} + b_{h}^{(ℓ)} $

$ h_{t}^{(ℓ)} = tanh (a_{t}^{(ℓ)}) $

The output layer sits on top of the final hidden layer $L$ :

$ y^_{t} = softmax (W_{h y} h_{t}^{(L)} + b_{y}) $

Key differences from a single-layer RNN:

Each layer has its own weight matrices and bias — no sharing across layers.
Layer $ℓ$ receives $h_{t}^{(ℓ - 1)}$ (output of the layer below) at the same time step as its "input."
Layer $ℓ$ also receives $h_{t - 1}^{(ℓ)}$ (its own past hidden state) through its recurrent connection.
The hidden dimensions can differ per layer: $h^{(1)} = 128$ , $h^{(2)} = 64$ , $h^{(3)} = 32$ is common.

Worked Example — 2-Layer RNN Tracing Feature Hierarchies.

Consider a speech recognition task. Input $x_{t}$ is a 4-dimensional acoustic feature vector at each 10ms frame. Layer 1 has $h^{(1)} = 3$ hidden units. Layer 2 has $h^{(2)} = 2$ hidden units. Output is a phoneme label at each frame.

Frame $t = 1$ : Raw audio features arrive.

Layer 1 weights:

$ W_{x h}^{(1)} = 0.5 0.1 0.3 0.2 0.6 0.1 0.1 0.4 0.7 0.3 0.2 0.4, W_{hh}^{(1)} = 0.1 00 0 0.1 0 00 0.1 $

Layer 2 weights:

$ W_{x h}^{(2)} = [0.4 0.3 0.6 0.1 0.2 0.5], W_{hh}^{(2)} = [0.1 0 0 0.1] $

Initial states: $h_{0}^{(1)} = 0$ , $h_{0}^{(2)} = 0$ .

Input: $x_{1} = [0.8, 0.3, 0.5, 0.2]^{⊤}$ .

Layer 1 at $t = 1$ :

$ W_{x h}^{(1)} x_{1} = 0.5 (0.8) + 0.2 (0.3) + 0.1 (0.5) + 0.3 (0.2) 0.1 (0.8) + 0.6 (0.3) + 0.4 (0.5) + 0.2 (0.2) 0.3 (0.8) + 0.1 (0.3) + 0.7 (0.5) + 0.4 (0.2) = 0.57 0.50 0.70 $

$ h_{1}^{(1)} = tanh 0.57 0.50 0.70 = 0.515 0.462 0.604 $

This is layer 1's representation — it has detected low-level acoustic patterns like formant frequencies and energy bands.

Layer 2 at $t = 1$ : Takes $h_{1}^{(1)}$ as its input.

$ W_{x h}^{(2)} h_{1}^{(1)} = [0.4 (0.515) + 0.6 (0.462) + 0.2 (0.604) 0.3 (0.515) + 0.1 (0.462) + 0.5 (0.604)] = [0.604 0.503] $

$ h_{1}^{(2)} = tanh (0.604 0.503) = [0.540 0.463] $

Now interpret what happened: layer 1's three numbers $[0.515, 0.462, 0.604]$ capture raw acoustic features. Layer 2's two numbers $[0.540, 0.463]$ are a coarser summary — perhaps distinguishing voiced vs. unvoiced sounds. The dimension compressed from 3 to 2, forcing the network to abstract.

Frame $t = 2$ : New input $x_{2} = [0.4, 0.7, 0.3, 0.6]^{⊤}$ .

Layer 1 at $t = 2$ receives both $x_{2}$ and its own past $h_{1}^{(1)}$ :

$ W_{x h}^{(1)} x_{2} = 0.57 0.58 0.64, W_{hh}^{(1)} h_{1}^{(1)} = 0.052 0.046 0.060 $

$ h_{2}^{(1)} = tanh 0.622 0.626 0.700 = 0.552 0.555 0.604 $

Layer 2 at $t = 2$ receives $h_{2}^{(1)}$ and its own past $h_{1}^{(2)}$ :

$ h_{2}^{(2)} = tanh (0.4 (0.552) + 0.6 (0.555) + 0.2 (0.604) + 0.1 (0.540) 0.3 (0.552) + 0.1 (0.555) + 0.5 (0.604) + 0.1 (0.463)) = [0.616 0.572] $

The hierarchy is visible. Layer 1 at $t = 2$ still encodes acoustic detail. Layer 2 at $t = 2$ is closer to a phoneme-level representation. If we added a layer 3, it would combine several frames of layer 2 into syllable-level features.

Assumptions & Scope.

Scope: Deep RNNs assume that higher-level features can be built by composing lower-level ones. This holds for structured sequences. Examples include speech (phoneme → syllable → word). Other examples are text (character → morpheme → word → phrase) and video (pixel patch → edge → object part). The number of layers $L$ is a hyperparameter chosen by the designer.

When it applies: Tasks where raw signals need multiple levels of abstraction. Speech recognition, handwriting recognition, and hierarchical time series are prime examples.

When it breaks: If the sequence task is simple enough, one hidden layer already captures the relevant patterns. Examples include binary sentiment on short sentences. Stacking layers then adds unnecessary parameters and training time without meaningful improvement.

Visual Intuition. Draw three horizontal rows on the page. The bottom row (layer 1) has a chain of boxes connected by rightward arrows. These are the recurrent edges at layer 1 carrying $W_{hh}^{(1)}$ . The middle row (layer 2) has its own chain of boxes with rightward arrows. Those are recurrent edges at layer 2 carrying $W_{hh}^{(2)}$ . The top row is the output layer. Now add vertical arrows at every time column. Layer 1 feeds upward into layer 2. Layer 2 feeds upward into the output. The picture looks like a grid. There are horizontal connections within each layer (recurrent). There are vertical connections between layers (feedforward). The whole grid advances from left to right in time. Takeaway: a deep RNN is a 2D lattice — depth (layers) × time. Every cell at position $(ℓ, t)$ connects right $\to (ℓ, t + 1)$ and up $\to (ℓ + 1, t)$ .

Pitfalls.

Forgetting separate weights per layer. Each layer has its own $W_{x h}^{(ℓ)}$ and $W_{hh}^{(ℓ)}$ . Do not share weights across layers — that defeats the purpose of stacking.
Mismatched dimensions between layers. $W_{x h}^{(2)}$ must have shape $h^{(2)} \times h^{(1)}$ , not $h^{(2)} \times d$ . The input to layer 2 is the hidden state of layer 1, not the original raw input.
Not using return_sequences=True on intermediate layers. When stacking, every layer except possibly the last must output the full sequence. Otherwise the next layer receives a single vector instead of a temporal signal.
Ignoring the doubled computational cost. A 2-layer RNN computes roughly twice the operations of a single-layer RNN with the same hidden size. Stacking without reducing per-layer dimensions blows up training time.

Recap + Bridge. A deep RNN stacks multiple recurrent layers. Each one learns progressively more abstract features. They range from raw input patterns at layer 1 to task-level concepts at layer $L$ . The key equations are a simple extension. Layer $ℓ$ receives the hidden state from layer $ℓ - 1$ at the same time step. It also receives its own past from the previous time step. This stacking is what gives the network its hierarchical representational power. Next, we see how BPTT operates across this stacked structure. Each layer now has its own horizontal gradient path through time.

Real-World & Domain Connection. Stacked RNNs formed the backbone of Baidu's Deep Speech (2014-2015) recognition system. That system used 5 recurrent layers to process spectrogram frames into character-level text output. The same stacking principle is used in modern architectures. A Transformer is effectively a stack of self-attention layers. Each layer refines the previous layer's representations. It follows the same hierarchical assembly-line logic, but with a different building block.

13.8.2 BPTT in Deep RNN

Hook. In a single-layer RNN, the gradient at $h_{t}$ comes from two paths. Path 1 is vertical from the output at time $t$ . Path 2 is horizontal from future time steps. In a deep RNN, every hidden state $h_{t}^{(ℓ)}$ receives gradient from three directions. Can you name the third?

Intuition + Analogy. In a single-layer RNN, a relay runner gets feedback from the coach. The coach represents the output loss. The next runner represents the future time step. In a deep RNN, a runner in the middle of the assembly line gets feedback from three sources. The coach provides direct feedback. The next runner at her own station provides feedback from the future. And the station above her provides feedback from the layer above. The station above depends on the quality of her output. If layer 2 does poorly, some of that blame flows down to layer 1 through the feedforward connection between layers. This is the third gradient path.

Formalize — The Three-Source Gradient at $h_{t}^{(ℓ)}$ .

For a hidden state $h_{t}^{(ℓ)}$ in a deep RNN, the total gradient is the sum of three contributions:

$ \frac{\partial L}{\partial h _{t}^{(ℓ)}} = Path 1: from layer above (vertical) \frac{\partial L _{t}^{vert}}{\partial h _{t}^{(ℓ)}} + Path 2: from future time (horizontal) \frac{\partial L}{\partial h _{t + 1}^{(ℓ)}} \cdot \frac{\partial h _{t + 1}^{(ℓ)}}{\partial h _{t}^{(ℓ)}} + Path 3: from the layer below (only if ℓ = L) \frac{\partial L}{\partial h _{t}^{(ℓ + 1)}} \cdot \frac{\partial h _{t}^{(ℓ + 1)}}{\partial h _{t}^{(ℓ)}} $

Wait — let us be precise. The third path actually depends on whether $ℓ$ is the top layer or an intermediate layer:

For the top layer $ℓ = L$ : Path 1 is the direct output gradient (from the loss at time $t$ ). Path 2 is the horizontal recurrent gradient from $t + 1$ . There is no path 3 because there is no layer above $L$ .

For intermediate layers $ℓ < L$ : Path 1 is the gradient coming down from layer $ℓ + 1$ (the feedforward backward path). Path 2 is the horizontal gradient from $t + 1$ within the same layer. Path 3 does not exist because the output sits above layer $L$ , not above layer $ℓ$ .

So for layer $ℓ = 1$ (bottom layer), the gradient is:

$ \frac{\partial L}{\partial h _{t}^{(1)}} = vertical: from layer 2 (W_{x h}^{(2)})^{⊤} \cdot (1 - (h_{t}^{(2)})^{2}) ⊙ \frac{\partial L}{\partial h _{t}^{(2)}} + horizontal: from time t + 1 (W_{hh}^{(1)})^{⊤} \cdot (1 - (h_{t + 1}^{(1)})^{2}) ⊙ \frac{\partial L}{\partial h _{t + 1}^{(1)}} $

For the top layer $ℓ = L$ , the vertical path comes directly from the output:

$ \frac{\partial L}{\partial h _{t}^{(L)}} = vertical: from output W_{h y}^{⊤} (y^_{t} - y_{t}) + horizontal: from time t + 1 (W_{hh}^{(L)})^{⊤} \cdot (1 - (h_{t + 1}^{(L)})^{2}) ⊙ \frac{\partial L}{\partial h _{t + 1}^{(L)}} $

The BPTT algorithm for a deep RNN runs backward through both time and layers. Start at the top layer at the last time step. Work backward through time for that layer. Then drop down one layer and repeat. This is $O (L \cdot T)$ in both time and memory.

Worked Example — Gradient Trace in 2-Layer RNN.

Reuse the 2-layer setup from Section 13.8.1. Suppose at $t = 2$ (the final time step), the output $y^_{2}$ incurs some loss. Let us trace the gradient to layer 1's hidden state $h_{1}^{(1)}$ .

Step 1 — Top layer, final time step: $\frac{\partial L}{\partial h _{2}^{(2)}} = W_{h y}^{⊤} (y^_{2} - y_{2})$ . This is the vertical gradient at the top.

Step 2 — Top layer, previous time step: The horizontal gradient flows from $h_{2}^{(2)}$ to $h_{1}^{(2)}$ . It passes through $W_{hh}^{(2)}$ and the tanh derivative:

$ \frac{\partial L}{\partial h _{1}^{(2)}} = (W_{hh}^{(2)})^{⊤} \cdot (1 - (h_{2}^{(2)})^{2}) ⊙ \frac{\partial L}{\partial h _{2}^{(2)}} $

Step 3 — Bottom layer, final time step: The vertical gradient flows from $h_{2}^{(2)}$ down to $h_{2}^{(1)}$ . It passes through $W_{x h}^{(2)}$ :

$ \frac{\partial L}{\partial h _{2}^{(1)}} = (W_{x h}^{(2)})^{⊤} \cdot (1 - (h_{2}^{(2)})^{2}) ⊙ \frac{\partial L}{\partial h _{2}^{(2)}} $

Step 4 — Bottom layer, first time step: Now $h_{1}^{(1)}$ receives the sum of two gradients:

$ \frac{\partial L}{\partial h _{1}^{(1)}} = from layer 2, same time (W_{x h}^{(2)})^{⊤} \cdot (1 - (h_{1}^{(2)})^{2}) ⊙ \frac{\partial L}{\partial h _{1}^{(2)}} + from bottom layer, future time (W_{hh}^{(1)})^{⊤} \cdot (1 - (h_{2}^{(1)})^{2}) ⊙ \frac{\partial L}{\partial h _{2}^{(1)}} $

Notice: the gradient signal has traversed three tanh gates and two weight matrices to reach $h_{1}^{(1)}$ . Each tanh gate multiplies by a number less than 1. This compounding is why the bottom layer's earliest time steps see the weakest gradient. The signal fades doubly — across layers and across time.

Pitfalls.

Forgetting the vertical gradient between layers. In a single-layer RNN, $h_{t}$ receives gradient from the output layer. In a deep RNN, intermediate layers receive gradient from the layer above instead — but the principle is the same. Omitting this path means the bottom layers learn nothing.
Running out of memory. BPTT for a deep RNN stores $L \times T$ hidden states. For $L = 3$ , $T = 100$ , and hidden size 256, we get $3 \times 100 \times 256 \times 4$ bytes. This is roughly 300 KB per sample. With batch size 64, memory hits ~2 GB just for activations. Plan buffer accordingly.
Confusing layer index with time index. $h_{t}^{(ℓ)}$ is always: layer index $ℓ$ as superscript, time index $t$ as subscript. Swapping them reverses the gradient flow (horizontal vs. vertical). Write them in the same order every time.

13.8.3 Diminishing Returns Beyond Four Layers

Hook. If 2 layers are better than 1, and 3 are better than 2, shouldn't 10 layers be incredible? The answer is a firm "no" — and the reason is written into the mathematics of backpropagation itself.

Intuition + Analogy. Imagine a game of telephone with 10 people in a line. Person 1 whispers a message to person 2, who whispers to person 3, and so on. By person 10, the message is garbled. Now add another twist. Instead of a single line, stack 4 telephone lines on top of each other. The top line's message depends on what the bottom line transmitted. The bottom line's earliest whisperer can barely influence the top line's final listener at layer 4. That whisperer sits at layer 1, $t = 1$ . That listener is at time $t = T$ . The signal has passed through too many squashing functions and matrix multiplications. This is the vanishing gradient problem compounded by depth.

Formalize — Gradient Amplification Across Layers and Time.

For a deep RNN with $L$ layers, consider the gradient at the bottom layer's earliest time step $h_{1}^{(1)}$ . It involves roughly $L + T$ multiplicative factors. Each multiplication is through a weight matrix and a tanh Jacobian:

$ \frac{\partial L}{\partial h _{1}^{(1)}} \approx time: t = 2 \prod T (W_{hh}^{(1)} ⊙ (1 - h^{2})) \cdot layer: ℓ = 2 \prod L (W_{x h}^{(ℓ)} ⊙ (1 - h^{2})) \cdot \frac{\partial L}{\partial h _{T}^{(L)}} $

Each factor shrinks the gradient. The cumulative product of many numbers smaller than 1 rapidly approaches zero — exponential decay. This decay scales roughly as $O ((λ_{m a x})^{L + T})$ where $λ_{m a x}$ is the largest eigenvalue of the combined weight-Jacobian product. If $λ_{m a x} < 1$ , the gradient vanishes. If $λ_{m a x} > 1$ , it explodes. Both are bad.

Empirically, the sweet spot is between 2 and 4 hidden recurrent layers. Beyond 4:

The gradient at layer 1 is effectively zero — those parameters stop updating.
The extra parameters ( $h^{2}$ per extra layer) increase the risk of overfitting.
Training time grows linearly with $L$ but performance gains plateau.

Practical Evidence. In the original Deep Speech paper (Hannun et al., 2014), experiments tested different layer counts. Moving from 1 to 5 recurrent layers reduced word error rate from ~30% to ~16%. But moving from 5 to 7 layers gave less than 0.3 points of extra improvement. That came at the cost of nearly 40% more training time. The authors concluded that 5 layers were the practical maximum for their task. Modern systems use gated architectures (LSTM, GRU) or residual connections to push beyond 4 layers. For a plain stacked RNN, 2 to 4 is the effective range.

Recap + Bridge. Stacking more RNN layers buys you hierarchical feature learning — phonemes to syllables to words. But the gradient must travel farther, and each extra layer and time step multiplies by a shrinking factor. Beyond roughly 4 layers, the bottom layer stops learning. This diminishing returns problem is exactly what residual connections and gated architectures were designed to solve. The next section connects this to a practical parameter counting example, so you can estimate model size before you train.

Real-World & Domain Connection. Google's first production-grade speech recognition system (2015) used a 5-layer stacked LSTM. It had 800 hidden units per layer and trained on thousands of hours of audio. The stacking allowed it to learn acoustic features at multiple timescales without hand-engineering phoneme detectors. Today's voice assistants (Siri, Alexa, Google Assistant) all rely on stacked recurrent or attention-based layers. They follow the same principle — just with better gradient flow mechanisms.

13.8.4 Student Questions and Exam Guidance

Q: A deep RNN has $L$ layers and sequence length $T$ . How many hidden states does it maintain during the forward pass?

A: $L \times T$ hidden state vectors. At each time step $t$ , all $L$ layers compute their hidden states before the next time step begins. All of them must be stored for BPTT.

Q: What happens if I make every layer the same hidden size?

A: It works — many papers do exactly this. But there is no requirement that $h^{(1)} = h^{(2)} = \dots = h^{(L)}$ . A common design is to taper the hidden sizes. Start with a larger layer near the input to capture raw detail. Then shrink in higher layers to force abstraction. For example: 256 → 128 → 64.

Q: Does stacking RNN layers always help?

A: Not always. For short sequences with simple patterns (binary sentiment on sentences under 10 words), a single-layer RNN is often enough. Adding layers increases training time and parameter count with no meaningful accuracy gain. Try one layer first; add a second only if validation loss plateaus above an acceptable level.

Exam Guidance.

Exam note — Parameter counting for deep RNN. Consider a stacked RNN with $L$ layers. The input dimension is $d$ . The layer-wise hidden dimensions are $h^{(1)}, h^{(2)}, \dots, h^{(L)}$ . The total trainable parameters follow:

$ Params = Layer 1 h^{(1)} (d + h^{(1)} + 1) + ℓ = 2 \sum L Layer ℓ h^{(ℓ)} (h^{(ℓ - 1)} + h^{(ℓ)} + 1) + Output layer v (h^{(L)} + 1) $

Where $v$ is the output vocabulary size. Note: layer $ℓ \geq 2$ uses $h^{(ℓ - 1)}$ as its input dimension. That is the previous layer's hidden size, not $d$ . This is the most common exam trick — students mistakenly use $d$ for every layer.

Numerical example. Try $L = 3$ with $d = 50$ . Use $h = [128, 64, 32]$ and $v = 10$ :

Layer 1: $128 (50 + 128 + 1) = 128 \times 179 = 22, 912$
Layer 2: $64 (128 + 64 + 1) = 64 \times 193 = 12, 352$
Layer 3: $32 (64 + 32 + 1) = 32 \times 97 = 3, 104$
Output: $10 (32 + 1) = 330$
Total: 38,698 parameters. Know this breakdown by heart.

Exam note — Gradient flow diagram. Be able to draw the gradient paths for a 2-layer, 3-time-step deep RNN. Show these gradient paths. First is the output gradient down to layer 2 at each $t$ . Second are horizontal gradients within layer 2 across time. Third are vertical gradients from layer 2 down to layer 1 at each $t$ . Fourth are horizontal gradients within layer 1 across time. Graders check that you include the tanh Jacobian $diag (1 - h^{2})$ at every step.

Exam note — Diminishing returns justification. If asked "why not just stack 50 layers?", answer with the exponential decay of the gradient. Each extra layer multiplies the gradient signal by another factor $< 1$ . The bottom layers receive an exponentially weaker signal. Cite the typical 2-4 layer range and mention residual connections as the solution covered in later lectures.

Exam note — return_sequences in stacked RNNs. When stacking, all layers except possibly the last must use return_sequences=True. If you forget this on an intermediate layer, the next recurrent layer receives a single vector instead of a sequence. The model compiles but the shape mismatch causes a runtime error. Worse, it can produce silently wrong training.

13.9 Deep RNN Parameter Counting Example

13.9.1 Two-Hidden-Layer Architecture

Hook. You have seen the equations for a stacked RNN. But can you translate a simple architecture diagram into an exact parameter count? The diagram has 2 input nodes, 2 hidden neurons in layer 1, 2 in layer 2, and 2 output nodes. If you cannot do this on paper, you cannot estimate model size before training. This example trains that muscle.

Intuition + Analogy — Counting Parts in a Two-Stage Engine. Picture a two-stage turbocharged engine. Stage 1 compresses raw air and feeds it to stage 2. Stage 2 further compresses and sends it to the cylinders. Each stage has its own turbine wheel, its own shaft, its own bearings. There are also connections. These include the intake pipe from atmosphere to stage 1. They also include the inter-stage duct and the exhaust pipe from stage 2 to the manifold. To order spare parts, you count every component in every stage, plus every connecting pipe. A deep RNN is the same. Each hidden layer is a stage. It has its own recurrent machinery (turbine wheel), its own input connection (intake pipe), and its own bias (bearing). The connections between stages are the inter-stage ducts. Count every one — or your parts order comes up short.

Formalize — All Weight Matrices with Shapes.

Given a 2-hidden-layer architecture with all dimensions equal to 2. For example, input is 2, hidden layer 1 is 2, hidden layer 2 is 2, and output is 2. The complete set of trainable weight matrices is:

Matrix	Connects	Shape	Reasoning
$W_{x h}$	Input → Hidden layer 1	$2 \times 2$	2 input features map to 2 hidden neurons
$W_{hh}^{(1)}$	Hidden layer 1 → Hidden layer 1 (recurrent)	$2 \times 2$	2 hidden neurons recurrently connect to themselves
$W_{h 1 h 2}$	Hidden layer 1 → Hidden layer 2	$2 \times 2$	Layer 1's 2 outputs become layer 2's 2 inputs
$W_{hh}^{(2)}$	Hidden layer 2 → Hidden layer 2 (recurrent)	$2 \times 2$	Layer 2's 2 neurons have their own recurrence
$W_{h y}$	Hidden layer 2 → Output	$2 \times 2$	Layer 2's 2 outputs map to 2 output nodes

Five matrices. Each $2 \times 2$ . Each has 4 entries. Total: $5 \times 4 = 20$ weights.

Critical observation: $W_{h 1 h 2}$ is the inter-layer feedforward matrix. It is NOT recurrent — it connects two different layers at the same time step. $W_{hh}^{(1)}$ and $W_{hh}^{(2)}$ are the recurrent matrices — they connect a layer to itself at the previous time step. The two types serve different purposes.

With biases: Add one bias per neuron. Hidden layer 1 has 2 neurons. Hidden layer 2 has 2 neurons. Output layer has 2 nodes. That is 6 biases. Total trainable parameters: $20 + 6 = 26$ .

13.9.2 Forward Propagation at Time Step 2

Worked Example — Step-by-Step Forward Pass at $t = 2$ .

At the second time step, the input is $x_{2} = [0, 1]^{⊤}$ . Assume all biases are zero and initial hidden states $h_{0}^{(1)} = 0$ , $h_{0}^{(2)} = 0$ . All weight matrices contain small random values.

Step 1 — Hidden Layer 1 at $t = 2$ :

$ a_{2}^{(1)} = W_{x h} x_{2} + W_{hh}^{(1)} h_{1}^{(1)} $

The first term $W_{x h} x_{2}$ transforms the current input (shape $2 \times 2 \cdot 2 \times 1 = 2 \times 1$ ). The second term $W_{hh}^{(1)} h_{1}^{(1)}$ carries information from hidden layer 1's own state at the previous time step. Its shape is $2 \times 2 \cdot 2 \times 1 = 2 \times 1$ . Both are $2 \times 1$ vectors, added elementwise, then pushed through tanh:

$ h_{2}^{(1)} = tanh (a_{2}^{(1)}) $

Step 2 — Hidden Layer 2 at $t = 2$ :

$ a_{2}^{(2)} = W_{h 1 h 2} h_{2}^{(1)} + W_{hh}^{(2)} h_{1}^{(2)} $

Notice the dual input. $W_{h 1 h 2} h_{2}^{(1)}$ is the feedforward signal from layer 1 at the same time step $t = 2$ . $W_{hh}^{(2)} h_{1}^{(2)}$ is the recurrent signal from layer 2 at the previous time step $t = 1$ . This is the key pattern. Every hidden layer above the first receives two inputs. It gets a vertical input from the layer below at the same time. It also gets a horizontal input from its own past at the previous time.

$ h_{2}^{(2)} = tanh (a_{2}^{(2)}) $

Step 3 — Output Layer at $t = 2$ :

$ y^_{2} = softmax (W_{h y} h_{2}^{(2)}) $

The output depends only on hidden layer 2's state at time step 2. There is no recurrent connection at the output layer.

Trace the data flow. $x_{2}$ is the raw input with 2 numbers. It goes to $h_{2}^{(1)}$ in layer 1 with 2 low-level features. Then to $h_{2}^{(2)}$ in layer 2 with 2 higher-level features. Finally $y^_{2}$ is the output with 2 class probabilities. Every arrow is a weight matrix. That is why there are exactly 5 matrices.

13.9.3 Total Weight Count — The Full Parameter Table

Worked Example — The Complete Parameter Table.

Connection	Matrix Symbol	Shape	Individual Weights	Bias Terms	Total This Matrix
Input → Hidden 1	$W_{x h}$	$2 \times 2$	4	—	4
Hidden 1 → Hidden 1 (recurrent)	$W_{hh}^{(1)}$	$2 \times 2$	4	—	4
Hidden 1 → Hidden 2	$W_{h 1 h 2}$	$2 \times 2$	4	—	4
Hidden 2 → Hidden 2 (recurrent)	$W_{hh}^{(2)}$	$2 \times 2$	4	—	4
Hidden 2 → Output	$W_{h y}$	$2 \times 2$	4	—	4
Hidden 1 biases	$b_{h}^{(1)}$	$2 \times 1$	—	2	2
Hidden 2 biases	$b_{h}^{(2)}$	$2 \times 1$	—	2	2
Output biases	$b_{y}$	$2 \times 1$	—	2	2

Weights only: $4 + 4 + 4 + 4 + 4 = 20$

Biases only: $2 + 2 + 2 = 6$

Total trainable parameters: $20 + 6 = 26$

General formula for this architecture. Define the dimensions. Let $d$ = input dimension, $h_{1}$ = hidden layer 1 size, $h_{2}$ = hidden layer 2 size, and $v$ = output dimension. Then:

$ Params = Layer 1 h_{1} d + h_{1}^{2} + h_{1} + Layer 2 h_{2} h_{1} + h_{2}^{2} + h_{2} + Output v h_{2} + v $

Plugging in $d = 2, h_{1} = 2, h_{2} = 2, v = 2$ :

$ Params = (4 + 4 + 2) + (4 + 4 + 2) + (4 + 2) = 10 + 10 + 6 = 26 $

Matches the table exactly.

Memory check: For this tiny network, each weight is a 32-bit float (4 bytes). 26 parameters × 4 bytes = 104 bytes. A real network with $h_{1} = 256, h_{2} = 128, d = 300$ would have:

$ Params = (256 \times 300 + 25 6^{2} + 256) + (128 \times 256 + 12 8^{2} + 128) + \dots \approx 175, 000 + $

Always estimate parameters before training. If your parameter count exceeds your training examples, you are almost certainly overfitting.

Assumptions & Scope.

Scope: This example uses the smallest possible dimensions (2-2-2-2) to make counting trivial. All dimensions are equal here purely for illustration. In practice, $h_{1}$ , $h_{2}$ , and $v$ are almost never equal.

When the 5-matrix structure applies. For any 2-hidden-layer deep RNN, there will always be exactly 5 weight matrices. They are $W_{x h}, W_{hh}^{(1)}, W_{h 1 h 2}, W_{hh}^{(2)}, W_{h y}$ . The shapes change when $d, h_{1}, h_{2}, v$ change, but the count of matrices stays at 5. For an $L$ -layer deep RNN, the number of weight matrices is $2 L + 1$ . There is one $W_{x h}$ (or inter-layer) matrix per layer ( $L$ ), one $W_{hh}$ per layer ( $L$ ), plus one $W_{h y}$ at the output.

When the bias count matters: If a question explicitly says "assume zero biases," count only weights (20). If the question does not specify, include biases (26). On exams, state both numbers and label which is which.

When this does not apply: If the architecture is bidirectional, double all layer-1 matrices (forward and backward copies). If the architecture uses gated units (LSTM, GRU), each gate introduces extra matrices. The parameter count then multiplies by roughly 4× (LSTM) or 3× (GRU).

Visual Intuition. Draw the 2-layer RNN unrolled over 3 time steps. Label every arrow. At time $t = 2$ : an arrow from $x_{2}$ to $h_{2}^{(1)}$ — label it $W_{x h}$ (4 weights). A horizontal arrow from $h_{1}^{(1)}$ to $h_{2}^{(1)}$ — label it $W_{hh}^{(1)}$ (4 weights). A vertical arrow from $h_{2}^{(1)}$ to $h_{2}^{(2)}$ — label it $W_{h 1 h 2}$ (4 weights). A horizontal arrow from $h_{1}^{(2)}$ to $h_{2}^{(2)}$ — label it $W_{hh}^{(2)}$ (4 weights). A vertical arrow from $h_{2}^{(2)}$ to $y^_{2}$ — label it $W_{h y}$ (4 weights). If you can count the labeled arrows, you can count the parameters: 5 arrows, each $2 \times 2$ , gives 20. Biases are the intercepts at each neuron — annotate them at the hidden and output nodes. Final count: 26.

Pitfalls.

Missing the inter-layer matrix $W_{h 1 h 2}$ . Students often count only the recurrent matrices and the input/output matrices. But layer 2 needs a feedforward connection from layer 1. Omitting $W_{h 1 h 2}$ undercounts by 4 weights (or $h_{2} \times h_{1}$ in general).

Confusing which matrix is $W_{x h}^{(2)}$ . In a deep RNN, the input to layer 2 is NOT $x_{t}$ . It is $h_{t}^{(1)}$ . So the matrix is $W_{h 1 h 2}$ (shape $h_{2} \times h_{1}$ ), not $W_{x h}$ again. Calling it $W_{x h}^{(2)}$ can be misleading — the notation varies across textbooks. The safe approach is to check the shapes. Whatever connects layer 1 to layer 2 must have shape $h_{2} \times h_{1}$ . It should not be $2 \times 2$ simply because $d = 2$ .

Counting biases when told to ignore them. If the problem states "assume biases are zero" or "count weights only," do not add the bias terms. For this example, skip the 6 biases. Your answer should be 20, not 26.

Forgetting that bias count equals neuron count. Each hidden neuron has one bias. Each output neuron has one bias. The number of biases is always $h_{1} + h_{2} + \dots + h_{L} + v$ . It does not depend on the number of connections — it depends only on the number of neurons.

Assuming all hidden layers must have the same size. The 2-2-2-2 example is convenient but misleading if you generalize incorrectly. A real architecture might be 100-256-128-10 (input → h1 → h2 → output). The shapes become $256 \times 100$ , $256 \times 256$ , $128 \times 256$ , $128 \times 128$ , $10 \times 128$ . The number of matrices is still 5, but the parameter counts are very different.

Recap + Bridge. A 2-layer deep RNN with equal 2-dimensional layers has exactly 5 weight matrices: $W_{x h}$ , $W_{hh}^{(1)}$ , $W_{h 1 h 2}$ , $W_{hh}^{(2)}$ , and $W_{h y}$ . All are $2 \times 2$ , giving 20 weights. With 2 biases per hidden layer and 2 at the output, the total is 26. The core counting principle is simple. There is one input matrix per layer, one recurrent matrix per layer, one inter-layer matrix per layer pair, and one output matrix. This generalizes to any depth and any hidden sizes. Next, we will see what happens when recurrence is replaced entirely: feedforward networks that process sequences without memory.

Real-World & Domain Connection. Engineers at Google Brain (2014) used parameter counting to budget their GPU memory. They did this before training stacked RNNs for machine translation. A 4-layer deep LSTM with 1,000 hidden units per layer contained roughly 13 million parameters. At 4 bytes per float, that is 52 MB just for the weights. Another 52 MB is for gradients during training. That is on top of activation storage. Estimating parameter count before launching a training job prevented out-of-memory crashes that would have wasted hours of GPU time. The same counting principle applies today: if you cannot compute the parameter count on paper, you cannot plan your hardware budget.

13.9.4 Student Questions, Answers, and Exam Guidance

Q: How do we know that $W_{h 1 h 2}$ connects hidden layer 1 to hidden layer 2 — and not the other way around?

A: By convention, the first subscript is the destination and the second is the source. $W_{h 1 h 2}$ reads as "weights from h1 to h2" — hidden layer 1 feeds hidden layer 2. Think of it like a function in matrix multiplication ( $Wx$ ). The input is on the right, and the output is on the left. The matrix's columns match the input dimension, rows match the output dimension. For $W_{h 1 h 2}$ with shape $2 \times 2$ , the 2 columns correspond to the 2 neurons in layer 1. The 2 rows correspond to the 2 neurons in layer 2. Information flows upward: input → layer 1 → layer 2 → output.

Q: A student asked: "At time step 2, does hidden layer 1 receive input from hidden layer 2?"

A: No. Information flows forward in layers but not backward. At time step 2, hidden layer 1 receives the raw input $x_{2}$ and its own past state $h_{1}^{(1)}$ . It does NOT receive anything from hidden layer 2. Hidden layer 2, on the other hand, receives hidden layer 1's output $h_{2}^{(1)}$ at the same time step. The flow is strictly bottom-up for feedforward connections. Recurrent connections are strictly within-layer, carrying information from past to present. There are no diagonal connections from layer 2 back to layer 1 across time.

Q: Why are the initial recurrent weights $W_{hh}^{(1)}$ set to zero at the very first time step?

A: At $t = 1$ , there is no previous hidden state $h_{0}^{(1)}$ . The term $W_{hh}^{(1)} h_{0}^{(1)}$ evaluates to zero regardless of what $W_{hh}^{(1)}$ contains, because $h_{0}^{(1)}$ is initialized to the zero vector. This is not the same as setting $W_{hh}^{(1)}$ itself to zero. The weight matrix still needs to be learned. It will be used at $t = 2$ and beyond. Only the initial hidden state is zero. Do not confuse initial hidden state with initial weight values.

Exam Guidance.

Exam note — Parameter counting for 2-layer deep RNN. Consider a 2-layer deep RNN. It has input dimension $d$ , hidden dimensions $h_{1}$ and $h_{2}$ , and output dimension $v$ . The breakdown is:

$ Params = Layer 1: W_{x h}, W_{hh}^{(1)}, b_{h}^{(1)} h_{1} d + h_{1}^{2} + h_{1} + Layer 2: W_{h 1 h 2}, W_{hh}^{(2)}, b_{h}^{(2)} h_{2} h_{1} + h_{2}^{2} + h_{2} + Output: W_{h y}, b_{y} v h_{2} + v $

Memorize: layer 2 uses $h_{1}$ (NOT $d$ ) as its input dimension. This is the single most common exam trap — students write $h_{2} d$ instead of $h_{2} h_{1}$ for the inter-layer matrix.

Numerical example (different dimensions): $d = 10, h_{1} = 8, h_{2} = 4, v = 3$ .

Layer 1: $8 \times 10 + 8 \times 8 + 8 = 80 + 64 + 8 = 152$
Layer 2: $4 \times 8 + 4 \times 4 + 4 = 32 + 16 + 4 = 52$
Output: $3 \times 4 + 3 = 12 + 3 = 15$
Total: 219 parameters. Walk through this breakdown step by step on your answer sheet. Graders award partial credit for correct intermediate values even if the final sum is wrong.

Exam note — Number of weight matrices formula. For an $L$ -layer deep RNN, there are $2 L + 1$ weight matrices. There are $L$ input matrices (including $W_{x h}$ for layer 1 and inter-layer matrices for subsequent layers). There are $L$ recurrent matrices. And there is 1 output matrix. For $L = 2$ : $2 \times 2 + 1 = 5$ matrices. For $L = 3$ : $2 \times 3 + 1 = 7$ matrices. Count them and verify your count before summing parameters.

Exam note — "Assume zero biases." If this phrase appears, the parameter count is:

$ Params_{no bias} = h_{1} d + h_{1}^{2} + h_{2} h_{1} + h_{2}^{2} + v h_{2} $

For the 2-2-2-2 example: $4 + 4 + 4 + 4 + 4 = 20$ . Write "20 (weights only, biases excluded)" so the grader knows you read the instruction.

Exam note — Common wrong answers and why.

16 (student forgot $W_{h 1 h 2}$ ). Missing the inter-layer connection.
22 (student added 2 biases instead of 6). Counted bias at output only, missed hidden layer biases.
32 (student doubled all matrices). Confused with bidirectional RNN.
12 (student used $d = 2$ for all three input matrices: $h_{1} d + h_{1}^{2} + h_{2} d + h_{2}^{2} + v h_{2} = 4 + 4 + 4 + 4 + 4 = 20...$ ). Wait, that gives 20 too. But with larger dimensions, using $d$ instead of $h_{1}$ for layer 2's input gives a wrong answer. For example, if $d = 100$ and $h_{1} = 256$ : using $d$ gives $h_{2} d = 128 \times 100 = 12, 800$ . Using the correct $h_{1}$ gives $h_{2} h_{1} = 128 \times 256 = 32, 768$ . The difference is 19,968 parameters — a massive error.

13.10 Gate Mechanism Intuition

13.10.1 The Kid-and-Door Analogy

Hook. A basic RNN has no choice at every time step. It blends the past hidden state and the new input together using fixed weight matrices. But what if the input is noise? What if the past is mostly irrelevant? The RNN cannot decide to ignore something. It must process everything equally. Gates change that. A gate lets the network choose. It decides how much of the past to remember, how much new information to accept, and how much to forget.

Intuition + Analogy — The Kid Home Alone. Picture a small child left at home with a door that has a peephole. The child does not yet know who should be let in and who should not. But after repeated experiences:

Day	Who arrives	What the kid does	Gate value	Meaning
1	Salesperson	Peeps, sees a stranger waving leaflets. Door stays locked.	0.0	Block entirely
2	Food delivery	Peeps, sees a package. Opens door just a crack — enough to grab the bag.	0.3	Let in a little
3	Grandma	Peeps, recognizes her face. Opens the door wide.	1.0	Let in fully
4	Neighbor's dog barking	Peeps, sees nothing relevant. Ignores.	0.0	Block entirely
5	Mom carrying groceries	Peeps, sees Mom but with heavy bags. Opens wider than usual to help.	1.0	Let in fully

Over many such days, the child learns a policy: how much to open the door for different kinds of visitors. The child does not treat every knock the same way. The response is learned, flexible, and continuous — not just "yes" or "no," but any degree in between.

Now replace the child with an RNN hidden state and the door with a gate. The peephole is the current input $x_{t}$ combined with the past memory $h_{t - 1}$ . Looking through the peephole means multiplying $x_{t}$ and $h_{t - 1}$ by learned weight matrices. The decision to open a crack, halfway, or fully is the sigmoid output — a number between 0 and 1. The RNN learns from training data which visitors (input patterns) deserve a wide-open door and which deserve to be blocked.

Extend the analogy further. Imagine the child not only controls the front door (the update gate). The child also has a back door (the reset gate) for letting out old memories. The child learns a new rule. "I should forget yesterday's argument with my sibling before welcoming Grandma. The past is not useful right now." This is exactly what a reset gate does in a GRU. The child is now not just a doorman but a full memory manager. They decide what stays, what goes, and what new information enters.

13.10.2 Formalizing the Gate

Formalize — Sigmoid as a Learnable Filter.

A gate in a gated RNN is defined by two ingredients:

A linear transformation of the input and the previous hidden state:

$ g = W_{g} x_{t} + U_{g} h_{t - 1} + b_{g} $

A sigmoid function applied element-wise to squash every value into $(0, 1)$ :

$ σ (g_{i}) = \frac{1}{1 + e ^{- g_{i}}} $

The output is a vector of values between 0 and 1. This vector is then multiplied element-wise (Hadamard product, $⊙$ ) with the information it is supposed to control:

$ filtered_output = σ (g) ⊙ v $

where $v$ is whatever the gate is filtering — past hidden state, candidate new state, or any other vector.

The element-wise multiplication is the key operation. Each element of the sigmoid output controls exactly one element of the target vector independently. Gate element 1 controls feature 1. Gate element 2 controls feature 2. There is no mixing across features in the multiplication itself. The mixing happens in the weight matrices that produce the gate values.

Two Regimes of a Gate: Soft and Hard.

The same sigmoid output can be used in two different ways:

Soft gate (continuous attenuation): Multiply the sigmoid output directly with the target vector. The gate value itself determines the degree of attenuation. A gate value of 0.64 means "keep 64%, discard 36%." This is the standard usage in GRU and LSTM. Gradients can flow through, and the gate can take intermediate values during training.

Hard gate (binary threshold): Apply a threshold at 0.5 to create a hard on/off switch:

$ hard_gate_{i} = {10 if σ (g_{i}) > 0.5 if σ (g_{i}) \leq 0.5 $

A hard gate is either fully open or fully closed — no in-between. This is conceptually cleaner (it literally is a binary decision). But it blocks gradient flow during training because the threshold operation has zero gradient. Real architectures use soft gates so the network can learn through backpropagation.

Assumptions & Scope.

Scope: This lecture section covers the conceptual gate mechanism. It explains what a gate is and why it is useful. It also covers how sigmoid and tanh serve as gating functions. It does not yet cover the specific equations of GRU or LSTM (those come in Sections 13.11 and 13.14).

What gates are: Soft, continuous, learnable filters. During training, a gate's value slides smoothly between 0 and 1 as the weight matrices update. The gate does not "snap" to 0 or 1. It learns a continuous policy, just like the kid learns degrees of door-opening.

What gates are not: Gates are not binary switches. They are not fixed attention masks. They are not post-hoc filters applied after the fact. The gate is computed alongside everything else in the forward pass. Its gradient is computed in the backward pass. The gate learns at the same time as everything else in the network.

13.10.3 Worked Examples

Worked Example — Sigmoid Gate on the Input $[5, - 30, 0.58, - 2]$ .

Consider a 4-dimensional input vector arriving at a gate. The raw values before sigmoid are:

$ g = [5, - 30, 0.58, - 2] $

Step 1 — Apply sigmoid element-wise.

$ σ (5) = \frac{1}{1 + e ^{- 5}} = \frac{1}{1 + 0.0067} \approx 0.9933 (nearly fully open) $

$ σ (- 30) = \frac{1}{1 + e ^{30}} = \frac{1}{1 + 1.07 \times 1 0 ^{13}} \approx 0.0000 (fully closed) $

$ σ (0.58) = \frac{1}{1 + e ^{- 0.58}} = \frac{1}{1 + 0.5599} \approx 0.6411 (mostly open) $

$ σ (- 2) = \frac{1}{1 + e ^{2}} = \frac{1}{1 + 7.389} \approx 0.1192 (mostly closed) $

The gate vector is: $σ (g) = [0.99, 0.00, 0.64, 0.12]$ .

Step 2 — Interpret before filtering. The raw value 5 is very positive. Sigmoid saturates near 1, so this gate position will let nearly everything through. The raw value -30 is extremely negative. Sigmoid saturates near 0, so this gate position blocks almost everything. The values 0.58 and -2 are intermediate, producing partial gate openings.

Step 3 — Element-wise multiplication with the input. Suppose this gate applies to a specific input vector. The example uses $[5, - 30, 0.58, - 2]$ . In a real network, the gate would filter a different vector. It could target the past hidden state or a candidate. Just to show the effect:

$ [5 \times 0.99, - 30 \times 0.00, 0.58 \times 0.64, - 2 \times 0.12] = [4.95, 0, 0.37, - 0.24] $

Raw value	Gate value	Filtered output	What happened
5	0.99	4.95	Slightly attenuated — almost fully preserved
-30	0.00	0	Completely erased — forgotten
0.58	0.64	0.37	Reduced by ~36%
-2	0.12	-0.24	Mostly suppressed — kept only 12%

Key insight: The gate did not treat all features equally. Feature 1 sailed through. Feature 2 was annihilated. Features 3 and 4 were partially dampened. This selectivity is what makes gates powerful — the network learns per-feature decisions about what to keep and what to discard.

Worked Example — Tanh as an Alternative Gating Function.

Now apply tanh to the same raw values $[5, - 30, 0.58, - 2]$ :

$ tanh (5) \approx 0.9999 (nearly maximum positive) $

$ tanh (- 30) \approx - 0.9999 (nearly maximum negative) $

$ tanh (0.58) \approx 0.5227 (moderate positive) $

$ tanh (- 2) \approx - 0.9640 (strong negative) $

The tanh output is: $[1.00, - 1.00, 0.52, - 0.96]$ .

Compare sigmoid and tanh side by side:

Raw value	Sigmoid	Tanh	Difference
5	0.99	1.00	Tanh saturates at 1, sigmoid at 1 — similar for large positives
-30	0.00	-1.00	Sigmoid maps to 0; tanh maps to -1 — completely different interpretation
0.58	0.64	0.52	Similar range, tanh slightly lower
-2	0.12	-0.96	Sigmoid outputs a small positive; tanh outputs a large negative

Why tanh matters for gates. Tanh outputs values in $[- 1, 1]$ , not $[0, 1]$ . A negative tanh value of -0.96 means more than just damping a feature. It means "actively flip its sign and suppress it." This is useful for candidate state proposals. If the network decides a particular feature should be reduced, a negative contribution from tanh can pull it down. Sigmoid can only reduce magnitude (multiply by a number < 1) — it cannot flip a sign. That is why GRU and LSTM use tanh for the candidate hidden state $h ~_{t}$ and sigmoid for the gates themselves. Each serves a different purpose in the gating unit.

Visual Intuition. Draw a grouped bar chart with four groups on the x-axis (Feature 1, Feature 2, Feature 3, Feature 4). Each group has three side-by-side bars: Raw Value (blue), Sigmoid Gate Value (orange), and Filtered Output (green, attenuated or erased). The orange bars always stay between 0 and 1. Feature 1: the blue bar is tall (5). The orange bar is almost equally tall (0.99). The green bar is slightly shorter (4.95). Visually, the gate barely touched it. Feature 2: the blue bar is a deep negative spike (-30). The orange bar is invisible (0.00). The green bar is flat at zero. The gate erased it completely. Feature 3: blue is moderate (0.58), orange is a bit lower (0.64), green is in between (0.37) — partial filtering. Feature 4: blue is negative (-2), orange is tiny (0.12), green is a faint blip (-0.24) — mostly suppressed. The takeaway from the chart: the gate does not reduce everything proportionally. It learns which bars to shrink and by how much. Feature 1 got a pass. Feature 2 got deleted. Features 3 and 4 got adjusted by different amounts.

13.10.4 Pitfalls, Q&A, and Bridge to GRU

Pitfalls.

Confusing the sigmoid gate with sigmoid activation. A sigmoid gate controls flow — it multiplies another vector element-wise to attenuate it. A sigmoid activation produces an output — it squashes a layer's output into $(0, 1)$ for binary classification. The function is the same; the purpose and position in the network are different. A gate uses sigmoid because its range $(0, 1)$ is perfect for "how much to let through." The output sits between 0 and 1. That is the point — not the classification label.

Thinking gates are binary during training. The threshold-at-0.5 trick creates a binary gate, but this is a conceptual exercise to build intuition. Real GRU and LSTM networks use soft gates throughout training. A gate value of 0.64 really does let 64% through. Backpropagation computes gradients through this soft multiplication. If you make the gate binary (hard threshold), the gradient is zero almost everywhere and the gate stops learning.

Assuming one gate is enough. A single gate can only do one thing: filter one vector. But a memory cell has multiple operations — keep some old info, add some new info, forget some old info. Each operation needs its own gate. The GRU uses two gates (reset and update). The LSTM uses three (forget, input, output). One gate cannot serve multiple independent filtering decisions.

Forgetting that gates have their own weight matrices. The gate values are not magic numbers. They are computed from $x_{t}$ and $h_{t - 1}$ through learned weight matrices $W_{g}$ and $U_{g}$ . These weights determine which input patterns cause the gate to open or close. Training the gate means training these weight matrices — and they are updated by the same backpropagation that updates everything else.

Confusing horizontal vs. vertical gating. A student asked whether gates control "horizontal" flow through time or "vertical" flow within a layer. The answer is both — gates filter information flowing both directions. The update gate in a GRU blends $h_{t - 1}$ with $h ~_{t}$ , which controls horizontal flow (past → present). The reset gate controls how much of $h_{t - 1}$ is used to compute the candidate $h ~_{t}$ . This influences how the network internally transforms information. It affects vertical flow from hidden state to candidate to output layer. A gate is not a directional valve. It is a filtering operation that happens wherever element-wise multiplication is applied.

Student Q&A.

> Q: Is a gating mechanism just a "horizontal activation function"? It controls flow through time, as opposed to a "vertical activation function" like tanh that transforms outputs within a layer.

> A: That distinction is tempting but oversimplified. Within one GRU unit, sigmoid is used multiple times for different purposes, and tanh is also used multiple times. Both functions receive the same inputs $x_{t}$ and $h_{t - 1}$ . But each has its own weight matrices. So each computes a different result. The final $h_{t}$ that emerges flows vertically to the output layer and horizontally to the next time step. It is not a simple horizontal-vs-vertical split. Think of the gating unit as a computational module that controls information flow in both directions. The sigmoid provides the control signal (how much to open each gate). The tanh provides the candidate content (what new information to propose). Together, they manage the memory of the network across time and depths simultaneously.

> Q: If sigmoid outputs a value of 0.64, does that mean exactly 64% of the information passes through?

> A: Yes — for that one element. The gate value 0.64 multiplies the corresponding element of the target vector. If the target element was 3.0, the output becomes $0.64 \times 3.0 = 1.92$ . The gate attenuated that element to 64% of its original magnitude. But note: information in a neural network is distributed across many elements. A gate value of 0.64 on one element and 0.12 on another means the network selectively preserves some features while suppressing others. The overall "information passing through" is not a single percentage — it is a vector of independent per-feature decisions.

Recap + Bridge to GRU. A gate is a learned, continuous, element-wise filter built from a sigmoid function. It takes values in $(0, 1)$ and multiplies them with a target vector to selectively preserve or suppress information. The network learns when to open each gate and how much. It does not use hand-coded rules. Instead, it adjusts weight matrices during backpropagation. The kid-and-door analogy captures the essence. Experience teaches the agent what to block. It also teaches what to let through partially and what to welcome fully. Every gated architecture — GRU, LSTM, and beyond — is built from these primitive gate blocks. In the next section, we assemble two gates and a tanh candidate into the GRU. The GRU is the simplest gated RNN that demonstrably improves on the vanishing gradient problem. You already understand the building material. Now you will see the blueprint.

Real-World & Domain Connection. The concept of a learned continuous gate is not unique to RNNs. The same principle appears in the Transformer's multi-head attention. Each attention head computes a softmax distribution with values between 0 and 1 summing to 1. This distribution gates which parts of the input sequence to focus on. In the Mixture of Experts architecture, a learned routing gate (using softmax) decides which expert sub-networks process each token. In all these cases, the gate is a differentiable filter. It is continuous during training so gradients can flow. Yet it remains interpretable as "how much" attention or selection. The gate mechanism introduced in this section is a fundamental building block of modern deep learning. Its importance reaches far beyond its origins in recurrent architectures.

13.11 Gated Recurrent Unit (GRU)

13.11.1 Hook and Symbol Registry

A plain RNN has no way to decide which parts of the past matter. It processes everything in one blended step. If the past is 100 words long and only word 47 matters, the RNN has no built-in mechanism to isolate word 47.

A GRU fixes this. It gives the network two knobs per neuron at every time step. One knob decides what past to forget. The other decides how much of the new proposal to accept. The network learns to turn these knobs on its own, from data.

Symbol	Meaning	LaTeX	Type
$x_{t}$	Input at time $t$	$x_{t}$	vector in $R^{d}$
$h_{t - 1}$	Previous hidden state	$h_{t - 1}$	vector in $R^{h}$
$r_{t}$	Reset gate output	$r_{t}$	vector in $(0, 1)^{h}$
$z_{t}$	Update gate output	$z_{t}$	vector in $(0, 1)^{h}$
$h ~_{t}$	Candidate hidden state	$h ~_{t}$	vector in $R^{h}$
$h_{t}$	Final hidden state	$h_{t}$	vector in $R^{h}$
$W_{r}, U_{r}$	Reset gate weight matrices	$W_{r} \in R^{h \times d}, U_{r} \in R^{h \times h}$	matrices
$W_{z}, U_{z}$	Update gate weight matrices	$W_{z} \in R^{h \times d}, U_{z} \in R^{h \times h}$	matrices
$W, U$	Candidate state weight matrices	$W \in R^{h \times d}, U \in R^{h \times h}$	matrices
$b_{r}, b_{z}, b$	Bias vectors for reset, update, candidate	$b_{r}, b_{z}, b \in R^{h}$	vectors
$σ$	Sigmoid function	$σ (x) = \frac{1}{1 + e ^{- x}}$	scalar → $(0, 1)$
$⊙$	Element-wise (Hadamard) product	$⊙$	—

13.11.2 Analogy — Editing a Document

Imagine you are editing a long document. Two questions arise at every sentence:

1. How much of the old paragraph should I delete before writing the new version? Some sections need a clean slate. Others only need a few words changed.

2. Once I draft the new version, how much of it should actually replace the old one? Sometimes your new draft is better. Sometimes the old version was fine and the edit should be minor.

These two questions map directly to the GRU:

- Reset gate ( $r_{t}$ ): decides how much of the old paragraph to erase before drafting the new one. When you start a fresh section on a completely different topic, you reset — wipe the slate. When you only need to add a clarifying sentence, you keep most of the old content and draft around it.

- Update gate ( $z_{t}$ ): decides how much of the new draft ( $h ~_{t}$ ) to adopt. It controls the blend into the final version ( $h_{t}$ ). A high value means the new draft is much better, so replace most of the old. A low value means the old version was good, so keep most of it.

The reset gate acts before the new draft is written. The update gate acts after the draft is ready. This order matters: you cannot decide how much of the draft to keep until the draft exists.

13.11.3 Formalize — The Four Key Equations

Equation 1 — Reset Gate

$r_{t} = σ (W_{r} \cdot x_{t} + U_{r} \cdot h_{t - 1} + b_{r})$

The reset gate looks at the current input $x_{t}$ and the previous hidden state $h_{t - 1}$ . It produces a vector of values between 0 and 1, one per hidden neuron. If $r_{t}^{(i)} \approx 0$ for neuron $i$ , that neuron's past is effectively erased before computing the candidate. If $r_{t}^{(i)} \approx 1$ , the full past is retained.

Concrete example: you feed a document into the network. The first three paragraphs are irrelevant boilerplate. Then the real content starts. At that transition, the network learns to push $r_{t}$ close to zero across all neurons. It hits reset. The noise from the first three paragraphs does not pollute the candidate computation for the meaningful content.

Equation 2 — Update Gate

$z_{t} = σ (W_{z} \cdot x_{t} + U_{z} \cdot h_{t - 1} + b_{z})$

The update gate has the same mathematical form as the reset gate. It has different weight matrices ( $W_{z}, U_{z}$ ) and a different bias ( $b_{z}$ ). These weights are learned independently, so the update gate learns a different filtering policy than the reset gate.

If $z_{t}^{(i)} \approx 1$ , the final hidden state will mostly use the new candidate information. If $z_{t}^{(i)} \approx 0$ , the final hidden state will mostly keep the old hidden state. If $z_{t}^{(i)} = 0.5$ , old and new get equal weight.

Equation 3 — Candidate Hidden State

$h ~_{t} = tanh (W \cdot x_{t} + U \cdot (r_{t} ⊙ h_{t - 1}) + b)$

This is the "proposed new draft". Notice that $h_{t - 1}$ is multiplied element-wise by $r_{t}$ before it enters the computation. The reset gate filters the past. Whatever the reset gate suppressed is gone and does not influence the candidate. The filtered past and the current input are combined and weighted by $U$ and $W$ . They are then squashed through tanh to $[- 1, 1]$ .

This candidate is not the final answer. It is a proposal. The update gate decides how much of it survives.

Equation 4 — Final Hidden State

$h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$

The final hidden state is a simple linear interpolation between the old state and the new candidate. The update gate $z_{t}$ acts as the interpolation coefficient:

- $z_{t}$ close to 0: almost all old state, almost no candidate → information flows through unchanged.

- $z_{t}$ close to 1: almost all candidate, almost no old state → the past is overwritten.

- Any value in between: a smooth blend.

This additive structure is the reason the GRU helps with vanishing gradients. We explore why in section 13.12.

13.11.4 Worked Example — Numerical Walkthrough and Parameter Count

Reset gate numerical example. A GRU layer with 2 hidden neurons, input dimension 2. Given specific weight matrices (shown on the lecture slide), the reset gate computation at one time step yields:

$ r_{t} = [0.67, 0.63] $

Interpretation: for neuron 1, 67% of the past hidden state flows into the candidate computation — 33% is reset away. For neuron 2, 63% flows through — 37% is reset. On average, the reset gate retains roughly two-thirds of the past at this time step.

Update gate continuation. The update gate at the same time step produces similar values — around 0.6. When plugged into equation 4:

$ h_{t} = (1 - 0.6) ⊙ h_{t - 1} + 0.6 ⊙ h ~_{t} $

The network keeps about 40% of the old hidden state and mixes in about 60% of the new candidate. For one specific neuron the split is 31% old / 69% new. These proportions are not hard-coded. They come from learned weight matrices that the network adjusted during training to minimize loss on the specific task.

Parameter count. A single GRU layer has three sets of weight matrices. It uses $d$ input features and $h$ hidden neurons. Each set has an input-to-hidden component ( $h \times d$ ). Each set has a hidden-to-hidden component ( $h \times h$ ) and a bias ( $h$ ):

Reset gate: $h d + h^{2} + h$
Update gate: $h d + h^{2} + h$
Candidate state: $h d + h^{2} + h$

Total hidden layer: $3 h (d + h + 1)$ parameters.

Add an output layer with $v$ output nodes: $v h + v$ .

Grand total: $3 h (d + h + 1) + v (h + 1)$ .

Plug in $d = 100$ , $h = 256$ , $v = 10$ :

$ 3 \times 256 \times (100 + 256 + 1) + 10 \times (256 + 1) $

$= 3 \times 256 \times 357 + 10 \times 257 $

$= 768 \times 357 + 2570 $

$= 274, 176 + 2, 570 = 276, 746 $

For comparison, a two-layer deep RNN with the same dimensions has fewer parameters. It would have $2 \times 256 \times (100 + 256 + 1) + 10 \times 257 = 185, 354$ . The GRU has about 50% more due to the three extra gate weight sets. That is the price paid for gating.

13.11.5 Scope — When the GRU Works Best

The GRU is designed for short to medium sequences (roughly up to a few hundred time steps). Within this range, its gating mechanism reliably captures dependencies. For very long sequences (thousands of steps), the GRU still outperforms a plain RNN. But the LSTM often does better. The LSTM has a separate cell state and three independent gates.

Compared to the LSTM, the GRU has one fewer gate. It has no separate forget and input gates — the update gate handles both roles. It also has no separate cell state. This makes the GRU:

- Faster to train: fewer parameters, fewer matrix multiplications per time step.

- Less prone to overfitting on small datasets: the simpler architecture is a form of implicit regularization.

- Often just as accurate as LSTM on many standard benchmarks, especially when the dataset is moderately sized.

A good rule of thumb: start with GRU. If the sequence is very long or the task demands fine-grained memory control, move to LSTM.

13.11.6 Visual — Computational Graph Flow

Picture the flow within one GRU unit at time $t$ . The same two inputs — $x_{t}$ and $h_{t - 1}$ — feed into three separate branches:

        x_t ────────────────────────────────────────────┐
            │                                           │
        h_{t-1} ────────────────────────────────────────┤
            │                                           │
            ├── [W_r, U_r] ──→ σ ──→ r_t ──┐           │
            │                                │           │
            ├── [W_z, U_z] ──→ σ ──→ z_t ───┤           │
            │                                │           │
            └── [W, U] ──────────────────────┤           │
                                             │           │
                              r_t ⊙ h_{t-1} ─┘           │
                                    │                     │
                              tanh( W·x_t + U·(r_t⊙h_{t-1}) + b )
                                    │                     │
                                    └──→ ̃h_t              │
                                          │               │
                              z_t ⊙ ̃h_t   │   (1-z_t) ⊙ h_{t-1}
                                    │     │         │
                                    └──→  +  ←──────┘
                                          │
                                         h_t

Step-by-step data dependencies:

1. Branches 1 and 2 (parallel): Compute the reset gate $r_{t}$ and update gate $z_{t}$ . Both use sigmoid. Both take the same $x_{t}$ and $h_{t - 1}$ but different weight matrices. These two computations are independent and can be done simultaneously.

2. Step 3: Apply element-wise multiplication $r_{t} ⊙ h_{t - 1}$ . This produces the reset-filtered past. This step must wait for $r_{t}$ from branch 1.

3. Step 4: Compute the candidate $h ~_{t}$ using tanh. This step must wait for the reset-filtered past from step 3.

4. Step 5: Blend old and new using the update gate. The final state $h_{t}$ is computed as $(1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$ . This step must wait for $z_{t}$ from branch 2 and $h ~_{t}$ from step 4.

The critical path is: input → reset gate → candidate → final blend. The update gate computation runs in parallel with the reset gate, so it does not lengthen the critical path.

13.11.7 Pitfalls — Common Confusions

Pitfall 1: Swapping the roles of reset and update gates.

Students often think the reset gate controls the final blending (equation 4). It does not. The reset gate only affects the candidate computation (equation 3). The update gate is the one that blends old and new in equation 4.

Think of it this way: the reset gate modifies the raw material that goes into the candidate. The update gate decides how much of the finished candidate makes it into the final answer. They operate at different stages of the pipeline.

Pitfall 2: Confusing the GRU update equation with other formulations.

Different textbooks write the final hidden state equation with the roles of $z_{t}$ and $1 - z_{t}$ swapped. For example, the d2l.ai textbook (Zhang et al., 2023) writes:

$H_{t} = Z_{t} ⊙ H_{t - 1} + (1 - Z_{t}) ⊙ H ~_{t}$

This is exactly equivalent to the professor's version:

$h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$

The only difference is the interpretation of $z_{t}$ . In the professor's version, $z_{t} \approx 1$ means "use the new candidate". In the d2l version, $Z_{t} \approx 1$ means "keep the old state". The two conventions are opposite in meaning but mathematically identical. Just relabel $z_{t}$ as $1 - Z_{t}$ . When reading external resources or exam questions, always check which convention is being used before interpreting gate values.

Pitfall 3: Thinking gates are binary.

Both gates use sigmoid, so outputs are continuous values in $(0, 1)$ . They are soft, learnable filters — not hard on/off switches. The network learns how aggressively to forget or retain at each neuron and each time step. This smoothness is what makes backpropagation through the gates possible.

13.11.8 Recap — Bridge to Vanishing Gradients

Look again at equation 4:

$h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$

This is an additive update. The previous hidden state $h_{t - 1}$ appears directly in the update. It is multiplied by $(1 - z_{t})$ without passing through tanh or a weight matrix. If $z_{t}$ is close to 0, then $h_{t} \approx h_{t - 1}$ . This means the old state is still good, so keep it. Information can flow backwards through many time steps without being repeatedly squashed by nonlinearities.

In a plain RNN, the hidden state update is:

$h_{t} = tanh (W \cdot x_{t} + U \cdot h_{t - 1} + b)$

Every time step applies tanh, which has a derivative at most 1 (and often much less). Multiply many tanh derivatives together during backpropagation, and the gradient shrinks exponentially. This is the vanishing gradient problem.

The GRU's additive path provides a "gradient highway." Its multiplicative factor is $(1 - z_{t})$ rather than a tanh derivative. When the update gate learns to stay small, gradients flow back almost undiminished.

This is the central insight that section 13.12 explores in detail. The GRU helps with vanishing gradients because it adds rather than replaces the hidden state. The update gate learns when to keep the additive highway open and when to allow fresh information in.

13.11.9 Real-World Usage, Q&A, and Exam Guidance

Real-world usage. GRUs power applications where sequences have moderate length and fast training matters:

- Machine translation: encoding a source sentence (typically 20–80 words) into a fixed-length context vector.

- Speech recognition: processing audio frames where local context within a few hundred milliseconds matters most.

- Time-series forecasting: stock price prediction, weather forecasting, sensor data — sequences where recent history carries more weight than distant history.

- Text generation: character-level or word-level language models where GRU's speed advantage over LSTM is significant during training.

Student questions and answers.

Q: "Will $z_{t}$ values be between 0 and 1, not just binary 0 or 1?"

A: Yes. Sigmoid outputs any real number between 0 and 1. The gate is a continuous, soft filter, not a hard switch.

Q: A student asked whether the gating mechanism counts as a "horizontal activation function". That function would control information flow across time steps. Is it versus a "vertical activation function" like tanh that transforms values within a layer?

A: "In this GRU unit, activations (sigmoid and tanh) are still generating the result at the hidden layer. But you are not doing a simple weighted-sum-then-activate. Instead, you control which parts of the result flow through, and in what proportion. Multiple sigmoids and multiple tanhs operate inside one GRU unit. They act on the same $x_{t}$ and $h_{t - 1}$ , each producing a different output. The same $h_{t}$ serves both directions. It propagates upward to the output layer and forward to the next time step. It is better to think of the GRU differently. It is not a horizontal or vertical activation. It is a computational unit that learns to route information."

Exam guidance.

- Four equations must be memorized with correct weight matrix notation. Expect to be asked to write them out or identify which weight matrix belongs to which gate.

- Parameter count formula: $3 h (d + h + 1) + v (h + 1)$ . Be ready to plug in numbers and compute. Common exam question: "Given d=100, h=256, v=10, compute the total parameters." Answer: 276,746.

- Gate roles: reset gate → candidate only (equation 3); update gate → final blending (equation 4). Do not swap them. A typical exam trap says the reset gate controls how much of the old hidden state is kept in the final output. That is wrong — the update gate does that.

- GRU vs plain RNN: GRU has the additive path $(1 - z_{t}) ⊙ h_{t - 1}$ that lets gradients flow without repeated tanh squashing. Plain RNN passes everything through tanh, causing gradients to vanish across long sequences.

- GRU vs LSTM: GRU has 2 gates (reset, update), one state. LSTM has 3 gates (forget, input, output), two states ( $c_{t}$ and $h_{t}$ ). GRU is simpler and faster; LSTM is more expressive for very long sequences.

- d2l notation trap: Some textbooks reverse the meaning of $z_{t}$ . In the professor's notation, $z_{t} \approx 1$ means "use new candidate." In d2l notation, $Z_{t} \approx 1$ means "keep old state." Read the definition before interpreting numbers.

13.12 Why GRU Helps with Vanishing Gradients

13.12.1 The Gradient Highway

Why does a simple element-wise addition fix one of deep learning's hardest problems? The answer lies in what the equation does — and, more importantly, what it does not do.

Imagine a city where every road between districts is a winding back alley. To travel from the outskirts to the center, you must pass through dozens of intersections. Each has a gatekeeper who flips a coin to decide whether to let you through. By the time you reach downtown, you have been stopped at almost every gate — your signal has faded to nothing. That is the basic RNN: every time step multiplies the gradient by a weight matrix $W_{hh}$ . Repeated multiplications by numbers less than 1 shrink the gradient toward zero.

Now picture the same city with a direct elevated highway running from the outskirts straight to the center. No gates. No intersections. No gatekeepers flipping coins. You can travel the entire distance without a single stop. The gradient arrives at full strength. That highway is the GRU's additive update path.

Formalizing the highway. Recall the GRU's final state equation:

$h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$

All the weight matrices — $W_{r}, U_{r}, W_{z}, U_{z}, W, U$ — live inside the gate computations and the candidate. But this final blend uses zero learned matrices. No matrix multiplication. No activation function. Only element-wise multiplication by gate values, which are numbers between 0 and 1.

Now trace the gradient backward through this equation. Differentiating $h_{t}$ with respect to $h_{t - 1}$ :

$\frac{\partial h _{t}}{\partial h _{t - 1}} = diag (1 - z_{t}) + (terms from h ~_{t} path)$

The first term is a diagonal matrix whose entries are $1 - z_{t, i}$ . No weight matrix appears. No repeated shrinkage over time. The gradient can flow from $h_{100}$ all the way back to $h_{1}$ . It does not pass through a single $W_{hh}$ multiplication. This is the gradient highway: a direct route bypassing the vanishing-gradient bottleneck.

13.12.2 Two Paths, Scope, and Pitfalls

Visualize the two parallel gradient paths. Picture a fork in the road at every time step. One path goes through the candidate branch. The gradient flows through $h ~_{t}$ , then through $tanh$ . After that, it passes through weight matrices $W$ and $U$ . This path can still suffer from vanishing gradients — those matrix multiplications are still there. The other path goes through the direct additive blend. The gradient flows through $(1 - z_{t}) ⊙ h_{t - 1}$ with no weight matrix at all. The update gate $z_{t}$ sets the balance between the two paths. When $z_{t} \approx 0$ , the direct path dominates — the gradient sails through unhindered. When $z_{t} \approx 1$ , the candidate path dominates and the highway is closed.

Scope. This design reduces vanishing gradients but does not eliminate them. The gate values themselves are computed through sigmoids and weight matrices, and those computations can still saturate. The GRU handles short-to-medium sequences well (roughly up to a few hundred time steps). For very long sequences spanning hundreds to thousands of steps, even the highway begins to lose signal. The gate values accumulate small errors. The candidate path still contributes gradient that can vanish.

Pitfall. A common mistake is believing the GRU completely solves vanishing gradients. It does not. The highway helps — significantly — but long-range dependencies beyond 500–1000 steps remain challenging. Students often confuse "mitigation" with "elimination."

Recap and bridge to LSTM. The GRU uses one update gate to blend old and new states additively. The LSTM, which you study next, extends this idea further. It separates the cell state from the hidden state. The cell state acts as a dedicated memory highway. This gives an even wider gradient freeway. Where the GRU has one additive path, the LSTM has two — one for memory and one for the hidden output. This makes LSTM the stronger choice for very long sequences, at the cost of more parameters and slower training.

Real-world. Speech recognition systems (hundreds of audio frames) and sentiment analysis on paragraph-length text both benefit from GRU's gradient highway. Machine translation or document-level tasks (thousands of tokens) typically use LSTM because the GRU's single highway eventually runs out of range.

Exam note: Expect a certain exam question. It asks why the GRU handles vanishing gradients better than a basic RNN. The answer centers on the additive update $h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$ . No weight matrix multiplies the $h_{t - 1}$ term. So the gradient avoids repeated shrinkage. Do not claim the GRU eliminates vanishing gradients. Say it reduces or mitigates them. The GRU works best for short-to-medium sequences.

13.13 Long Short-Term Memory (LSTM)

13.13.1 Hook and Analogy: The Diary and Working Memory

Imagine you are studying for final exams. You have two tools on your desk:

- A diary — thick, permanent, holds everything you have ever learned. You rarely erase from it. You flip through it whenever you need to recall old facts. It sits open, and you write new entries into it sparingly. This is the cell state $C_{t}$ .

- A sticky note — small, temporary, holds only what you are thinking about right now. You scribble on it, read from it, and replace it every few minutes. This is the hidden state $h_{t}$ .

At every moment, when a new fact $x_{t}$ arrives:

1. You look at the diary. Some old entries are no longer relevant — you cross them out (forget gate $f_{t}$ ).

2. You inspect the new fact. Parts of it are important enough to record in the diary (input gate $i_{t}$ ).

3. You write the selected new facts into the diary, adding them alongside whatever survived the crossing-out (cell state update).

4. You glance at the diary, pick the parts that matter right now, and copy them onto your sticky note (output gate $o_{t}$ ). That sticky note goes to the output layer and to the next time step.

This two-part design is the core idea of LSTM. The diary (cell state) is your long-term memory — it persists across time, changing only via deliberate, additive edits. The sticky note (hidden state) is your working memory. It is refreshed every time step. It is what you expose to the outside world.

GRU uses a single state vector that tries to be both diary and sticky note at once. LSTM splits them apart, giving each a dedicated job. The cost: one extra gate and roughly 33% more parameters. The reward: better handling of very long sequences because the diary has its own, undisturbed highway through time.

13.13.2 Symbol Registry

Before the equations, here is every symbol you will see:

Symbol	Meaning	Shape
$x_{t}$	Input vector at time $t$	$R^{d}$
$h_{t - 1}$	Previous hidden state (short-term memory)	$R^{h}$
$h_{t}$	Current hidden state (short-term memory, the sticky note)	$R^{h}$
$C_{t - 1}$	Previous cell state (long-term memory, the diary)	$R^{h}$
$C_{t}$	Current cell state	$R^{h}$
$f_{t}$	Forget gate — how much of $C_{t - 1}$ to keep	$(0, 1)^{h}$
$i_{t}$	Input gate — how much of $C ~_{t}$ to admit	$(0, 1)^{h}$
$o_{t}$	Output gate — how much of $C_{t}$ to expose	$(0, 1)^{h}$
$C ~_{t}$	Candidate cell state — proposed new long-term memory	$[- 1, 1]^{h}$
$W_{f}, U_{f}, b_{f}$	Forget gate: input weights, recurrent weights, bias	$h \times d$ , $h \times h$ , $h$
$W_{i}, U_{i}, b_{i}$	Input gate: input weights, recurrent weights, bias	$h \times d$ , $h \times h$ , $h$
$W_{c}, U_{c}, b_{c}$	Candidate cell state: input weights, recurrent weights, bias	$h \times d$ , $h \times h$ , $h$
$W_{o}, U_{o}, b_{o}$	Output gate: input weights, recurrent weights, bias	$h \times d$ , $h \times h$ , $h$
$σ$	Sigmoid function, outputs in $(0, 1)$	—
$tanh$	Hyperbolic tangent, outputs in $(- 1, 1)$	—
$⊙$	Element-wise (Hadamard) product	—
$d$	Input dimension	scalar
$h$	Hidden dimension (size of both hidden and cell state vectors)	scalar

13.13.3 Key Equation 1: The Forget Gate

The forget gate decides what to erase from long-term memory. It examines the current input and the previous short-term memory. Then it produces a filter — one number per dimension of the cell state:

$f_{t} = σ (W_{f} x_{t} + U_{f} h_{t - 1} + b_{f})$

Each component of $f_{t}$ lies in $(0, 1)$ because sigmoid squashes every real number into that interval. Think of each number as a retention coefficient:

- $f_{t}^{(j)} \approx 0$ : erase dimension $j$ entirely. Like crossing out a page in the diary.

- $f_{t}^{(j)} \approx 1$ : keep dimension $j$ fully intact. That diary page stays untouched.

- $f_{t}^{(j)} \approx 0.5$ : partially fade dimension $j$ . The information is half-forgotten.

The forget gate is applied to the previous cell state $C_{t - 1}$ (see Section 13.13.5). It only controls what is removed from the past. It has no influence over what new information is added.

Historical detail. The original LSTM (Hochreiter and Schmidhuber, 1997) did not have a forget gate. It had only input and output gates. Without a forget gate, the cell state could grow without bound on very long sequences. There was no mechanism to deliberately discard old information. The forget gate was added later by Gers, Schmidhuber, and Cummins (2000) and has been standard in every LSTM implementation since. If you read older papers, you may encounter the phrase "LSTM with forget gates" — today we just call it LSTM.

13.13.4 Key Equations 2 and 3: The Input Gate and Candidate Cell State

Adding new information to long-term memory is a two-step process. First, you propose what the new information could be. Second, you decide how much of it to actually accept.

Candidate cell state — the raw proposal. Computed with tanh so values lie in $(- 1, 1)$ :

$C ~_{t} = tanh (W_{c} x_{t} + U_{c} h_{t - 1} + b_{c})$

Tanh is used here (not sigmoid) because the proposal needs both positive and negative values. Adding to the cell state may require increasing some dimensions and decreasing others. Sigmoid would restrict the proposal to $(0, 1)$ , which prevents decreasing the cell state.

Input gate — the filter that decides how much of the proposal to accept:

$i_{t} = σ (W_{i} x_{t} + U_{i} h_{t - 1} + b_{i})$

Each component of $i_{t}$ is in $(0, 1)$ :

- $i_{t}^{(j)} \approx 0$ : reject the proposal for dimension $j$ . Do not write it into the diary.

- $i_{t}^{(j)} \approx 1$ : accept the full proposal for dimension $j$ . Copy it into the diary.

The input gate and the forget gate are fully independent. You can simultaneously erase old information using low $f_{t}$ . You can write new information using high $i_{t}$ . You can also do neither or just one. This independence — two gates, two decisions — is what gives LSTM its flexibility over GRU. In GRU, one update gate does both jobs.

The input gate filter is applied to the candidate: $i_{t} ⊙ C ~_{t}$ . If $i_{t}$ is zero everywhere, the entire candidate proposal is discarded. If it is one everywhere, the full proposal enters the cell state.

13.13.5 Key Equation 4: Cell State Update

Now we combine the old memory (after forgetting) with the new proposal (after gating):

$C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ C ~_{t}$

This is the heart of the LSTM. Notice three key properties:

1. It is additive, not multiplicative. There is no weight matrix here — no $W$ or $U$ multiplying $C_{t - 1}$ . Information flows from $C_{t - 1}$ to $C_{t}$ through only element-wise multiplication ( $f_{t} ⊙$ ) and element-wise addition ( $+$ ). No matrix multiply means no gradient vanishing from this path.

2. The Constant Error Carousel (CEC). In the original 1997 LSTM, the cell state update was simply additive. Before the forget gate was added, it was $C_{t} = C_{t - 1} + i_{t} ⊙ C ~_{t}$ . The term $C_{t - 1}$ was copied forward with a self-recurrent edge of weight exactly 1 — no gating, no weights, no nonlinearity. This direct copy path is called the Constant Error Carousel. It lets error signals flow backward through time undiminished. The forget gate later upgraded this from a constant weight of 1 to a learned, gated weight $f_{t}$ . This gives the network control over when to break the carousel. But the core insight remains: the additive path from $C_{t - 1}$ to $C_{t}$ is LSTM's gradient highway.

3. Information can persist indefinitely. If $f_{t} = 1$ (keep everything) and $i_{t} = 0$ (add nothing), then $C_{t} = C_{t - 1}$ . The cell state is copied verbatim across that time step. In principle, information stored in one time step can survive through hundreds or thousands of subsequent steps. The condition: the gates must keep the carousel spinning.

The two terms in the update have clean, separate meanings:

- $f_{t} ⊙ C_{t - 1}$ : the part of the old diary that survived forgetting (selective retention).

- $i_{t} ⊙ C ~_{t}$ : the part of the new proposal that was admitted (selective writing).

13.13.6 Key Equations 5 and 6: Output Gate and Hidden State

The cell state $C_{t}$ is the diary — everything you know. But not everything in the diary is relevant right now. The output gate decides what to pull into working memory:

$o_{t} = σ (W_{o} x_{t} + U_{o} h_{t - 1} + b_{o})$

Then the hidden state (working memory, the sticky note) is the gated, squashed version of the cell state:

$h_{t} = o_{t} ⊙ tanh (C_{t})$

Two things happen here:

- tanh squashing. The cell state $C_{t}$ can grow large in magnitude. There is no bound on its values because the additive update can accumulate. The tanh compresses it to $(- 1, 1)$ , giving the hidden state a stable, bounded range.

- Output gating. Even after squashing, not every dimension of the cell state matters right now. The output gate $o_{t}$ selects which dimensions of the squashed cell state to expose. Think of it as picking the relevant pages from the diary to copy onto the sticky note.

This $h_{t}$ serves three purposes:

- It goes to the next time step's gate computations (as $h_{t - 1}$ at time $t + 1$ ).

- It goes to the next hidden layer (if you are stacking LSTMs on top of each other).

- It goes to the output layer for making predictions at the current time step.

Note that $C_{t}$ is never directly fed into an output layer or the next LSTM layer. Only $h_{t}$ is exposed. The cell state stays internal to the LSTM cell. It is the private, long-term memory that the cell manages for itself.

13.13.7 Computational Graph and Visual Cell Diagram

Step-by-step flow at a single time step:

Inputs: x_t, h_{t-1}, C_{t-1}

  x_t ──┬──► [W_f, U_f, b_f] ──► σ  ──► f_t ──┐
        │                                       │
        ├──► [W_i, U_i, b_i] ──► σ  ──► i_t ──┤
        │                                       ├──► C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t
        ├──► [W_c, U_c, b_c] ──► tanh ──► C̃_t ─┘        │
        │                                                 │  tanh
        ├──► [W_o, U_o, b_o] ──► σ  ──► o_t ─────────────┤
        │                                                 │
  h_{t-1}─────────────────────────────────────────────────┘──► h_t = o_t ⊙ tanh(C_t)

Outputs: h_t, C_t

All four affine transformations (for $f_{t}, i_{t}, C ~_{t}, o_{t}$ ) can run in parallel. They share the same inputs $x_{t}$ and $h_{t - 1}$ . In practice, they are implemented as a single large matrix multiplication of size $4 h \times (d + h)$ . The result is then split and the appropriate nonlinearities are applied — sigmoid for the three gates, tanh for the candidate.

Steps 5 (cell state update) and 6 (hidden state production) involve no weight matrices — only element-wise operations and nonlinearities. The gradient flows through these steps without attenuation from repeated matrix multiplies. Weight updates happen only at the gate computation stage (steps 1 through 4).

Visual description of the full LSTM cell diagram:

Picture a thick horizontal line running across the top of the cell. This is the cell state highway. Information in $C_{t - 1}$ enters from the left. It first passes through the forget gate — drawn as a multiplicative node where $f_{t} ⊙ C_{t - 1}$ is computed. The result continues to an addition node (a circle with a plus sign). At this node, the gated candidate $i_{t} ⊙ C ~_{t}$ is added. The output of the addition node is $C_{t}$ , which exits to the right.

Below the highway, the hidden state pipeline branches off from $C_{t}$ . The cell state goes through a tanh nonlinearity (drawn as a box labeled "tanh"). Then it passes through the output gate, a multiplicative node where $o_{t} ⊙ tanh (C_{t})$ is computed. The result is $h_{t}$ , which exits downward and to the right.

The four gate computation blocks (forget, input, candidate, output) are drawn as small rectangular units along the bottom or sides. Each receives two inputs: $x_{t}$ (from below) and $h_{t - 1}$ (from the left). Each outputs its gate value upward into the cell state highway or the hidden state pipeline.

The entire cell is then replicated horizontally for the next time step. The outputs $C_{t}$ and $h_{t}$ feed into the next time step. They become the inputs $C_{t - 1}$ and $h_{t - 1}$ in the adjacent copy.

13.13.8 Worked Example: Parameter Count and Scope

Parameter count. Let $d = 100$ (input dimension, e.g., word embedding size) and $h = 256$ (hidden dimension).

An LSTM layer has four parameter sets — one for each gate and one for the candidate:

Parameter set	Weights $W$	Recurrent weights $U$	Bias $b$	Subtotal
Forget gate	$256 \times 100 = 25, 600$	$256 \times 256 = 65, 536$	$256$	$91, 392$
Input gate	$25, 600$	$65, 536$	$256$	$91, 392$
Candidate	$25, 600$	$65, 536$	$256$	$91, 392$
Output gate	$25, 600$	$65, 536$	$256$	$91, 392$
Total				365,568

Compact formula for the hidden LSTM layer:

$Params_{LSTM} = 4 \times (h d + h^{2} + h) = 4 h (d + h + 1)$

For comparison, a GRU with the same dimensions:

$Params_{GRU} = 3 \times (h d + h^{2} + h) = 3 h (d + h + 1) = 274, 176$

The LSTM has roughly $\frac{4}{3}$ the parameters of a GRU — about 33% more. The extra $\frac{1}{3}$ is the price of the two-state memory design. You get a dedicated long-term state $C_{t}$ and an extra output gate, at the cost of one extra parameter set.

Scope: When to choose LSTM. LSTM is the go-to choice when sequences are very long and you need reliable long-term memory. The additive cell state update gives it a gradient highway that is even cleaner than GRU's. The forget and input paths are fully separated. There is no coupling like GRU's $(1 - z_{t})$ and $z_{t}$ constraint. For sequences of hundreds or thousands of time steps, LSTM often outperforms GRU. For shorter sequences (tens to low hundreds of steps), GRU is often competitive with fewer parameters and faster training.

13.13.9 Common Pitfalls, Recap, Real-World Applications, Student Q&A, and Exam Guidance

Common pitfalls.

1. Confusing which gate does what. Each gate has a distinct, non-overlapping job:

- Forget gate $f_{t}$ → controls what to erase from the past (old cell state).

- Input gate $i_{t}$ → controls what to admit from the new (candidate cell state).

- Output gate $o_{t}$ → controls what to expose as the hidden state.

Mnemonic: Forget = Fade the past. Input = Insert the new. Output = Outward exposure.

2. Mixing up cell state and hidden state. The cell state $C_{t}$ is the internal long-term diary. It changes slowly via additive updates and is never directly sent out. The hidden state $h_{t}$ is the public short-term sticky note — it is a filtered, squashed snapshot of the cell state. Only $h_{t}$ goes to output layers and to the next layer. The cell state stays private to the LSTM cell.

3. Forgetting that tanh appears twice. The candidate cell state uses tanh (to propose values in $[- 1, 1]$ ). The hidden state computation also uses tanh (to squash the potentially unbounded $C_{t}$ before gating). These are two separate tanh applications with different roles. Do not mentally combine them.

4. Thinking the gates are binary switches. Gates output continuous values in $(0, 1)$ . They are soft, differentiable filters — not hard on/off switches. This differentiability is what allows backpropagation to train them. A gate value of 0.7 means "keep 70%."

Recap: LSTM vs GRU — the bridge to comparison.

Feature	LSTM	GRU
State vectors	2 ( $C_{t}$ for long-term, $h_{t}$ for short-term)	1 ( $h_{t}$ for both)
Gates	3 (forget, input, output)	2 (reset, update)
Parameter sets	4 (three gates + candidate)	3 (two gates + candidate)
Cell state update	$C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ C ~_{t}$	$h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$
Exposure control	Output gate $o_{t}$ filters tanh( $C_{t}$ )	No separate exposure gate; $h_{t}$ used as-is
Forgetting and inputting	Independent (two separate gates)	Coupled (one gate $z_{t}$ does both)
Best for	Very long sequences (hundreds to thousands of steps)	Moderate-length sequences, smaller/faster models

GRU's update gate $z_{t}$ does double duty. It simultaneously forgets with $1 - z_{t}$ and inputs with $z_{t}$ . These two operations are locked together: if you admit 80% new information, you must forget 80% of the old. LSTM separates these decisions: $f_{t}$ and $i_{t}$ are independent vectors. You can forget a lot of old information while admitting very little new, or vice versa. This independence is the main argument for LSTM's superior performance on very long sequences.

Real-world applications.

- Machine translation. LSTM encoder-decoder architectures (with attention, covered next lecture) were the dominant approach to translation before Transformers. The LSTM encoder reads the source sentence into a cell state. The LSTM decoder generates the target sentence from it, one word at a time. The cell state's ability to carry information across the entire source sentence is critical for handling long, complex sentences.

- Speech recognition. Bidirectional LSTMs process audio frames with both past and future context. An acoustic model must relate a phoneme at time $t$ to sounds that occurred seconds earlier and seconds later. LSTM's long-range memory handles this naturally.

- Time series forecasting. Stock price prediction, weather forecasting, energy demand estimation — all involve patterns that span long seasonal intervals. An LSTM can remember a pattern from 100 time steps ago and connect it to the current prediction.

- Handwriting recognition. Pen stroke sequences contain characters separated by gaps. An LSTM can maintain the context of the current word across pen lifts and reconnects.

- Video captioning. A CNN extracts per-frame visual features. A stacked LSTM processes the feature sequence and generates a natural language caption describing the video.

Student Q&A.

Q: A student asked whether the session covers both LSTM and GRU. Some students had already seen these architectures and others were new to them.

A: This session walks through both GRU and LSTM step by step. It covers every gate, every equation, every parameter count, and the reasoning behind each design choice. If you have seen the slides before, this deeper walkthrough reinforces the computational graphs and the gradient flow intuition.

Q: Why does LSTM have two states while GRU has only one?

A: LSTM was designed from the start around the Constant Error Carousel. This is a dedicated long-term memory path ( $C_{t}$ ) that is separate from the short-term output ( $h_{t}$ ). The hidden state is a derived quantity, computed from the cell state via the output gate. GRU was designed later as a simplification. It merges both roles into a single state vector $h_{t}$ and uses the update gate to balance retention and inputting. The two-state design gives LSTM finer control over memory. You can hold onto information in $C_{t}$ while exposing something different in $h_{t}$ . The cost is one extra gate and 33% more parameters.

Q: When should I use LSTM vs GRU on a real project?

A: If your sequences are very long (hundreds to thousands of time steps), start with LSTM. If they are shorter and you care about training speed or model size, try GRU first. When in doubt, try both — the difference is often dataset-dependent. The parameter count difference is only about 33%, so for small to medium models, computation cost is rarely the deciding factor. Many practitioners default to LSTM for translation, speech, and text generation. They use GRU for on-device models and rapid prototyping.

Q: What happens if I omit the forget gate?

A: The cell state would operate like the original 1997 LSTM: $C_{t} = C_{t - 1} + i_{t} ⊙ C ~_{t}$ . Information would accumulate additively with no mechanism for deliberate erasure. On very long sequences, the cell state norms can grow without bound. The forget gate was introduced precisely to fix this. It lets the network learn when to reset or fade specific dimensions of memory.

Exam guidance — formulas and facts to memorize.

You must be able to write down all six LSTM equations from memory:

1. $f_{t} = σ (W_{f} x_{t} + U_{f} h_{t - 1} + b_{f})$ — forget gate

2. $i_{t} = σ (W_{i} x_{t} + U_{i} h_{t - 1} + b_{i})$ — input gate

3. $C ~_{t} = tanh (W_{c} x_{t} + U_{c} h_{t - 1} + b_{c})$ — candidate cell state

4. $C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ C ~_{t}$ — cell state update

5. $o_{t} = σ (W_{o} x_{t} + U_{o} h_{t - 1} + b_{o})$ — output gate

6. $h_{t} = o_{t} ⊙ tanh (C_{t})$ — hidden state

Know the parameter count formula: $4 h (d + h + 1)$ for the LSTM hidden layer.

Know which activation function goes where. Sigmoid for all three gates (forget, input, output) → outputs in $(0, 1)$ . Tanh for the candidate cell state → outputs in $(- 1, 1)$ . Tanh for squashing the cell state before the output gate.

Know the historical fact: the forget gate was added by Gers, Schmidhuber, and Cummins (2000). The original 1997 LSTM (Hochreiter and Schmidhuber) had only input and output gates. The Constant Error Carousel (CEC) is the self-recurrent edge on the cell state. It is a weight-1 connection that lets error signals flow backward undiminished. The forget gate made this edge's effective weight learnable.

13.14 GRU vs LSTM: When to Use Which

Every carpenter picks the right tool for the job. A hammer is not always better than a screwdriver — it depends on what you are building. GRU and LSTM are two tools for gated sequence modeling. Neither is universally superior. This section gives you a practical decision framework.

13.14.1 Expanded Comparison

The table below extends what you already know with dimensions that matter in practice:

Dimension	GRU	LSTM
Gates	2 (reset, update)	3 (forget, input, output)
Memory states	1 ( $h_{t}$ only)	2 ( $C_{t}$ and $h_{t}$ )
Long/short-term separation	No — one state handles both	Yes — cell state for long-term, hidden state for output
Parameters per layer	$3 h (d + h + 1)$	$4 h (d + h + 1)$
Training speed per epoch	Faster (fewer parameters, fewer gate computations)	Slower (~33% more gate computations)
Gradient flow	Strong. Additive update: $h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$	Stronger for very long range. Cell state has unbroken additive highway: $C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ C ~_{t}$
Vanishing gradient resistance	Good up to ~200 time steps	Excellent up to thousands of time steps
Effective sequence length	Short to medium (10–200 steps)	Long (100–5000+ steps)
Memory usage (GPU)	Lower (~75% of LSTM)	Higher (more weights + extra state vector)
Typical use cases	Sentiment analysis, short text classification, time-series with moderate lookback, rapid prototyping	Machine translation, speech recognition, language modeling, long-document classification

The core trade-off: GRU gives you speed and simplicity. LSTM gives you stronger long-range memory at the cost of more parameters and slower training.

Why does LSTM handle very long sequences better? The cell state $C_{t}$ is never squashed by an activation function before being stored. It just gets additively updated each step — no tanh, no sigmoid applied to the stored state. In the GRU, $h_{t}$ goes through tanh during the candidate computation. While the additive blend still helps, the hidden state itself is always involved in active computation. LSTM isolates its long-term memory from the active filtering. This gives it an edge when dependencies span hundreds of time steps.

13.14.2 Worked Example: Training Cost Comparison

You build a sequence model with input dimension $d = 300$ , hidden dimension $h = 512$ , and vocabulary size $v = 10, 000$ . You have a dataset of 100,000 sequences with average length 150 tokens.

Parameter count:

- GRU: $3 \times 512 \times (300 + 512 + 1) = 3 \times 512 \times 813 = 1, 248, 768$ hidden-layer parameters.

- LSTM: $4 \times 512 \times 813 = 1, 665, 024$ hidden-layer parameters.

- Output layer (shared): $10, 000 \times 512 + 10, 000 = 5, 130, 000$ .

Total: GRU ~6.38M, LSTM ~6.80M. About 6.6% more for LSTM at this scale (the output layer dominates). For an encoder-only model without the large output layer, the gap is exactly 33% more parameters for LSTM.

Training time per epoch:

On the same GPU, LSTM computes 4 affine transformations per time step (forget, input, output gates + candidate). GRU computes 3. The GRU per-step cost is roughly 75% of the LSTM per-step cost. For 100,000 sequences × 150 steps each:

- GRU: ~85 seconds per epoch (hypothetical GPU).

- LSTM: ~113 seconds per epoch. That is roughly 33% longer.

Epochs to convergence:

GRU trains faster per epoch. But on very long-range tasks, LSTM may converge in fewer epochs. Its separate cell state learns long-term patterns more efficiently. On a machine translation task with long sentences, you might see:

- LSTM: converged at epoch 12 (× 113s = 22.6 minutes).

- GRU: converged at epoch 18 (× 85s = 25.5 minutes).

The winner is task-dependent. This is why you must try both.

13.14.3 Scope: Trade-offs in Practice

When GRU wins:

- Short-to-medium sequences (≤200 steps).

- Tight training budgets or limited GPU memory.

- Rapid experimentation cycles where iteration speed matters.

- Many time-series forecasting tasks, short text classification, simple chatbot response models.

- Situations where the data does not have very long-range dependencies.

When LSTM wins:

- Long sequences (hundreds to thousands of steps).

- Tasks where information at step 1 must influence step 500. Example: a pronoun at the end of a paragraph referring to a name at the beginning (anaphora resolution).

- Large-scale industrial deployments where final quality matters more than training cost.

- Tasks with complex, nested dependencies benefit from LSTM. Its independent forget and input gates allow simultaneous forgetting and adding. GRU's coupled gates cannot do this.

The middle ground: For moderate-length sequences (50–250 steps), the two often perform within 1–2% of each other on benchmark metrics. In this regime, pick based on your compute budget and deployment constraints. If you are deploying to a mobile device with limited RAM, the 25% parameter savings of GRU matters. If you are training on a GPU cluster, LSTM is the safer bet for squeezing out the last 0.5 BLEU score.

13.14.4 Common Pitfalls

Pitfall 1: Assuming LSTM is always better. This is the most common mistake. More gates do not give you better results automatically. They give you more capacity to model long-term dependencies. If your task does not have long-term dependencies, the extra capacity goes unused. It still costs you training time and risks overfitting.

Pitfall 2: Never trying both. Students often pick one architecture and stick with it. The right choice is empirical. Train a GRU. Train an LSTM. Compare on your validation set. The result often surprises you. A well-tuned GRU can beat a poorly-tuned LSTM on almost any task.

Pitfall 3: Ignoring the output layer. The comparison above showed that for tasks with large vocabularies, the output layer dominates the parameter count. The GRU-vs-LSTM difference in total parameters shrinks from 33% to ~7% in that case. If training speed is your bottleneck, focus on the output layer first. Try weight tying or a smaller vocabulary before switching from LSTM to GRU.

Pitfall 4: Not accounting for sequence length in architecture choice. A model that works perfectly on validation data with 50-token sequences may fail catastrophically at test time with 500-token sequences. LSTM was designed for this gap. GRU often was not.

13.14.5 Recap and Bridge to Next Module

You now have two gated RNN architectures in your toolkit:

- GRU: Two gates (reset, update), one state, additive highway through the update gate. Built for speed and simplicity. Works best on short-to-medium sequences. Parameter count: $3 h (d + h + 1)$ .

- LSTM: Three gates (forget, input, output), two states (cell and hidden), additive highway through the cell state with no activation squashing. Built for long-range dependencies. Parameter count: $4 h (d + h + 1)$ .

The choice is empirical. Start with GRU for speed, switch to LSTM if the task demands long-range memory.

What comes next? Both GRU and LSTM process one time step at a time. The next lecture introduces architectures that process entire sequences simultaneously — attention mechanisms and transformers. These build on the gating concepts you learned here. The forget gate in LSTM and the additive update in GRU reappear as residual connections and layer normalization in transformers. You are climbing a ladder. Each rung rests on the one below.

13.14.6 Real-World Usage

In industry, LSTM remains the default choice for production-grade sequence modeling that involves long-range context. Google's original Neural Machine Translation system (GNMT) used stacked LSTMs. Speech recognition systems at major companies use LSTMs for acoustic modeling. Financial time-series anomaly detection at scale often uses LSTMs for the long memory window.

GRU sees heavy adoption in resource-constrained environments: on-device text prediction (phone keyboards), lightweight chatbots, edge-computing sensors, and rapid prototyping pipelines. A notable success: the GRU was the recurrent unit of choice in several early neural conversational models. Training speed made hyperparameter search tractable.

The transformer architecture (next module) has largely superseded both for many NLP tasks. But GRU and LSTM remain essential as components inside hybrid architectures. For example, an LSTM layer inside a transformer decoder handles sequential reasoning. Understanding when to use which is a skill you will carry forward.

Exam note: Expect questions that ask you to compare GRU and LSTM across multiple dimensions. These include gates, states, parameter counts, gradient flow mechanisms, and suitable sequence lengths. A common question asks why LSTM handles longer sequences than GRU. The answer is the isolated cell state with additive-only updates. It has no squashing function applied to the stored memory. Also be ready to calculate parameter counts for both architectures given $d$ , $h$ , and $v$ .

13.15 Preview: Encoder-Decoder and Transformers

13.15.1 Encoder-Decoder Architecture

Hook. So far, every model we have built maps a fixed-size input to a fixed-size output. But what happens when the input and output have different lengths? A machine translation system takes a 7-word English sentence and produces a 9-word French sentence. A summarizer reads a 500-word article and writes a 30-word summary. Standard RNNs cannot handle this mismatch directly. We need a design that separates reading from writing.

Analogy. Think of a professional translator. She does not translate word-by-word as each word arrives. Instead, she reads the entire document first. She forms a complete mental picture of what it says. Then she produces the translation in the target language. The reading phase is encoding — compressing the input into ideas. The speaking phase is decoding — expanding those ideas into a new sequence.

Formalization. The encoder-decoder architecture splits the task into two networks:

- Encoder: reads the input sequence $x_{1}, x_{2}, \dots, x_{T}$ . It compresses the sequence into a single fixed-length vector $c$ . This is often called the context vector or thought vector.

- Decoder: takes $c$ as its initial state and generates the output sequence $y_{1}, y_{2}, \dots, y_{U}$ one token at a time.

Mathematically: $c = t e x t E n co d er (x_{1}, x_{2}, d o t s, x_{T})$ $y_{1}, y_{2}, d o t s, y_{U} = t e x t D eco d er (c)$

Both the encoder and decoder are typically RNNs (or their gated variants — LSTM, GRU). The encoder's last hidden state becomes $c$ . The decoder starts from $c$ and produces output tokens. It feeds each predicted token back as input for the next step. This process is called autoregressive decoding.

Visual Description. Picture two rectangles side by side. On the left is the encoder. A vertical stack of RNN cells receives input tokens $x_{1}, x_{2}, x_{3}, d o t s$ one at a time. The final hidden state, shown as a single vector $c$ , flows across a bridge from the encoder to the decoder. On the right, the decoder: another unrolled RNN initialized with $c$ . It generates $y_{1}$ , feeds $y_{1}$ back to produce $y_{2}$ , and continues. It stops when a special <EOS> (end-of-sequence) token is emitted. The input and output sequences can be different lengths.

One Limitation. The context vector $c$ is a bottleneck. For long input sequences, compressing everything into one fixed-size vector loses information. This is where attention mechanisms come in. Instead of relying on a single $c$ , the decoder learns to look back at all encoder hidden states. At each decoding step, it decides which parts of the input are relevant. This is the bridge from encoder-decoder to the next big idea.

Real-World Applications.

Task	Input	Output
Machine Translation	English sentence	French/German/Japanese sentence
Text Summarization	News article (500 words)	Summary (30 words)
Image Captioning	Image (via CNN encoder)	Description sentence
Speech Recognition	Audio features (long)	Recognized text (shorter)

Scope and Recap. In this lecture, you have explored the full RNN family: vanilla RNNs, LSTMs, GRUs, and bidirectional architectures. The encoder-decoder pattern is the natural next step. It reuses the gated units you already understand, but arranges them into two cooperating networks. Next lecture begins with attention. This mechanism solves the bottleneck problem. It sets the stage for transformers — the architecture behind GPT, BERT, and all modern large language models.

Exam Guidance Summary

This is a consolidated reference of all exam-relevant guidance points from Lecture 13 — RNNs, GRUs, LSTMs, and related concepts. Use it as a quick review checklist before the exam.

Parameter Counting

Exam note: Expect small numerical questions. Given input dimension $d$ , hidden dimension $h$ , and output dimension $v$ , compute total trainable parameters.

GRU: $3 h (d + h + 1) + v (h + 1)$
LSTM: $4 h (d + h + 1) + v (h + 1)$

Each gate contributes: $h d$ (input-to-hidden weights), $h^{2}$ (hidden-to-hidden weights), and $h$ (bias). The output layer adds $v (h + 1)$ parameters.

Activation Functions: Softmax vs Sigmoid

Exam note: Read the use case description carefully.

Softmax → Multi-class classification with mutually exclusive classes. The number of output nodes equals the number of classes.
Sigmoid → Binary classification or multi-label classification (each label is independent).

Backpropagation Through Time (BPTT)

Exam note: Be prepared to trace both forward and backward propagation step by step on a small numerical example.

Forward: current input $\times W_{x h} +$ previous hidden $\times W_{hh}$ $\to$ tanh.
Backward: gradient from output $\to$ through $W_{h y}$ $\to$ through $W_{hh}$ and tanh derivative.
Show all intermediate calculations. Graders reward clear step-by-step work.

Bidirectional RNNs

Exam note: A bidirectional RNN with identical hidden dimensions has 2× the parameters of a single-direction RNN. The output layer is not included in this doubling, but its input dimension does double.

Use case: When the entire sequence is available before making a prediction — documents, webpages, offline speech recordings.

Not for: Streaming/real-time data where future context is unavailable.

Gate Mechanisms

Exam note: Know the role of every gate and be able to write the formula for each.

Gate	LSTM	GRU	Role
Forget / Reset	Forget gate	Reset gate	Decide what past information to discard
Input / Update	Input gate	Update gate	Decide what new information to add / blend old and new
Output	Output gate	—	Control what the cell state exposes to the hidden output

GRU formulas (two gates):

Reset gate: $r_{t} = σ (W_{r} \cdot [h_{t - 1}, x_{t}])$
Update gate: $z_{t} = σ (W_{z} \cdot [h_{t - 1}, x_{t}])$

LSTM formulas (three gates):

Forget gate: $f_{t} = σ (W_{f} \cdot [h_{t - 1}, x_{t}])$
Input gate: $i_{t} = σ (W_{i} \cdot [h_{t - 1}, x_{t}])$
Output gate: $o_{t} = σ (W_{o} \cdot [h_{t - 1}, x_{t}])$

GRU vs LSTM

Exam note: Be ready to compare the two architectures.

Property	GRU	LSTM
Gates	2 (reset, update)	3 (forget, input, output)
Memory states	1 (hidden state only)	2 (hidden state + cell state)
Parameters	$3 h (d + h + 1)$	$4 h (d + h + 1)$
When to use	Smaller datasets, faster training	Longer sequences, more complex dependencies

Deep RNNs

Exam note: Diminishing returns beyond roughly 4 hidden layers due to vanishing gradients.

Mitigation strategies: residual connections and batch normalization.

Data Handling

Exam note: Never randomly shuffle data within a long continuous sequence. Shuffling individual independent sentences is fine; shuffling segments of a continuous sequence destroys temporal dependencies.

`return_sequences` Keras Parameter

Exam note: Know when to set return_sequences to False vs True.

False → Only the final time step output is returned. Use for sentence-level classification (one label per sequence).
True → Output at every time step. Use for word-level tagging (one label per token, e.g., POS tagging, NER).

General Exam Tips

Exam note: All formulas should use clear notation. Show intermediate steps in numerical problems. Writing assumptions explicitly (e.g., "biases assumed zero") is good practice. It makes your work easier for graders to follow and earns partial credit.

Key Industry Applications

- Stock market prediction: sequential temporal data with autocorrelation across days — classic RNN use case for time series forecasting. Quant funds like Renaissance Technologies and Two Sigma use such models to predict asset prices and execute trades.

- Natural language processing — POS tagging: each word in a sentence tagged as noun, verb, preposition, etc. Multi-class classification at every time step. SpaCy and NLTK ship pre-trained POS taggers built on recurrent architectures that handle millions of sentences daily.

- Sentiment analysis: classifying a sentence or review as positive, negative, or neutral using the RNN's final hidden state. Twitter and Facebook use RNN-based sentiment classifiers to detect harmful content and gauge public mood on trending topics.

- Machine translation: encoder RNN reads the source language, decoder RNN generates the target language. Google Translate and DeepL rely on encoder-decoder architectures to serve billions of translations per day across 100+ languages.

- Text summarization: encoder compresses the document, decoder generates the summary — requires encoder-decoder architecture beyond a plain RNN. Tools like Otter.ai and Fireflies.ai use these models to produce meeting summaries and lecture notes automatically.

- Speech recognition: hierarchical deep RNN — first layer detects phonemes, second layer assembles syllables, third layer forms words. Amazon Alexa, Apple Siri, and Google Assistant all depend on deep recurrent layers. These layers convert spoken audio into text in real time.

DNN Lecture 13 notes · Recurrent Neural Networks — Advanced Architectures

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

113.1 RNN Recap and Backward Propagation Through Time

13.1 RNN Recap and Backward Propagation Through Time

213.2 BPTT Numerical Example

13.2 BPTT Numerical Example

313.3 Output Activation: Softmax vs Sigmoid

13.3 Output Activation: Softmax vs Sigmoid

413.4 Vanishing and Exploding Gradients in RNN

13.4 Vanishing and Exploding Gradients in RNN

513.5 Memory Metaphor for Specialized RNN Architectures

13.5 Memory Metaphor for Specialized RNN Architectures

613.6 Bidirectional RNN

13.6 Bidirectional RNN

713.7 Python Implementation Notes

13.7 Python Implementation Notes

813.8 Deep (Stacked) RNN

13.8 Deep (Stacked) RNN

913.9 Deep RNN Parameter Counting Example

13.9 Deep RNN Parameter Counting Example

1013.10 Gate Mechanism Intuition

13.10 Gate Mechanism Intuition

1113.11 Gated Recurrent Unit (GRU)

13.11 Gated Recurrent Unit (GRU)

1213.12 Why GRU Helps with Vanishing Gradients

13.12 Why GRU Helps with Vanishing Gradients

1313.13 Long Short-Term Memory (LSTM)

13.13 Long Short-Term Memory (LSTM)

1413.14 GRU vs LSTM: When to Use Which

13.14 GRU vs LSTM: When to Use Which

1513.15 Preview: Encoder-Decoder and Transformers

13.15 Preview: Encoder-Decoder and Transformers

16Exam Guidance Summary

Exam Guidance Summary

17Key Industry Applications

Key Industry Applications

Postgraduate students in Deep Neural Networks

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review. If something does not make sense, go back to the full explanation in the main content.

RNN Forward Propagation

Must-know: RNNs use shared weights across all time steps. The hidden state is a lossy summary of the past. Forward equations: h_t = tanh(W_hh h_{t-1} + W_xh x_t + b_h); y_hat_t = softmax(W_hy h_t + b_y).

$h_{t} = tanh (W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h})$

⚠️ Top pitfall: Forgetting that weights are shared across time steps. Each weight matrix receives gradient contributions from every time step.

Self-check: If input dimension d=2, hidden h=3, sequence T=3, how many parameters in the RNN (include biases)?

Connects to: BPTT, Vanishing Gradients, GRU

Backpropagation Through Time (BPTT)

Must-know: BPTT is standard backprop applied to the unrolled RNN graph. Each hidden state h_t receives gradient from two sources: the output at time t (vertical) and future states through the recurrent connection (horizontal).

$\frac{\partial L}{\partial h _{t}} = W_{h y}^{⊤} (y^_{t} - y_{t}) + W_{hh}^{⊤} \cdot diag (1 - h_{t + 1}^{2}) \cdot \frac{\partial L}{\partial h _{t + 1}}$

⚠️ Top pitfall: Forgetting to sum both gradient paths at h_t. Omitting either path gives incorrect gradients and prevents learning.

Self-check: Trace the gradient from L_2 back to h_1 for a 2-step scalar RNN with given weights.

Connects to: Two-Fold Gradient, Vanishing Gradients, Computational Graphs

Softmax vs Sigmoid Output Activation

Must-know: Softmax is for multi-class classification (mutually exclusive classes, probabilities sum to 1). Sigmoid is for binary or multi-label classification (independent binary decisions per class).

$Softmax: y^_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}} Sigmoid: y^= \frac{1}{1 + e ^{- z}}$

⚠️ Top pitfall: Using softmax for multi-label tasks forces a trade-off between classes. Using sigmoid for multi-class fails to enforce exclusivity.

Self-check: A medical diagnosis system needs to detect multiple co-occurring conditions. Should the output use softmax or sigmoid?

Connects to: Multi-class Classification, Multi-label Classification, Cross-entropy Loss

Vanishing and Exploding Gradients in RNNs

Must-know: Gradients in vanilla RNNs decay or explode exponentially with sequence length because each time step multiplies the gradient by W_hh^T and the tanh derivative (always ≤ 1).

$\nabla_{h_{T - k}} L \approx (W_{hh}^{⊤})^{k} \cdot \nabla_{h_{T}} L$

⚠️ Top pitfall: Stacking layers compounds the problem. Effective depth becomes T times L. Beyond roughly 4 layers, gradient vanishes entirely.

Self-check: With spectral radius λ=0.85 and avg tanh derivative 0.5, what fraction of gradient survives after 20 steps?

Connects to: BPTT, GRU Additive Path, LSTM Cell State, Gradient Clipping

Bidirectional RNN

Must-know: A bidirectional RNN runs two independent RNNs, one forward and one backward, and concatenates their hidden states at each time step. This gives each position full left and right context.

$h_{t}^{bi} = [h_{t}; h_{t}]$

⚠️ Top pitfall: Cannot use for streaming or real-time since the backward pass requires the full sequence. Parameter count roughly doubles.

Self-check: A 2-input, 2-hidden, 2-output bidirectional RNN has how many parameters with biases?

Connects to: Encoder-Decoder, Deep RNN, POS Tagging, NER

Deep (Stacked) RNN

Must-know: Each layer has its own weight matrices. Layer ℓ receives input from layer ℓ-1 at the same time step and its own past hidden state. Beyond 4 layers, diminishing returns set in due to gradient vanishing.

$Params = h_{1} (d + h_{1} + 1) + ℓ = 2 \sum L h_{ℓ} (h_{ℓ - 1} + h_{ℓ} + 1) + v (h_{L} + 1)$

⚠️ Top pitfall: Using d instead of h_{ℓ-1} as input dimension for layer ℓ≥2. Forgetting return_sequences=True on intermediate layers.

Self-check: A 2-layer RNN with d=10, h1=8, h2=4, v=3 has how many parameters?

Connects to: BPTT, Vanishing Gradients, Residual Connections

Gate Mechanism

Must-know: Gates are continuous, learned filters using sigmoid to output values in (0,1). Each element independently controls one feature. Tanh is used for candidate proposals (range -1 to 1).

$Gate: g = σ (W_{g} x_{t} + U_{g} h_{t - 1} + b_{g}); Filtered: σ (g) ⊙ v$

⚠️ Top pitfall: Confusing a sigmoid gate (controls flow) with a sigmoid activation (produces output). Gates are NOT binary, they are soft and differentiable.

Self-check: Why does GRU use tanh for the candidate hidden state but sigmoid for the gates?

Connects to: GRU, LSTM, Attention Mechanisms

GRU (Gated Recurrent Unit)

Must-know: GRU has two gates: reset (controls how much past to erase before the candidate) and update (controls blend of old and new). Four key equations with 3h(d+h+1) parameters.

$r_{t} = σ (W_{r} x_{t} + U_{r} h_{t - 1} + b_{r}) z_{t} = σ (W_{z} x_{t} + U_{z} h_{t - 1} + b_{z})$ $h ~_{t} = tanh (W x_{t} + U (r_{t} ⊙ h_{t - 1}) + b) h_{t} = (1 - z_{t}) ⊙ h_{t - 1} + z_{t} ⊙ h ~_{t}$

⚠️ Top pitfall: Swapping reset and update gate roles. Reset affects the candidate only; update controls the final blend. Confusing the d2l notation with the professor's notation.

Self-check: With d=100, h=256, v=10, compute GRU total parameters. Answer: 276,746.

Connects to: Vanishing Gradients, LSTM, Gate Mechanism

Why GRU Mitigates Vanishing Gradients

Must-know: The additive update h_t = (1-z_t) ⊙ h_{t-1} + z_t ⊙ h_tilde_t creates a gradient highway. No weight matrix multiplies the h_{t-1} term, so gradients flow backward without repeated shrinkage.

$\frac{\partial h _{t}}{\partial h _{t - 1}} = diag (1 - z_{t}) + (candidate path terms)$

⚠️ Top pitfall: Claiming GRU eliminates vanishing gradients. It mitigates them. Long-range dependencies beyond roughly 500 steps still challenge GRU.

Self-check: Compare the GRU gradient highway with the plain RNN gradient path. Why does the GRU's additive path not shrink gradients?

Connects to: LSTM Cell State, Residual Connections, Skip Connections

LSTM (Long Short-Term Memory)

Must-know: LSTM has two states: cell state C_t (long-term diary) and hidden state h_t (short-term sticky note). Three gates: forget (erase), input (write), output (expose). 4h(d+h+1) parameters.

$f_{t} = σ (W_{f} x_{t} + U_{f} h_{t - 1} + b_{f}) i_{t} = σ (W_{i} x_{t} + U_{i} h_{t - 1} + b_{i})$ $C ~_{t} = tanh (W_{c} x_{t} + U_{c} h_{t - 1} + b_{c}) C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ C ~_{t}$ $o_{t} = σ (W_{o} x_{t} + U_{o} h_{t - 1} + b_{o}) h_{t} = o_{t} ⊙ tanh (C_{t})$

⚠️ Top pitfall: Forgetting which gate does what. Mnemonic: Forget = Fade past, Input = Insert new, Output = Outward exposure. Cell state is NEVER directly exposed to output layers.

Self-check: Write all 6 LSTM equations from memory. Why is the cell state update called the Constant Error Carousel?

Connects to: GRU, Vanishing Gradients, Constant Error Carousel

GRU vs LSTM

Must-know: GRU: 2 gates, 1 state, 3h(d+h+1) params, faster, good for short to medium sequences. LSTM: 3 gates, 2 states, 4h(d+h+1) params, better for very long sequences (100-5000+ steps).

$GRU: 3 h (d + h + 1) LSTM: 4 h (d + h + 1)$

⚠️ Top pitfall: Assuming LSTM is always better. If the task has no long-range dependencies, the extra capacity goes unused and increases overfitting risk.

Self-check: You have a text classification task with 50-token sentences. Which architecture would you try first and why?

Connects to: Encoder-Decoder, Transformers, Sequence Modeling

Encoder-Decoder Architecture

Must-know: Two-network design: encoder compresses input to a context vector, decoder generates output from it. Handles variable-length input and output. The context vector bottleneck motivates attention mechanisms.

$c = Encoder (x_{1}, \dots, x_{T}) y_{1}, \dots, y_{U} = Decoder (c)$

⚠️ Top pitfall: Forgetting that a plain encoder-decoder with a single context vector loses information for long inputs. This is what attention mechanisms fix.

Self-check: Name three tasks that require encoder-decoder architecture and explain why.

Connects to: Attention, Transformers, Machine Translation

Previous LectureLecture 12 · Recurrent Neural Networks — Foundations and Architecture

Next LectureLecture 14 · Attention Mechanisms

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.