Deep Neural Networks

Attention Mechanisms

[Published] Published: 2026-07-15

[Level] Level: postgraduate

[Audience] Audience: Postgraduate students in Deep Neural Networks

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Recurrent Neural Networks — Architecture and Training — covered in Lecture 12
Backpropagation Through Time (BPTT) — covered in Lecture 13
Vanishing and Exploding Gradients in RNNs — covered in Lecture 13
GRU and LSTM Architectures — covered in Lecture 13

Attention Mechanisms

Encoder-Decoder Foundations

Hook: You can glance at a photo and describe it in one word or a hundred. But a standard neural network demands one output slot for every input slot. How do you build a network that can read a sentence of any length and write a response of a different length?

Motivation and Applications

Intuition + Analogy: Think of a simultaneous translator at the UN. They hear a sentence in French, process it fully in their head, and only then speak the English translation. The translator cannot produce word-for-word in lockstep because the languages order words differently. The listener phase (encoder) absorbs the source. The speaker phase (decoder) produces the target. One reads. The other writes. The two phases can take different amounts of time — which is exactly what encoder-decoder models do.

Where the analogy breaks: The human translator actively references the source throughout translation — something the basic encoder-decoder cannot do. That limitation is the whole reason attention was invented.

A machine does not see words, images, or speech. It sees numbers. Every input — a sentence, an image, an audio clip — must first turn into a numerical representation. A neural network then extracts features from these numbers.

Several important tasks need this capability. You might want to classify a sentence into topics. You might need to translate a sentence into another language. You might want a model that, given prior words, predicts the next word and generates free-flowing text. In all of these, the system must first understand the words, their order, and the context they create together.

In recurrent neural networks (RNNs), the unfolded version processes one input at each time step. The hidden state at the final time step carries a summary of the entire sequence. This hidden state is a compressed essence of all inputs seen so far.

Computer vision has similar needs. Given an image, a machine should understand its content and produce a caption — a sequence of words describing the image. Again, the image turns into a feature map (a set of numbers). The caption turns into a sequence of token representations.

All of these tasks share one structure: an input of some length, and an output of a possibly different length. The input must be read and compressed. The output must be generated from that compression. These two phases need separate components. That separation gives you the encoder-decoder architecture.

Real-world: These architectures power machine translation (Google Translate), image captioning, text summarization, and question answering systems.

Architecture and Components

An encoder is the part of the network that reads the input and compresses it into an essence. A decoder is the part that takes that essence and generates the output. These are just names — any neural network can serve as an encoder or decoder depending on the task.

You need an encoder-decoder when input and output lengths differ. A standard RNN can tag each word of a sentence with its part of speech. That is a same-length task — one output per input. The output layer sits directly on top. But what if the output is shorter or longer than the input? You cannot line up output units with input units one-to-one.

The solution: separate the reading phase from the generation phase. The encoder reads and compresses. The decoder takes the compression and generates output of any length needed.

Tasks that use this pattern include question answering and text summarization. In question answering, the input is a question plus a context document. The output is an answer of variable length. In text summarization, the input is a long document and the output is a short abstract.

Mathematical Formulation

Formalize — Encoder

The encoder processes a sequence of tokens $X_{1}, X_{2}, \dots, X_{T}$ . At each time step $t$ , it takes the current input $X_{t}$ . It also takes the previous hidden state $H_{t - 1}$ . These produce a new hidden state $H_{t}$ .

H_{t} = f (W_{H X} X_{t} + W_{H H} H_{t - 1} + b)

Let's walk through this term by term.

$X_{t} \in R^{d_{x}}$ is the input token at time $t$ — a vector such as a word embedding.
$W_{H X} \in R^{d_{h} \times d_{x}}$ is the matrix connecting the input to the hidden layer. It tells the RNN how to transform the incoming token.
$W_{H H} \in R^{d_{h} \times d_{h}}$ is the matrix connecting the previous hidden state to the current one. This is the recurrent connection — it carries information across time.
$b \in R^{d_{h}}$ is the bias term, shared across all time steps. Every RNN time step uses the same parameters.
$f$ is a non-linear activation function, like $tanh$ .
$H_{t} \in R^{d_{h}}$ is the hidden state at time $t$ . It encodes the network's memory of tokens $X_{1}$ through $X_{t}$ .

After processing all $T$ input tokens, the final hidden state $H_{T}$ becomes the context vector $C$ . This single vector tries to summarize the entire input sequence.

C = H_{T}

Symbol Registry — Encoder.

Symbol	Meaning	Dimension
$H_{t}$	Hidden state at time $t$	$R^{d_{h}}$
$W_{H X}$	Input-to-hidden weight matrix	$R^{d_{h} \times d_{x}}$
$W_{H H}$	Hidden-to-hidden weight matrix	$R^{d_{h} \times d_{h}}$
$b$	Bias term (shared across time)	$R^{d_{h}}$
$f$	Non-linear activation function	—
$T$	Number of input time steps	Integer

Formalize — Decoder

The decoder takes the context vector $C$ and generates output tokens one by one. At each decoder time step $t$ , the hidden state $H_{t}^{(d)}$ depends on three pieces of information.

H_{t}^{(d)} = f (W_{H H}^{(d)} H_{t - 1}^{(d)} + W_{H Y} Y_{t - 1} + W_{H C} C + b^{(d)})

Break this down:

$H_{t - 1}^{(d)}$ is the decoder's own previous hidden state. This carries the decoder's running memory of what it has generated so far.
$Y_{t - 1}$ is the previous output token (either the ground truth during training, or the model's own prediction during inference).
$C$ is the encoder's context vector — the fixed summary of the entire input. Notice that $C$ is the same for every decoder time step in this basic formulation. That is the bottleneck that attention will later fix.
$W_{H H}^{(d)}, W_{H Y}, W_{H C}$ are the decoder's weight matrices. The superscript $(d)$ distinguishes decoder parameters from encoder parameters.

The output at each decoder step passes through a softmax over the vocabulary of size $V$ .

P (Y_{t} ∣ Y_{< t}, C) = softmax (W_{O} H_{t}^{(d)})

The matrix $W_{O} \in R^{V \times d_{h}}$ is the unembedding weight matrix. It projects the decoder hidden state $H_{t}^{(d)}$ from the hidden dimension $d_{h}$ up to vocabulary-size probabilities. The word with the highest probability is the model's predicted output for this step.

The embedding dimension $d_{x}$ is a hyperparameter you choose. The weight matrices bridge any dimension mismatch between the embedding size and the hidden state size.

Worked Example — Tiny Encoder Trace

Suppose you have a 3-word input: "cat sat mat". The embedding dimension is $d_{x} = 4$ and the hidden dimension is $d_{h} = 3$ . The activation is $tanh$ . Let the token embeddings be.

X_{1} = [0.5, 0.1, - 0.3, 0.2]

X_{2} = [0.2, - 0.4, 0.1, 0.5]

X_{3} = [- 0.1, 0.3, 0.4, - 0.2]

And let all weight matrices be initialized as identity-like, for simplicity.

W_{H X} = 100010001000

W_{H H} = 0.5 00 0 0.5 0 00 0.5

b = [0, 0, 0]

Initial hidden state: $H_{0} = [0, 0, 0]$ .

Step 1: $t = 1$

W_{H X} X_{1} = [0.5, 0.1, - 0.3], W_{H H} H_{0} = [0, 0, 0]

H_{1} = tanh ([0.5, 0.1, - 0.3] + [0, 0, 0] + [0, 0, 0]) = [0.46, 0.10, - 0.29]

Step 2: $t = 2$

W_{H X} X_{2} = [0.2, - 0.4, 0.1], W_{H H} H_{1} = [0.23, 0.05, - 0.15]

H_{2} = tanh ([0.2, - 0.4, 0.1] + [0.23, 0.05, - 0.15]) = tanh ([0.43, - 0.35, - 0.05]) = [0.41, - 0.34, - 0.05]

Step 3: $t = 3$

W_{H X} X_{3} = [- 0.1, 0.3, 0.4], W_{H H} H_{2} = [0.20, - 0.17, - 0.02]

H_{3} = tanh ([- 0.1, 0.3, 0.4] + [0.20, - 0.17, - 0.02]) = tanh ([0.10, 0.13, 0.38]) = [0.10, 0.13, 0.36]

The context vector is $C = H_{3} = [0.10, 0.13, 0.36]$ . This 3-dimensional vector is the RNN's summary of the entire 3-word sentence. The decoder will use this single vector to generate every output word.

Sense-check: The hidden state evolves smoothly — no single token dominates. The final context captures a blend of all three positions, weighted by recency.

Why Standard RNNs Fail for Sequence-to-Sequence Tasks

A standard RNN works well for many-to-one tasks. Given a sentence of many words, the final hidden state can classify sentiment or topic. That is a single prediction from one compressed vector.

But sequence-to-sequence tasks need many outputs from one compressed vector. The thing coming out of the encoder's last node — the context — must now drive the generation of every output word. For long output sequences, the influence of the context fades as you move through decoder time steps. The first decoder word gets the context directly. The tenth decoder word barely feels it. This is one fundamental problem.

Another problem: the encoder might forget. Even with LSTM or GRU cells designed to hold long-term memory, there is a limit. By the time the encoder reaches later input tokens, earlier tokens may have faded from the context vector. Suppose the answer sits in the first few words of a long document. The context vector may not carry that information at all.

A third problem: uniform treatment. The standard encoder gives equal weight to every input token when forming the context. But for answering a specific question, some words matter more than others. The word "1947" in "India gained independence in 1947" should carry more weight than "the". Consider the question: "What year was independence"? A single fixed context cannot express this selective importance.

Scope: The encoder-decoder architecture makes exactly one assumption: that the input can be compressed into a fixed-size vector without losing essential information. This holds for short sequences (up to roughly 20–30 tokens). It breaks when:

The input sequence is long and the fixed vector cannot hold everything.
The relationship between source and target tokens is non-local. For example, a word at position 2 in the source maps to position 50 in the target.
The output needs to selectively reference specific input tokens at different decoder steps.

When these assumptions fail, we need attention — the subject of later sections.

Visual Intuition: Picture a diagram with two rectangular boxes. The left box contains a horizontal chain of circles — the encoder, reading tokens one by one from left to right. A single arrow exits from the rightmost circle carrying the context vector $C$ to the right box. The right box contains another horizontal chain — the decoder, generating output tokens one by one. The context vector $C$ feeds into every decoder circle. Its arrow grows thinner with each step. This symbolizes its fading influence. The key takeaway: one fixed vector must serve every generation step, and it grows stale.

Pitfalls:

Confusing encoder and decoder roles. The encoder reads, the decoder writes. They are separate networks with separate parameters. Do not mix up their weight matrices.
Thinking the decoder must have the same number of time steps as the encoder.

The decoder runs for as many steps as the target sequence needs. A 3-word input can produce a 10-word output.

Assuming the context vector is an input embedding.

The context vector is the encoder's final hidden state, not one of the original input tokens. It is a fully processed summary.

Forgetting that the decoder's output at step $t$ depends on its own previous output $Y_{t - 1}$ .

This auto-regressive loop makes generation possible. But it also makes training tricky. See teacher forcing in Section 14.3.

Q: Why not just use a regular RNN for tasks like translation? A: A standard RNN needs the same number of output units as input units. In tasks like image captioning, a single image can produce a two-word caption or a hundred-word paragraph. You cannot fix the number of output units in advance. Splitting the network into a reader (encoder) and a generator (decoder) solves this.

Recap: The encoder-decoder splits reading from generation. The encoder compresses the input into a context vector. The decoder uses that context to produce output tokens one by one. But the fixed-size context is a bottleneck. It cannot store everything. It treats all inputs equally. And it fades over long generations. Next we look at exactly what this bottleneck is and why it fails.

Real-World & Domain Connection: Encoder-decoder architectures are the backbone of Google Translate's original neural machine translation system (GNMT). The encoder ingests a sentence in the source language. The decoder produces the translation. The same architecture underpins speech recognition — the encoder processes raw audio frames, the decoder outputs the recognized text. In 2014, Sutskever et al. showed that a single LSTM-based encoder-decoder could translate English to French with surprising fluency. It used just a fixed-size context vector. But performance degraded severely on sentences longer than 30 words. This degradation was the direct motivation for Bahdanau's attention mechanism.

The Context Vector and Bottleneck

Hook: A single 12-dimensional vector must store everything about a 100-word sentence — the names, dates, relationships, and intent. Can it? Spoiler: it cannot, and that failure is why attention exists.

What the Context Vector Is

Intuition + Analogy: Imagine taking a 500-page novel and writing a one-sentence summary on a sticky note. Then give that sticky note to someone and ask them to rewrite the entire novel. The sticky note captures the big idea. But it loses the character names, subplots, and the exact wording of key scenes. This is the context vector's problem: it is a sticky-note summary asked to reconstruct a novel.

The context vector is like a zipped file. It must be losslessly decompressible. But a fixed-size zip can only hold so much. For short sentences, the zip is big enough. For long ones, information gets thrown away.

The context vector is the hidden state of the encoder at the final time step. It is a fixed-size vector of numbers — for example, a vector of dimension 12. It tries to capture everything about the input sequence.

Think of the context vector as the summary you would write after reading a paragraph. A good summary captures the key points. A short summary fitted into a fixed-size box will always lose some information. The context vector faces the same constraint — it is a bottleneck.

Formalize: The context vector $C$ is defined as:

C = H_{T} = f (W_{H X} X_{T} + W_{H H} H_{T - 1} + b)

where $H_{T} \in R^{d_{h}}$ is the hidden state after the encoder has processed all $T$ input tokens. This single vector is the only channel through which information flows from encoder to decoder. The decoder uses $C$ at every generation step:

H_{t}^{(d)} = f (W_{H H}^{(d)} H_{t - 1}^{(d)} + W_{H Y} Y_{t - 1} + W_{H C} C + b^{(d)})

The term in red is the only connection to the input. If $C$ lacks critical information, the decoder has no way to recover it. Symbol Registry — Section 14.2

Three Key Bottleneck Issues

Symbol	Meaning	Dimension
$H_{T}$	Encoder hidden state at final time step	$R^{d_{h}}$
$d_{h}$	Hidden state dimension	Integer
$T$	Number of input time steps	Integer
$H_{t}^{(d)}$	Decoder hidden state at time $t$	$R^{d_{h}}$
$W_{H C}$	Context-to-hidden weight matrix (decoder)	$R^{d_{h} \times d_{h}}$
$Y_{t - 1}$	Previous decoder output token	$R^{d_{y}}$

Fixed capacity. The context vector has a fixed dimension $d_{h}$ (e.g., 12). You choose this dimension when designing the network. If the input has complex, long-range dependencies, this fixed vector may be too small to hold everything needed. The capacity is set at design time and does not grow with the input length.

Information loss over long sequences. As the decoder generates more output words, the context vector's influence weakens. Early decoder steps get fresh context. Later steps get a diluted version propagated through many hidden states. Tokens from the start of the input are especially at risk of being forgotten.

Uniform encoding. All input tokens contribute equally to the context. There is no mechanism to say "this word matters more than that word".

Take a question-answering example: "India gained independence in 1947, Gandhi led the movement. What year was independence?" The context should spotlight "1947" when generating the answer. A uniform context cannot do this.

Failure with Long Sequences

The problem gets worse as the sequence length grows. Beyond 30 to 50 tokens, even LSTM cells struggle. Later tokens dominate the context. Earlier tokens fade. The decoder tends to focus on what came last. It is much like a student remembering the final lectures more vividly than the pre-midterm ones.

Worked Example — Sentiment Bottleneck

Consider a product review: "The screen flickers, battery drains fast, however it is good, very fast, but very disappointed, not recommended."

This review has 19 words. A fixed-size context vector (say, dimension 8) must compress all of them into 8 numbers. The negative signals — "flickers", "drains fast", "disappointed", "not recommended" — are diluted with positive ones — "good", "very fast". The context blends everything into a neutral muddle.

Now imagine classifying this review. The words "not recommended" should dominate. But in a uniform context, all 19 words get equal say. The signal-to-noise ratio drops as the sequence grows. A shorter review like "Not recommended, terrible product" would be classified correctly. The fixed vector can hold 4 words of strong signal. But 19 words with mixed sentiment confuse it.

Sense-check: This is why review classifiers often fail on nuanced, long reviews. The context vector acts as a lossy compressor whose fidelity drops with input length.

Real-world: The Sutskever et al. (2014) paper identified this fixed-bottleneck problem in sequence-to-sequence models. They showed that translation quality degrades sharply when sentences exceed 30 tokens.

Scope: The bottleneck affects any task where input length exceeds roughly 30 tokens. Short-sentence translation and classification work fine with a fixed context. The bottleneck also matters more when the input contains fine-grained information — numbers, names, dates — that cannot be approximated or summarized. A context vector can summarize the "gist" of a long paragraph. But it cannot preserve the exact number "1947". This is especially true if that number appeared in word position 2 of a 100-word document.

Visual Intuition: Imagine a funnel. The wide top opening receives all input tokens. The narrow spout at the bottom is the context vector dimension $d_{h}$ . Pour 5 tokens through and most liquid gets through (good compression). Pour 100 tokens through and the funnel overflows — information spills out and is lost. The visual shows the funnel with different input sizes. For 5 inputs the flow is clean. For 100 inputs, tokens spill over the sides.

Pitfalls:

Thinking bigger $d_{h}$ solves everything. Doubling the context size from 12 to 24 helps, but the fundamental problem remains. For very long sequences (200+ tokens), even a dimension-1024 context vector will lose information. The problem is architectural, not just about capacity.
Assuming LSTM/GRU fix the bottleneck. Gating mechanisms help with vanishing gradients, but they do not solve the fixed-capacity problem. The final hidden state is still one vector.
Confusing context vector with word embeddings. The context vector is a processed internal state, not an input representation. It lives in the RNN's hidden space, not the embedding space.
Overlooking the recency bias. Even symmetrically-designed RNNs favor later tokens. If the critical information is at the beginning of the sequence, expect it to be under-represented in the context.

Recap: The context vector is a fixed-size sticky-note summary. It creates three problems: fixed capacity, fading influence over long outputs, and uniform treatment of all inputs. These three failures together make the case for dynamic, selective attention — which we build toward in Section 14.4.

Real-World & Domain Connection: This bottleneck was the direct motivation for Bahdanau et al. (2014), who proposed the first attention mechanism. Their experiments on English-to-French translation showed that the fixed-context model performed well on short sentences (under 20 words). But BLEU scores dropped sharply on longer ones (30-50 words). The attention-based model maintained quality across all lengths. It let the decoder "look back" at specific encoder states. This replaced relying on a single summary. This insight — that dynamic, selective access beats static compression — now underpins every modern sequence model from BERT to GPT-4.

Teacher Forcing

Hook: What if your GPS recalculated the route from wherever you ended up — even if you missed the last five turns? That compounding error is exactly what happens when a decoder trains on its own mistakes.

Training vs Inference

Intuition + Analogy: A parent teaching a child to read does not let the child guess every word independently. If the child stumbles on the third word, the parent corrects it before the child reads the fourth word. That way, the child's attempt at the fourth word starts from the right place. This is teacher forcing. During training, you give the model the correct answer at each step. It never practices on a cascade of its own errors.

Where the analogy breaks: In real life, the child eventually must read alone, making mistakes and recovering. During inference, the decoder does exactly this — it feeds its own predictions forward. So the decoding strategy at test time is completely different from training. The model must learn from corrected contexts but perform with its own outputs.

During inference, the decoder uses its own previous prediction as input for the next time step. This is autoregressive generation — each output feeds into the next.

During training, this approach is dangerous. If the model predicts wrongly at step 1, that wrong prediction feeds into step 2, which then compounds the error. The errors cascade through time.

Formalize: Let $Y_{t}$ be the target output token at time $t$ and $Y^_{t}$ be the model's prediction.

Without teacher forcing (free-running): The decoder input at step $t + 1$ is $Y^_{t}$ . The hidden state update is:

H_{t + 1}^{(d)} = f (W_{H H}^{(d)} H_{t}^{(d)} + W_{H Y} Y^_{t} + W_{H C} C + b^{(d)})

With teacher forcing: The decoder input at step $t + 1$ is the ground truth $Y_{t}$ :

H_{t + 1}^{(d)} = f (W_{H H}^{(d)} H_{t}^{(d)} + W_{H Y} Y_{t} + W_{H C} C + b^{(d)})

The loss is still computed between $Y^_{t + 1}$ and $Y_{t + 1}$ . But the input for the next step is always the correct token $Y_{t}$ , not the prediction.

Symbol Registry — Section 14.3

Symbol	Meaning	Dimension
$Y^_{t}$	Model's predicted token at time $t$	Probability distribution, $R^{V}$
$H_{t}^{(d)}$	Decoder hidden state at time $t$	$R^{d_{h}}$
$W_{H Y}$	Output-to-hidden weight matrix (decoder)	$R^{d_{h} \times d_{y}}$
$C$	Context vector from encoder	$R^{d_{h}}$

The solution: teacher forcing. During training, you do not feed the model's prediction to the next time step. Instead, you feed the ground truth — the correct word from the training data. The model still makes its prediction, and you compare it against the ground truth using a loss function. But the next step receives the actual correct word, not the model's guess.

Why Teacher Forcing Matters

If the model trained on its own incorrect predictions, errors would propagate. The model would drift further from the ground truth with each step. Teacher forcing keeps the model on track during training. It sees the right context at each step, so backpropagation can assign credit correctly.

At every time step, you compare the predicted output against the true word. The predicted output is a probability distribution over the vocabulary. You compare using categorical cross-entropy. You sum the losses across all time steps, compute the gradient, and backpropagate through time.

Worked Example — Error Cascade Without Teacher Forcing

Consider a 3-word target sequence: "The cat sat". Vocabulary size is 5. At time step 1, the correct target is "The".

Without teacher forcing:

Step 1: Input is <start> token. Model predicts "A" (wrong) with 60% confidence. Loss computed against "The".
Step 2: Input is "A" (the wrong prediction). Model, now confused, predicts "dog" (wrong). Loss against "cat".
Step 3: Input is "dog". Model predicts "ran" (wrong). Loss against "sat".

The model never saw the correct input "The" → "cat" transition. Three wrong predictions compound, and the gradient backpropagates through a chain of errors.

With teacher forcing:

Step 1: Input is <start>. Model predicts "A" (wrong). Loss computed against "The". Next input is "The" (ground truth).
Step 2: Input is "The" (correct). Model predicts "feline" (wrong). Loss against "cat". Next input is "cat" (ground truth).
Step 3: Input is "cat" (correct). Model predicts "sat" (correct — 90% confidence). Loss against "sat".

Even though the model made mistakes at steps 1 and 2, each subsequent step started from the correct context. The gradient at step 2 correctly reflects the error from seeing "The" and predicting "feline". It does not reflect the compounded error of seeing "A" and guessing randomly.

Sense-check: Teacher forcing ensures that each prediction error is isolated. The loss at step $t$ reflects only one thing. It is the model's failure to predict $Y_{t}$ from the correct history $Y_{1}, \dots, Y_{t - 1}$ .

Scope: Teacher forcing assumes you have access to the ground truth target sequence during training. This holds for supervised tasks like translation. You have the parallel corpus. It breaks when you want to train on unsupervised generation where no target exists. It also creates a train-test mismatch. The model never encounters its own errors during training. So at inference time it can be brittle. Techniques like scheduled sampling (occasionally feeding the model its own predictions during training) attempt to bridge this gap.

Visual Intuition: Picture two parallel timelines. The top timeline is teacher forcing: each decoder step receives a green checkmark (correct token) as input. The model's wrong guess appears as a red X above, but the red X never flows forward. The bottom timeline is free-running. The red X at step 1 flows into step 2. This creates a bigger red X, which flows into step 3. The top timeline is clean. The bottom is a cascade of growing errors.

Pitfalls:

Forgetting that teacher forcing is training-only. During inference, you never have the ground truth. The decoder must use its own predictions. This train-test gap is a known weakness of pure teacher forcing.
Confusing the input and the loss target. Teacher forcing changes the input to the next time step, not the loss. You still compute loss against the ground truth at every step.
Thinking teacher forcing prevents all training issues. It fixes the error-cascade problem but does not fix the bottleneck problem (Section 14.2). Even with perfect teacher forcing, a fixed context vector limits long-sequence quality.
Applying teacher forcing to bidirectional encoders incorrectly. Teacher forcing applies to the decoder only. The encoder processes the full input sequence in one pass and does not need teacher forcing.

Recap: Teacher forcing feeds ground-truth tokens as decoder inputs during training to prevent error cascading. It keeps training stable but creates a mismatch with inference, where the model must use its own predictions. This technique is a training strategy — not an architectural change — so the bottleneck problem from Section 14.2 remains. Next we introduce attention, the architectural fix.

Real-World & Domain Connection: Teacher forcing is used universally in training sequence-to-sequence models. It spans speech recognition (Whisper), machine translation (GNMT), and image captioning (Show, Attend and Tell). The standard practice is to use teacher forcing during training and autoregressive generation during inference. Beam search and nucleus sampling are common choices. The seminal "Sequence to Sequence Learning with Neural Networks" paper (Sutskever et al., 2014) popularized teacher forcing. It used it for LSTM-based translation models.

Introduction to Attention Mechanisms

Hook: The encoder-decoder gives every decoder step the same context vector — a blurry photograph of the entire input. What if, instead, the decoder could request a fresh, zoomed-in snapshot of whichever part of the input matters right now? That is attention.

Why Attention Is Needed

Intuition + Analogy: A student answers a question about a textbook chapter. They do not re-read the entire chapter word-for-word. They scan for the relevant paragraph, focus on it, and pull the answer from those words. Different questions need different paragraphs. A single fixed summary of the chapter would fail. Question 1 needs paragraph 3. Question 2 needs paragraph 7.

The decoder is the student. The input sequence is the textbook. The context vector is the one-paragraph summary. Attention lets the decoder "scan" the input and focus on the relevant parts for each output word. When generating the French word for "feet", attend to the English word "feet". When generating the French word for "hurt", attend to "hurt".

Where the analogy breaks: The student knows the question before scanning. In attention, the query (what the decoder needs) and the keys (what to scan) are learned representations, not natural language questions. The "relevance" is a dot product between vectors, not semantic understanding.

The context vector bottleneck creates a single point of failure. Three specific weaknesses drive the need for something better:

The context is given only to the first decoder time step. By the hundredth step, its influence is lost.
The context treats all input tokens equally. Selective importance is missing.
The fixed-size vector limits how much information can pass through.

The first solution attempt: pass a copy of the context to every decoder time step, like a residual connection. This helps with information propagation but does not solve the uniform-weighting problem.

The real solution: make the context dynamic. At each decoder time step, compute a fresh, customized context vector. This vector should weigh input tokens differently depending on what the decoder is trying to generate right now.

Dynamic Context Vectors

Formalize: Instead of one fixed context $C$ , compute $C_{t}$ for each decoder time step $t$ . Each $C_{t}$ is a weighted sum over all encoder hidden states $H_{1}, H_{2}, \dots, H_{T}$ :

C_{t} = i = 1 \sum T α_{t i} H_{i}

where $α_{t i}$ is the attention weight — a scalar between 0 and 1. It says how much the decoder at step $t$ should focus on encoder position $i$ . The weights sum to 1:

i = 1 \sum T α_{t i} = 1

These weights are computed fresh at every decoder step, based on:

The decoder's current state (what it has generated so far)
Each encoder hidden state (what is available in the input)

The attention mechanism itself is a function that computes $α_{t i}$ from the decoder state $S_{t - 1}$ and each encoder state $H_{i}$ :

α_{t i} = softmax (score (S_{t - 1}, H_{i}))

The softmax ensures the weights are non-negative and sum to 1. The scoring function measures compatibility between the decoder's need and each input position. Common scoring functions are covered in Section 14.8.

Symbol Registry — Section 14.4

Symbol	Meaning	Dimension
$α_{t i}$	Attention weight from decoder step $t$ to encoder position $i$	Scalar, $[0, 1]$
$H_{i}$	Encoder hidden state at position $i$	$R^{d_{h}}$
$S_{t}$	Decoder hidden state at time $t$	$R^{d_{h}}$
$T$	Number of encoder positions	Integer
$T^{'}$	Number of decoder time steps	Integer

These weights are learned. The network learns which input tokens to attend to, and how strongly, for each decoding step. This is the core idea of attention: selective, learned weighting of the input to produce a customized context.

Historical Development

The sequence-to-sequence bottleneck was identified by Sutskever et al. (2014). The attention solution was introduced by Bahdanau et al. (2014) for machine translation. In the Bahdanau model, the decoder's previous hidden state acts as the query. The encoder's hidden states serve as both keys and values. The attention weights are computed using an additive scoring function (a small feedforward network).

In 2017, Vaswani et al. took attention further. They introduced self-attention — a form where the query, key, and value all come from the same sequence. They also proposed multi-head attention, scaled dot-product scoring, and completely removed recurrence. This led to the Transformer architecture, which now dominates AI.

Worked Example — Dynamic Context in Action

Consider translating "my feet hurt" → French. The encoder produces 3 hidden states: $H_{1}$ (for "my"), $H_{2}$ ("feet"), $H_{3}$ ("hurt").

At decoder step 1 (generating "j'" — short for "je"):

The decoder state is queried against each $H_{i}$ .
Scores: $H_{1}$ gets 0.6, $H_{2}$ gets 0.3, $H_{3}$ gets 0.1.
Context: $C_{1} = 0.6 H_{1} + 0.3 H_{2} + 0.1 H_{3}$ — mostly "my".

At decoder step 2 (generating "ai" — from "avoir"):

Scores shift: $H_{1}$ gets 0.2, $H_{2}$ gets 0.1, $H_{3}$ gets 0.7.
Context: $C_{2} = 0.2 H_{1} + 0.1 H_{2} + 0.7 H_{3}$ — mostly "hurt".

At decoder step 3 (generating "mal"):

Scores shift again: $H_{1}$ gets 0.1, $H_{2}$ gets 0.8, $H_{3}$ gets 0.1.
Context: $C_{3} = 0.1 H_{1} + 0.8 H_{2} + 0.1 H_{3}$ — mostly "feet".

Sense-check: Each output word gets a custom context emphasizing the relevant input word. The decoder is not stuck with a single summary — it queries the input afresh at every step. The weights $α_{t i}$ tell the story: step 1 looked at "my", step 3 looked at "feet".

Real-world: Attention powers BERT, GPT, Vision Transformers (ViT), Flamingo (multimodal), and Whisper (speech processing). It is the dominant mechanism across NLP, vision, and speech.

Scope: Dynamic context vectors solve the uniform-weighting and information-propagation problems. But they introduce a new cost: computing attention weights for all (decoder step, encoder position) pairs. For a source of length $T$ and target of length $T^{'}$ , this costs $O (T \times T^{'})$ operations. For short sequences this is negligible; for very long sequences (thousands of tokens) it becomes the computational bottleneck. Modern solutions include sparse attention, linear attention, and FlashAttention.

Visual Intuition: Picture a heatmap. Decoder steps are on the y-axis (1, 2, 3, ...). Encoder positions are on the x-axis (1, 2, 3, ...). At each decoder row, some columns glow brightly (high attention) and others are dim (low attention). The bright spots shift as you go down the rows, tracing a path through the input. For a well-aligned translation, the bright spots fall on a diagonal. This heatmap is exactly what Section 14.11 explores.

Pitfalls:

Confusing attention weights with model parameters. Attention weights $α_{t i}$ are computed fresh for every input at every decoder step. The model learns the scoring function (how to compute $α_{t i}$ ), not the weights themselves. After training, $W_{Q}, W_{K}, W_{V}$ are fixed, but $α_{t i}$ changes for every new input.
Thinking attention completely replaces the context vector.

In Bahdanau attention, the context vector still exists — it is now $C_{t}$ instead of $C$ . The decoder still uses it. The difference is that $C_{t}$ is dynamic and weighted, not fixed and uniform.

Assuming attention weights are interpretable.

While heatmaps often look plausible, research shows attention weights are not always faithful explanations of model behavior (Jain & Wallace, 2019). Use them as hints, not proofs.

Overlooking the encoder. Attention weights are over all encoder positions. Even with dropout, the decoder can attend to any position — including padding tokens. Masking (Section 14.8) prevents this.

Several students asked about the role of the context vector.

Q: Is the context vector from the encoder enough to generate variable-length outputs? A: Yes, from the perspective of what a hidden state carries. The hidden state coming from the encoder's last node is a set of numbers (say, dimension 3). That compressed essence can drive a decoder of any length. The issue is not whether it works at all — it does. The issue is quality: the fixed context loses information for long sequences and treats all inputs equally.

Q: What if the context vector has a different dimension than the word embeddings? A: The weight matrix $W_{O}$ (the unembedding weights) handles this. You design $W_{O}$ so that multiplying the decoder hidden state by it produces a vector. This vector has the same dimension as your word embeddings. The embedding dimension is a hyperparameter you set. The weight matrices bridge any dimension mismatch.

Recap: Attention replaces the fixed context vector with dynamic, weighted contexts $C_{t}$ computed fresh at each decoder step. The network learns to weigh input positions by relevance to the current generation step. This solves all three bottleneck problems but introduces a computational cost of $O (T \times T^{'})$ . Next we look under the hood at how attention computes these weights. We'll explore the query, key, and value framework.

Real-World & Domain Connection: The Bahdanau attention mechanism was the first to show NMT could match traditional statistical MT. It even exceeded it on long sentences. When Google deployed their Neural Machine Translation (GNMT) system in 2016, attention was the critical component. It let the decoder handle sentences of arbitrary length. Today, every major language model — GPT-4, Claude, Gemini — uses attention (specifically multi-head self-attention inside Transformers). The dynamic weighting idea also extends beyond text. Vision Transformers use attention over image patches. AlphaFold uses attention over amino acid residue pairs to predict protein structure.

Query, Key, and Value Framework

Hook: A database lookup and a neural network layer have more in common than you would think. Both answer a question by searching over stored records and retrieving the best match. Attention is exactly that — a differentiable database lookup.

The Database Analogy

Intuition + Analogy: You walk into a library and ask the librarian: "Where can I find books on deep learning"? You are the query. The library catalog is a set of keys — topic labels for each shelf. The books themselves are the values — the content you actually want. The librarian matches your query against the catalog keys, finds the best match, and retrieves the corresponding books.

In attention, the query is a vector representing what you want right now (e.g., the decoder's current state). The keys are vectors representing what is available (e.g., each encoder hidden state). The values are the actual content to blend together. The matching is done by dot product, and the retrieval is a weighted sum — not a hard pick-one.

Where the analogy breaks: In a library, you get exactly one shelf's books (hard selection). In attention, you get a weighted blend of all values, with more weight on the best matches. This soft, differentiable selection is what lets backpropagation train the system end-to-end.

Step away from neural networks for a moment. Think about looking up information in a database.

You have a question: "What is the national capital?" This is your query. The database stores records. Each record has an identifier — a key — and the actual data — a value. Keys might be "Mumbai", "Delhi", "Bangalore". The value for "Delhi" might be detailed information about the city.

To answer the query, you measure similarity between the query and each key. A simple way is a dot product if both are vectors of the same dimension. A high score means the key is relevant to the query. A low or negative score means it is not.

The key with the highest similarity score wins. You retrieve its value. That is your answer.

This is the query-key-value (QKV) framework at its core.

Each Token Plays Three Roles

In NLP, every word can act as a query, a key, and a value. Sometimes all three roles apply in different parts of the computation.

Consider a sequence of words $W_{1}, W_{2}, W_{3}, W_{4}$ . At a decoder time step, suppose $W_{3}$ is the previous generated word. Then $W_{3}$ acts as a query — "what should come next"? In the same step, words $W_{1}, W_{2}, W_{3}, W_{4}$ act as keys. They say: here are the available things to match against. The hidden states of those words act as values — "here is the content to retrieve if matched."

The same word $W_{3}$ can be a query in one context and a key in another. It can also be a value if its hidden state is part of the weighted sum. This triple role is fundamental to how self-attention works.

Query, Key, and Value Projections

Formalize: Each input token's embedding $X_{i} \in R^{d}$ is multiplied by three learned weight matrices to produce three distinct vectors:

Q_{i} = X_{i} W_{Q}, K_{i} = X_{i} W_{K}, V_{i} = X_{i} W_{V}

Where:

$W_{Q} \in R^{d \times d_{k}}$ projects $X_{i}$ into the query space. The query says "here is what I am looking for."
$W_{K} \in R^{d \times d_{k}}$ projects $X_{i}$ into the key space. The key says "here is how you can find me."
$W_{V} \in R^{d \times d_{v}}$ projects $X_{i}$ into the value space. The value says "here is my actual content."

Note that $d_{k}$ (query/key dimension) and $d_{v}$ (value dimension) need not equal the embedding dimension $d$ . The projection matrices handle the dimension change. In many implementations, $d_{k} = d_{v} = d$ , but using a smaller $d_{k}$ can save compute.

These matrices are learned during training. They project the same token embedding into three different spaces. One is for querying. One is for being matched against. One is for providing content.

Symbol Registry — QKV Projections

Symbol	Meaning	Dimension
$W_{Q}$	Query projection matrix	$R^{d \times d_{k}}$
$W_{K}$	Key projection matrix	$R^{d \times d_{k}}$
$W_{V}$	Value projection matrix	$R^{d \times d_{v}}$
$Q_{i}$	Query vector for token $i$	$R^{d_{k}}$
$K_{i}$	Key vector for token $i$	$R^{d_{k}}$
$V_{i}$	Value vector for token $i$	$R^{d_{v}}$
$d_{k}$	Dimension of query/key vectors	Integer
$d_{v}$	Dimension of value vectors	Integer

The full attention computation:

Given a query $Q$ , keys $K_{1}, \dots, K_{m}$ , and values $V_{1}, \dots, V_{m}$ :

Compute compatibility scores: $score_{i} = \frac{Q \cdot K _{i}}{d _{k}}$ (scaled dot-product)
Normalize: $α_{i} = softmax (score_{1}, \dots, score_{m})_{i} = \frac{e ^{score_{i}}}{\sum _{j} e ^{score_{j}}}$
Aggregate: $Attention (Q, K, V) = \sum_{i = 1}^{m} α_{i} V_{i}$

The output is a weighted sum of value vectors, where weights are determined by query-key compatibility.

Worked Example — QKV for a 3-Word Sentence

Take the sentence "I love cats" with word embeddings $X_{1}, X_{2}, X_{3}$ each of dimension $d = 4$ . Let $d_{k} = 3$ and $d_{v} = 2$ . We want the attention output for position 2 ("love").

Projections (using simplified identity-like matrices).

W_{Q} = 100001000010

W_{K} = 010010000010

W_{V} = 10000100

With embeddings.

X_{1} = [1, 0, 0, 0]

X_{2} = [0, 1, 0, 0]

X_{3} = [0, 0, 1, 0]

Queries.

Q_{2} = X_{2} W_{Q} = [0, 1, 0]

Keys.

K_{1} = X_{1} W_{K} = [0, 1, 0]

K_{2} = X_{2} W_{K} = [1, 0, 0]

K_{3} = X_{3} W_{K} = [0, 0, 1]

Values.

V_{1} = X_{1} W_{V} = [1, 0]

V_{2} = X_{2} W_{V} = [0, 1]

V_{3} = X_{3} W_{V} = [0, 0]

Scores (scaled dot-product, $d_{k} = 3, 3 \approx 1.73$ ).

score_{1} = \frac{Q _{2} \cdot K _{1}}{3} = \frac{( 0 ) ( 0 ) + ( 1 ) ( 1 ) + ( 0 ) ( 0 )}{1.73} = \frac{1}{1.73} = 0.58

score_{2} = \frac{Q _{2} \cdot K _{2}}{3} = \frac{( 0 ) ( 1 ) + ( 1 ) ( 0 ) + ( 0 ) ( 0 )}{1.73} = 0

score_{3} = \frac{Q _{2} \cdot K _{3}}{3} = \frac{( 0 ) ( 0 ) + ( 1 ) ( 0 ) + ( 0 ) ( 1 )}{1.73} = 0

Softmax (approx). $α_{1} = 0.64, α_{2} = 0.18, α_{3} = 0.18$ (since $e^{0.58} \approx 1.79, e^{0} = 1, e^{0} = 1, sum = 3.79$ )

Output. $Attention = 0.64 [1, 0] + 0.18 [0, 1] + 0.18 [0, 0] = [0.64, 0.18]$

Sense-check: Position 2 ("love") attends most strongly to position 1 ("I"), moderately to itself and position 3. With these toy weights, the output blends mostly from position 1's value.

Scope: The QKV framework assumes you have learned projection matrices. At initialization (random weights), attention is uniform — all positions get roughly equal weight. As training progresses, the matrices learn to create meaningful queries and keys that produce sharp, selective attention distributions. This learning is what makes attention powerful. The framework also assumes the query and key share the same dimension $d_{k}$ for dot-product scoring. For different dimensions, use additive attention (Section 14.8.4).

Visual Intuition: Picture three parallel processing lanes for each token. The left lane multiplies by $W_{Q}$ producing query vectors (think: question marks). The middle lane multiplies by $W_{K}$ producing key vectors (think: address labels). The right lane multiplies by $W_{V}$ producing value vectors (think: the actual data packets). A routing layer computes dot products between queries and keys, then routes the values to the output in proportion to those scores. All three lanes start from the same token embedding but diverge through different learned matrices.

Pitfalls:

Thinking weight matrices $W_{Q}, W_{K}, W_{V}$ change per input. They are fixed after training. The attention weights $α_{i}$ change per input, but the projection matrices do not.
Confusing attention scores with attention weights. Scores are raw dot products (can be any real number). Weights are softmax-normalized scores (between 0 and 1, sum to 1). The distinction matters: large-dimensional dot products produce large scores that saturate softmax — which is why we need scaling.
Forgetting that Q and K must have the same dimension for dot-product scoring.

If they differ, use an intermediate matrix $M$ (i.e., score $Q^{⊤} M K$ ) or additive attention.

Assuming attention weights always reflect semantic similarity. With untrained random projections, attention is essentially random. The weights become meaningful only after the matrices $W_{Q}, W_{K}, W_{V}$ learn useful representations.

Several students asked about the nature of attention weights.

Q: How are attention weights different from regular neural network weights? A: Regular weights capture hierarchical features in the data. They learn which input features matter for which hidden units. In RNNs, the recurrent weights capture patterns across time. Attention weights are a third kind. They decide how much to focus on each input position. This focus is not based on fixed patterns. It depends on what the decoder needs at that exact moment. All three kinds of weights — feedforward, recurrent, and attention — are learned through backpropagation. But attention weights produce customized contexts that change per time step and per query.

Q: Do attention weights change dynamically after training? A: During training, the weight matrices $W_{Q}, W_{K}, W_{V}$ are learned and updated. Once training is done, these matrices are fixed. But the attention scores are dynamic. These scalar weights applied to values change for every new input. They change at every time step. They are computed fresh each time based on the specific tokens being processed.

Recap: The QKV framework casts attention as a differentiable database lookup. Every token projects into three spaces: queries (what to look for), keys (what to match against), and values (what to retrieve). Compatibility scores between queries and keys determine a weighted blend of values. This framework is the engine inside both cross-attention (Section 14.7) and self-attention (Section 14.6).

Real-World & Domain Connection: The QKV terminology comes directly from database systems and information retrieval. In those systems, queries match against indexed keys to retrieve stored values. Vaswani et al. (2017) formalized this for neural networks in "Attention Is All You Need". The framework is so general that it now appears beyond NLP. In Vision Transformers, image patches serve as keys and values. A learned "classification token" serves as the query. In Perceiver architectures (Jaegle et al., 2021), a small set of learned latent vectors act as queries. They attend over a large input array — pixels, audio frames, or point clouds. This drastically reduces the quadratic cost of self-attention.

Self-Attention

Hook: What if a sentence could read itself — each word looking at every other word to decide what it means? "It" glances at "animal" and knows its referent. "Bank" glances at "river" and knows it is not about money. This inward look is self-attention.

Definition

Intuition + Analogy: A group discussion where each person listens to everyone else — including themselves — before speaking. When it is your turn, you weigh what others said. The expert on the topic gets more weight. The off-topic comment gets less. Your response synthesizes the weighted contributions of the whole group. This is self-attention. Every position in a sequence creates a query about what it needs. It matches that query against keys from every other position. Then it blends their values to form a new representation.

In cross-attention, the discussion is between two separate groups — the encoder group and the decoder group. In self-attention, everyone is in the same room, talking to each other. That is why it is called "self."

Where the analogy breaks: In a real discussion, information flows sequentially. In self-attention, every position attends to every other position simultaneously. There is no time ordering — the computation is fully parallel.

Self-attention is a form of attention where the query, key, and value all come from the same sequence. There is no separate encoder and decoder. Every position in the sequence attends to every other position in the same sequence.

Why "self"? Because the sequence is attending to itself. While processing the third token, the query comes from token 3. But the keys and values come from all tokens — including token 3. The sequence looks inward to understand relationships between its own tokens.

This is the core building block of Transformers.

Step-by-Step Computation

Formalize: Given a sequence of $n$ token embeddings $X_{1}, X_{2}, \dots, X_{n}$ , each $X_{i} \in R^{d}$ . For every position $i$ , we want to compute an output $Z_{i}$ that is a weighted sum of all value vectors:

Z_{i} = j = 1 \sum n α_{ij} V_{j}

where the weights $α_{ij}$ are computed from queries and keys:

α_{ij} = softmax (\frac{Q _{i} \cdot K _{j}}{d _{k}})_{j}

and $Q_{i} = X_{i} W_{Q}$ , $K_{j} = X_{j} W_{K}$ , $V_{j} = X_{j} W_{V}$ .

Procedure for a single position $i$ :

Compute its query: $Q_{i} = X_{i} W_{Q}$ .
Compute keys for all positions: $K_{j} = X_{j} W_{K}$ for $j = 1, \dots, n$ .
Compute compatibility scores: $score_{ij} = \frac{Q _{i} \cdot K _{j}}{d _{k}}$ for each $j$ .
Normalize to attention weights: $α_{ij} = \frac{e ^{score_{ij}}}{\sum _{k = 1}^{n} e ^{score_{ik}}}$ .
Compute value vectors: $V_{j} = X_{j} W_{V}$ for all $j$ .
Produce weighted output: $Z_{i} = \sum_{j = 1}^{n} α_{ij} V_{j}$ .

Repeat for every position $i = 1, \dots, n$ . Each position gets its own set of attention weights from its own query.

Batch matrix form: Stack all queries into $Q = [Q_{1}; \dots; Q_{n}] \in R^{n \times d_{k}}$ . Stack all keys into $K \in R^{n \times d_{k}}$ . Stack all values into $V \in R^{n \times d_{v}}$ :

Self-Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

The softmax is applied row-wise. The output is an $n \times d_{v}$ matrix where each row is $Z_{i}$ .

Symbol Registry — Section 14.6

Symbol	Meaning	Dimension
$Z_{i}$	Self-attention output for position $i$	$R^{d_{v}}$
$α_{ij}$	Attention weight from position $i$ to $j$	Scalar, $[0, 1]$
$Q_{i}$	Query vector for position $i$	$R^{d_{k}}$
$K_{j}$	Key vector for position $j$	$R^{d_{k}}$
$V_{j}$	Value vector for position $j$	$R^{d_{v}}$
$W_{Q}, W_{K}, W_{V}$	Projection matrices	$R^{d \times d_{k}}, R^{d \times d_{k}}, R^{d \times d_{v}}$

Take a three-word input: $X_{1}, X_{2}, X_{3}$ . Suppose you want the output corresponding to $X_{3}$ .

Compute $Q_{3} = X_{3} W_{Q}$ . This is your query.
Compute $K_{1} = X_{1} W_{K}$ , $K_{2} = X_{2} W_{K}$ , $K_{3} = X_{3} W_{K}$ . These are your candidate keys.
Score each key against the query: $score (Q_{3}, K_{j}) = \frac{Q _{3} \cdot K _{j}}{d _{k}}$ .
These three scalar scores represent how similar each key is to the query. $X_{3}$ compared to itself will usually score highest. But context words like $X_{1}$ or $X_{2}$ may also score high if they are relevant.
Compute value vectors: $V_{1} = X_{1} W_{V}$ , $V_{2} = X_{2} W_{V}$ , $V_{3} = X_{3} W_{V}$ .
Convert scores to weights via softmax, then produce a weighted sum of value vectors: $Z_{3} = \sum_{j} α_{3 j} V_{j}$ .

The same procedure repeats for every position that needs an output. Each position gets its own set of attention weights, computed from its own query.

Worked Example — Scalar Score Computation

Suppose you have a query vector $Q$ and three key vectors $K_{1}, K_{2}, K_{3}$ . Each is 2-dimensional.

Q = [0.5, - 0.2]

K_{1} = [0.3, 0.8]

K_{2} = [0.9, 0.1]

K_{3} = [- 0.4, 0.6]

Step 1 — compute scaled dot-product scores (with $d_{k} = 2, 2 \approx 1.41$ ):

score_{1} = \frac{( 0.5 \times 0.3 ) + ( - 0.2 \times 0.8 )}{1.41} = \frac{0.15 - 0.16}{1.41} = \frac{- 0.01}{1.41} \approx - 0.007

score_{2} = \frac{( 0.5 \times 0.9 ) + ( - 0.2 \times 0.1 )}{1.41} = \frac{0.45 - 0.02}{1.41} = \frac{0.43}{1.41} \approx 0.305

score_{3} = \frac{( 0.5 \times - 0.4 ) + ( - 0.2 \times 0.6 )}{1.41} = \frac{- 0.20 - 0.12}{1.41} = \frac{- 0.32}{1.41} \approx - 0.227

Step 2 — apply softmax to get attention weights. First compute exponentials:

e^{- 0.007} \approx 0.993, e^{0.305} \approx 1.357, e^{- 0.227} \approx 0.797

sum = 0.993 + 1.357 + 0.797 = 3.147

α_{1} = \frac{0.993}{3.147} \approx 0.316

α_{2} = \frac{1.357}{3.147} \approx 0.431

α_{3} = \frac{0.797}{3.147} \approx 0.253

$K_{2}$ gets the highest weight (0.431). It is most relevant to the query. $K_{3}$ gets the lowest (0.253). The weights sum to 1.0.

Step 3 — multiply each attention weight by the corresponding value vector $V_{j}$ and sum:

Context = α_{1} V_{1} + α_{2} V_{2} + α_{3} V_{3}

This context is a weighted blend. Each input position contributes in proportion to its relevance to the query at position 3.

Sense-check: $K_{2} = [0.9, 0.1]$ is the vector most aligned with $Q = [0.5, - 0.2]$ (both have a positive first component). $K_{3} = [- 0.4, 0.6]$ points in the opposite direction of $Q$ , so it gets the lowest weight. The softmax amplifies the relative differences cleanly.

Intuition with a Linguistic Example

"The animal didn't cross the street because it was too tired."

The word "it" could refer to "animal" or "street". A standard context vector treats both equally. Self-attention lets the model learn a stronger connection between "it" and "animal". The query from "it" matches against keys from all preceding words. "Animal" gets a higher similarity score. So it contributes more to the representation of "it".

This is the power of attention: selective, learned relevance between words.

Scope: Self-attention is quadratic in sequence length — for $n$ tokens, it computes $n \times n$ attention scores. This costs $O (n^{2} \cdot d_{k})$ time and $O (n^{2})$ memory. For $n = 1000$ , this is fine. For $n = 100, 000$ , it is infeasible without approximations (sparse attention, Linformer, FlashAttention). Self-attention also has no built-in notion of position order. It treats the sequence as a set. That is why Transformers add positional encodings.

Visual Intuition: Draw a 3-by-3 grid. Rows are query positions (which word is "asking"). Columns are key positions (which word is "being asked about"). The diagonal always has high values (a word is relevant to itself). Consider the sentence "The cat sat on the mat". The cell for "sat" (row 3) looking at "cat" (column 2) is brightly colored. The verb attends to its subject. The cell for "sat" looking at "the" (column 1) is dim — stop words get low attention. This pattern of bright and dim cells is what the heatmap visualizations in Section 14.11 reveal.

Pitfalls:

Forgetting to scale. Using raw dot-product scores without dividing by $d_{k}$ causes softmax saturation for large $d_{k}$ . The attention weights degenerate to one-hot, killing gradients.
Assuming self-attention captures word order.

Without positional encodings, the sequence "cat eats mouse" and "mouse eats cat" produce the same self-attention output (up to permutation). Positional encodings fix this — they are added to the embeddings before attention.

Confusing self-attention with cross-attention. In self-attention, Q, K, V all come from the same sequence. There is one set of inputs. In cross-attention, Q comes from the decoder, K and V from the encoder. Different purpose, same machinery.
Overlooking the diagonal bias.

A word almost always attends strongly to itself because $Q_{i} \cdot K_{i}$ tends to be large. The vectors come from the same embedding via similar projections. This is usually desirable but can drown out cross-word attention early in training.

Comparison — Self-Attention vs Cross-Attention:

Property	Self-Attention	Cross-Attention
Source of K, V	Same sequence as Q	Encoder sequence
Purpose	Inward: relate tokens within one sequence	Outward: relate decoder tokens to encoder tokens
Typical use	Transformer encoder, language modeling	Transformer decoder, machine translation
Sequence length	$n$ tokens → $n \times n$ attention matrix	$n$ decoder tokens × $m$ encoder tokens

When to pick which: Use self-attention when building representations within a sequence (encoding, pretraining). Use cross-attention when generation needs to reference a different source (translation, question answering).

Recap: Self-attention is the inward-looking variant — Q, K, V all come from the same sequence. Every token attends to every other token (including itself) to build context-aware representations. It is quadratic in sequence length and needs positional encodings to capture order. Self-attention is the engine of the Transformer encoder. Cross-attention (Section 14.7) powers the decoder's look-up on the source.

Real-World & Domain Connection: Self-attention is the defining operation of BERT, GPT, and all Transformer variants. BERT uses bidirectional self-attention — every token attends to every other token in both directions — to build deep contextual embeddings. GPT uses causal (masked) self-attention — each token attends only to previous tokens — for autoregressive generation. Vision Transformers (Dosovitskiy et al., 2021) split an image into 16×16 patches. They apply self-attention across them, treating patches like words. AlphaFold 2 applies self-attention across amino acid residues to predict 3D protein structure, showing the concept transcends NLP entirely.

Cross-Attention

Hook: The decoder is generating a French word. It needs to know which English word to translate right now. Cross-attention lets the decoder "look back" at every single source word. It picks the most relevant one — at every step, for every output token.

Definition

Intuition + Analogy: A translator reads a French sentence and writes the English translation. While writing the third English word, they glance back at the French sentence. They do not re-read every word equally. Instead, they focus on the one or two French words that map to this English position. The translator's current thought (what word should come next?) is the query. The French words are the keys and values. Cross-attention is that glance. It goes from one sequence (the decoder's generation) to another sequence (the encoder's source).

Where the analogy breaks: The translator understands meaning. Cross-attention operates purely on vector similarity — $Q \cdot K$ dot products. It learns which source words matter through training, not through linguistic knowledge.

Cross-attention is the form used when the query comes from one sequence. The keys and values come from a different sequence. This is the classic encoder-decoder attention.

Formalize: Let $S_{1}, \dots, S_{T^{'}}$ be decoder hidden states and $H_{1}, \dots, H_{T}$ be encoder hidden states. At decoder time step $t$ :

Query: $Q = S_{t - 1} W_{Q}$ (from the decoder — what should I generate?)
Keys: $K_{i} = H_{i} W_{K}$ for $i = 1, \dots, T$ (from the encoder — here is what is available)
Values: $V_{i} = H_{i} W_{V}$ for $i = 1, \dots, T$ (from the encoder — here is the content)

The dynamic context vector at step $t$ :

C_{t} = i = 1 \sum T α_{t i} V_{i}, α_{t i} = softmax (\frac{Q \cdot K _{i}}{d _{k}})_{i}

The decoder then uses $C_{t}$ (along with its own state and the previous output) to generate the next token. Notice: the query comes from one set of parameters ( $W_{Q}$ with decoder input). Keys and values come from another set ( $W_{K}, W_{V}$ with encoder outputs). This asymmetry defines cross-attention.

Symbol Registry — Section 14.7

Symbol	Meaning	Dimension
$α_{t i}$	Attention weight from decoder step $t$ to encoder position $i$	Scalar, $[0, 1]$
$S_{t}$	Decoder hidden state at time $t$	$R^{d_{h}}$
$H_{i}$	Encoder hidden state at position $i$	$R^{d_{h}}$
$T$	Number of encoder positions (source length)	Integer
$T^{'}$	Number of decoder time steps (target length)	Integer

In a machine translation setup, the decoder generates the translation word by word. At each step, the query comes from the decoder's current state. The keys and values come from the encoder's hidden states — all the input words in the source language.

When Cross-Attention Applies

Cross-attention is essential for tasks where output words must reference input words selectively. In question answering, the question and context go through the encoder. The decoder generates the answer. At each decoder step, the query (from the decoder) is matched against all encoded words (keys and values). The attention mechanism finds which input words are most relevant to the current answer token being generated.

The query says: "Here is what I am trying to generate." The keys are all input tokens that the query matches against. The values are the content to pull through when a match occurs.

Worked Example — Cross-Attention in Question Answering

Input (encoded): "India gained independence in 1947. Gandhi led the movement." Question: "What year was independence?"

The encoder produces hidden states $H_{1}$ through $H_{10}$ (one per input token). The decoder generates the answer token by token.

Decoder step 1 (generating "1947"):

Query comes from decoder's initial state.
Scores: $H_{6}$ ("1947") gets highest score (0.85). $H_{2}$ ("gained") gets 0.05. All others near zero.
Context $C_{1} \approx 0.85 \cdot H_{6}$ — heavily focused on "1947".
Decoder predicts "1947" with high confidence.

Decoder step 2 (generating <EOS>):

Query now comes from decoder state after "1947".
Scores are uniform and low — nothing specific needed.
Decoder predicts <EOS> and the generation stops.

Sense-check: The cross-attention weights at step 1 act like a spotlight. They illuminate the one source token that answers the question and dim everything else. This is exactly what the fixed context vector in Section 14.2 could not do.

Scope: Cross-attention connects two different sequences. It requires that the encoder's hidden states be preserved for the decoder to attend over. This means the full encoder output must be stored during decoding — $O (T \cdot d_{h})$ memory for a source of length $T$ . For very long sources (book-length translation), this storage can be significant. Cross-attention also has the same $O (T \times T^{'})$ computational cost as self-attention. Here $T$ is source length and $T^{'}$ is target length.

Visual Intuition: Draw two vertical bars. The left bar represents the encoder sequence (input tokens stacked vertically). The right bar represents the decoder sequence (output tokens stacked vertically). From each decoder token position, draw arrows to all encoder positions. Thick arrows mean high attention. For a well-aligned translation, thick arrows form a diagonal from top-left to bottom-right. For question answering, thick arrows from the first answer token converge on the specific input position with the answer.

Pitfalls:

Mixing up which sequence provides what. In cross-attention, the decoder provides queries, the encoder provides keys and values. If you swap them, the decoder would be attending to itself — that is self-attention.
Forgetting that cross-attention needs the full encoder output.

Unlike the basic encoder-decoder which only passes the final state $H_{T}$ , cross-attention requires all encoder hidden states $H_{1}, \dots, H_{T}$ . This is a design trade-off: more memory for higher quality.

Assuming one cross-attention layer is enough.

Modern Transformers stack multiple cross-attention layers. Each layer sits in a different decoder block, helping the decoder build increasingly abstract source references as it goes deeper.

Neglecting padding masking. Encoder outputs include padding tokens. Without masking, the decoder can attend to padding — wasting capacity and potentially learning spurious patterns.

Recap: Cross-attention is the bridge between two sequences — queries from one, keys and values from another. It solves the uniform-context problem by letting every decoder step selectively focus on relevant source tokens. Self-attention (Section 14.6) builds representations within a sequence; cross-attention connects across sequences. Next we look at how to score query-key pairs — the attention scoring mechanisms that make this all work.

Real-World & Domain Connection: Cross-attention is the critical component in all encoder-decoder Transformers. In T5 (Text-to-Text Transfer Transformer), every task is cast as text-to-text. Translation, summarization, and question answering all use cross-attention to connect the encoded input to the decoded output. In multimodal models like Flamingo, cross-attention connects a vision encoder (image patches) to a language decoder (text generation). This lets the model "look at" specific image regions when describing them in text. In DALL-E 2, cross-attention maps text embeddings onto image generation. The prompt "an astronaut riding a horse" focuses image generation on the right concepts.

Attention Scoring Mechanisms

Hook: You have a query and a hundred keys. How do you measure "compatibility"? A dot product? A small neural network? The scoring function you pick affects speed, memory, and whether the model can learn sharp attention at all.

Dot-Product Scoring

Intuition + Analogy: Two people vote on a hundred issues. The dot product of their voting records tells you how aligned they are. Large positive means strong agreement. Near zero means unrelated. Large negative means strong opposition. In attention, the query and key are voting records over $d_{k}$ dimensions. The dot product measures their alignment on each dimension and sums it into a single number. High alignment = high attention.

Where the analogy breaks: Voting records are binary (yes/no). Query and key vectors are real-valued and learned — they encode nuanced relevance, not just agreement. A dot product of 0.8 does not mean "agree on 80% of issues". It means the network has learned that these two vectors are highly compatible for the current task.

The simplest scoring method is the dot product between query and key:

score (Q, K) = Q \cdot K = i = 1 \sum d_{k} Q_{i} K_{i}

If $Q$ and $K$ are both vectors of dimension $d_{k}$ , the result is a single scalar. A higher value means higher similarity. A lower value (possibly negative) means lower similarity.

This is a similarity-based score. It works well when query and key have the same dimension.

Softmax Normalization

Raw scores are just numbers. They could be 10, -5, 0.3. To turn them into weights, apply softmax:

α_{i} = \frac{e ^{score_{i}}}{\sum _{j} e ^{score_{j}}}

After softmax, all $α_{i}$ are between 0 and 1, and they sum to 1. Now you have a proper probability distribution over input positions. These $α_{i}$ values are the attention weights.

Scaled Dot-Product Attention

Formalize: In the Vaswani et al. (2017) paper, the researchers found that raw dot-product scores caused training problems. For large dimensions $d_{k}$ , the dot product values grow large. Large values pushed into softmax cause saturation — gradients vanish, and learning stalls.

Why does this happen? Assume the elements of $Q$ and $K$ are independent random variables with mean 0 and variance 1. The dot product $Q \cdot K = \sum_{i = 1}^{d_{k}} Q_{i} K_{i}$ is a sum of $d_{k}$ independent products. Each product $Q_{i} K_{i}$ has mean 0 and variance 1. By the central limit theorem, the variance of the dot product grows linearly with $d_{k}$ :

Var (Q \cdot K) = i = 1 \sum d_{k} Var (Q_{i} K_{i}) = i = 1 \sum d_{k} 1 = d_{k}

So for $d_{k} = 64$ , the raw dot products have a standard deviation of $64 = 8$ . These large values push the softmax into its saturation region, where gradients are near zero.

The fix: Scale the scores before softmax. Divide by $d_{k}$ :

score (Q, K) = \frac{Q \cdot K}{d _{k}}

Then:

Var (\frac{Q \cdot K}{d _{k}}) = \frac{d _{k}}{d _{k}} = 1

The variance is now 1 regardless of $d_{k}$ . The scores stay in a reasonable range. The softmax produces well-spread distributions, and gradients flow normally.

Complete scaled dot-product attention:

α_{i} = softmax (\frac{Q \cdot K _{i}}{d _{k}})_{i} = \frac{e ^{Q \cdot K_{i} / d_{k}}}{\sum _{j} e ^{Q \cdot K_{j} / d_{k}}}

Attention (Q, K, V) = i \sum α_{i} V_{i}

Symbol Registry — Section 14.8

Symbol	Meaning	Dimension
$α_{i}$	Attention weight for position $i$	Scalar, $[0, 1]$
$Q$	Query vector	$R^{d_{k}}$
$K_{i}$	Key vector for position $i$	$R^{d_{k}}$
$V_{i}$	Value vector for position $i$	$R^{d_{v}}$
$W_{Q}, W_{K}$	Additive projection matrices	$R^{h \times d_{q}}, R^{h \times d_{k}}$
$v$	Final projection vector (additive)	$R^{h}$
$h$	Hidden dimension in additive attention	Integer

Exam note: Use the scaled version ( $\frac{Q \cdot K}{d _{k}}$ ) in your answers for all transformer-based attention computations. This is the standard form.

Additive Attention (Bahdanau, 2014)

Formalize: The original attention paper (Bahdanau et al., 2014) used a different scoring method:

score (Q, K) = v^{⊤} tanh (W_{Q} Q + W_{K} K)

Here:

$W_{Q} \in R^{h \times d_{q}}$ and $W_{K} \in R^{h \times d_{k}}$ are parameter matrices. They project the query and key to a common hidden dimension $h$ .
$tanh$ is the activation introducing non-linearity into the scoring step itself.
$v \in R^{h}$ is a learned vector that projects the tanh output down to a scalar.

This is called additive attention because the query and key representations are added before scoring. The $tanh$ introduces a non-linearity into the scoring step itself.

Key properties of additive attention:

Query and key dimensions do not need to match. $W_{Q}$ and $W_{K}$ can project them to a common dimension $h$ before addition. This is the main advantage. You can attend from a decoder hidden state (dimension $d_{h}$ ) to encoder states (possibly different dimension) without reshaping.
It uses extra learned parameters ( $W_{Q}, W_{K}, v$ ) — three parameter matrices vs zero for scaled dot-product.
The non-linearity ( $tanh$ ) gives it more expressive power than a plain dot product. The scoring function itself can learn complex interactions between query and key.

Bahdanau vs Vaswani — Comparison

Property	Bahdanau (2014)	Vaswani (2017)
Extra parameters	Yes ( $W_{Q}, W_{K}, v$ )	No
Non-linearity in score	Yes ( $tanh$ )	No (purely linear dot product)
Dimension constraint	Q and K can have different dimensions	Q and K must have same dimension $d_{k}$
Computational cost	Higher: two matrix multiplies plus $tanh$	Lower: one dot product and one division
Parallelism	Harder to parallelize (sequential operations)	Easier to parallelize (purely matrix multiplications)

The dot-product approach is simpler and faster. It is the standard in modern transformers. The additive approach was critical for early sequence-to-sequence models. It remains useful when query and key dimensions differ.

Real-world: Almost all current transformer architectures (BERT, GPT, T5) use scaled dot-product attention. The additive form is mainly of historical interest.

Worked Example — Comparing Scoring Functions

Take $Q = [1, 0, - 1]$ and $K = [0, 1, - 1]$ , with $d_{k} = 3$ .

Scaled dot-product:

score = \frac{1 \cdot 0 + 0 \cdot 1 + ( - 1 ) \cdot ( - 1 )}{3} = \frac{1}{1.73} \approx 0.58

Additive (with $W_{Q} = I_{3 \times 3}, W_{K} = I_{3 \times 3}, v = [1, 1, 1]^{⊤}$ ):

W_{Q} Q + W_{K} K = [1, 0, - 1] + [0, 1, - 1] = [1, 1, - 2]

tanh ([1, 1, - 2]) \approx [0.76, 0.76, - 0.96]

score = [1, 1, 1] \cdot [0.76, 0.76, - 0.96] = 0.76 + 0.76 - 0.96 = 0.56

Both scores are similar here (0.58 vs 0.56). The difference emerges when $Q$ and $K$ have very different norms or dimensions. The additive form can learn to compensate via $W_{Q}$ and $W_{K}$ . Dot-product cannot.

Sense-check: With identity matrices and uniform $v$ , the additive score approximates the sum of elements of $(Q + K)$ , squashed by $tanh$ . It behaves roughly like dot-product in this simple case, but the learned matrices add degrees of freedom.

Scope: The scoring function choice matters most when:

$d_{k}$ is large (use scaled dot-product or gradients vanish).
$Q$ and $K$ have different dimensions (use additive or a bilinear form $Q^{⊤} M K$ ).
The model needs to learn complex non-linear query-key interactions. Additive attention may help, but multi-head attention in transformers offers a different path to expressiveness.
Training stability is a concern (scaled dot-product is more stable with layer normalization).

Visual Intuition: Two plots side by side. Left: dot-product scores for $d_{k} = 2$ produce a smooth distribution after softmax — probabilities are spread across keys. Right: unscaled dot product for $d_{k} = 64$ . One key gets probability near 1.0. All others get near 0.0 — softmax saturation. The scaling factor $d_{k}$ turns the right plot back into the left plot. A third plot shows additive attention as a learned, non-linear warping of the score space.

Pitfalls:

Using unscaled dot-product for large $d_{k}$ . This is the most common mistake. Always divide by $d_{k}$ . The math is clear — variance grows linearly with dimension — but students forget in implementations.
Confusing the scaling factor. It is $d_{k}$ (square root of key dimension), not $d_{k}$ and not $\frac{1}{d _{k}}$ . Dividing by $d_{k}$ over-scales and pushes all probabilities toward uniform.
Thinking additive attention is obsolete.

It is less common in transformers, but the general principle of projecting Q and K to a shared space before comparison appears in many architectures. Examples include Perceiver and cross-modal attention.

Forgetting that scores can be negative. After $tanh$ (additive) or direct dot-product, scores can be negative. The softmax's exponentiation $e^{score}$ always produces a positive number, so negative scores become small but still non-zero weights. This is desirable — zero weight would mean zero gradient.

Exam note: When asked about attention scoring, always present the scaled dot-product form: $score (Q, K) = \frac{Q \cdot K}{d _{k}}$ . The scaling factor prevents softmax saturation and vanishing gradients as $d_{k}$ grows large. Know the difference from Bahdanau's additive attention: $v^{⊤} tanh (W_{Q} Q + W_{K} K)$ . It adds learned parameters and non-linearity into the scoring itself.

Real-World & Domain Connection: The scaling factor $d_{k}$ was one of the key engineering insights that made Transformers trainable at scale. Without it, the original Transformer would have suffered from vanishing gradients in deep stacks (the paper used $d_{k} = 64$ ). Today, FlashAttention (Dao et al., 2022) optimizes scaled dot-product attention at the hardware level. It achieves 2-4x speedups by fusing the softmax and matrix multiply operations in GPU SRAM. But the mathematical core ( $Q K^{⊤} / d_{k}$ ) remains unchanged.

Attention-Based Pooling vs Traditional Pooling

Hook: Max pooling picks one winner and ignores everyone else. Average pooling treats everyone exactly the same — even the noisy neighbor. Attention pooling lets the network decide who matters, and when. That is the difference between a dictatorship, a democracy, and an expert committee.

Traditional Pooling

Intuition + Analogy: In a classroom, the teacher asks "What is 2+2"? Max pooling: only the loudest student's answer counts, even if wrong. Average pooling: every student's answer is averaged together — correct, incorrect, and "I don't know". Attention pooling: the teacher listens to everyone but weighs answers by how much each student knows about math. The math whiz gets more weight. The student who always guesses gets less.

Traditional pooling uses fixed rules. Attention pooling learns the rules.

Max pooling: takes the maximum value and discards everything else. One dominant input controls the result.
Average pooling: gives uniform weight to all values. Every input contributes equally. The average is biased by extreme values — a single outlier can shift the mean.

Both are non-learned, fixed operations. The weights (1 for the max, $\frac{1}{n}$ for average) do not adapt to the data or the context.

Learned Weighted Aggregation

Formalize: Attention pooling computes a weighted sum where the weights are learned functions of the data:

Output = i \sum α_{i} V_{i}

Compare this to traditional pooling:

Max pooling: $Output = max_{i} V_{i}$ — weight vector is one-hot at argmax.
Average pooling: $Output = \frac{1}{n} \sum_{i} V_{i}$ — weight vector is uniform $\frac{1}{n}$ .
Attention pooling: $α_{i} = softmax (score (Q, K_{i}))$ — weights are data-dependent and learned.

The softmax ensures $0 \leq α_{i} \leq 1$ and $\sum_{i} α_{i} = 1$ . So attention pooling is a convex combination of values. It always stays within the convex hull of the input set.

Symbol Registry — Section 14.9

Symbol	Meaning	Dimension
$V_{i}$	Value vector for element $i$	$R^{d_{v}}$
$n$	Number of elements in the set	Integer

Attention pooling is different. The weights are learned and context-dependent. Given a query, the network learns which values to emphasize and which to downplay. The weights come from learned matrices $W_{Q}, W_{K}, W_{V}$ and the specific tokens being processed.

The key idea: attention weights are customized per time step and per query. This makes the aggregation data-dependent. For time series (stock price prediction), recent days might get higher attention. For sentiment analysis, emotionally charged words get higher attention. The network learns what to focus on.

Worked Example — Pooling Comparison

Consider a set of values $V = [2, 8, 3, 100, 4]$ (e.g., five sensor readings, one is a spike).

Max pooling: Output = 100. The spike dominates. The other four readings are lost.

Average pooling: Output = $(2 + 8 + 3 + 100 + 4) /5 = 23.4$ . The spike pulls the average up by a factor of ~5. All readings contribute equally, so the outlier contaminates the result.

Attention pooling (with a query that selects "typical" values, learned weights):

α = [0.20, 0.35, 0.30, 0.05, 0.10]

Output = 0.20 (2) + 0.35 (8) + 0.30 (3) + 0.05 (100) + 0.10 (4) = 0.40 + 2.80 + 0.90 + 5.00 + 0.40 = 9.50

Sense-check: The learned weights gave the spike only 5% weight, correctly down-weighting the outlier. The output 9.50 is close to the "real" central tendency of (2, 8, 3, 4). Max pooling would give 100 (wrong). Average pooling would give 23.4 (also wrong due to the spike).

Comparison — Traditional vs Attention Pooling:

Property	Max Pooling	Average Pooling	Attention Pooling
Adaptivity	None	None	Adapts to input and query
Outlier handling	Amplifies outliers	Diluted by outliers	Can learn to down-weight outliers
Parameters	0	0	$W_{Q}, W_{K}, W_{V}$
Use case	Down-sampling, translation invariance	Smoothing, global context	Selective aggregation, context-aware blending

Scope: Attention pooling requires computing $n$ compatibility scores, making it $O (n)$ per query. Max and average pooling are $O (n)$ but with a much smaller constant factor and no learned parameters. For large-scale downsampling in CNNs, max/average pooling remains the practical choice. Use attention pooling when the aggregation needs to be context-dependent. Examples include Transformers, graph neural networks, and set-based models. In these cases, the "importance" of each element depends on the query.

Visual Intuition: Three panels showing a bar chart of 5 values (heights: 2, 8, 3, 100, 4). Panel 1 (max): a giant arrow points to the 100 bar, all others greyed out. Panel 2 (average): a horizontal line at 23.4, all bars equally faded. Panel 3 (attention): bars have different transparency levels. The 8 bar is darkest (highest weight). The 100 bar is barely visible (lowest weight). A weighted average line sits near 9.5.

Pitfalls:

Using attention pooling when uniform aggregation is enough. Attention pooling has learnable parameters and costs more compute. For simple tasks (e.g., global average pooling before a classifier), fixed pooling works fine and is faster.
Forgetting that attention weights must sum to 1.

If you compute scores but forget the softmax, weights can be any value. The output is no longer a convex combination and can explode.

Assuming attention pooling is always better. On small datasets, the extra parameters in $W_{Q}, W_{K}, W_{V}$ can overfit. Fixed pooling is more reliable when data is scarce.
Confusing pooling with downsampling. CNN max/average pooling reduces spatial dimensions (e.g., 2×2 pooling with stride 2). Attention pooling does not change sequence length — it aggregates across the sequence dimension, producing one output from $n$ inputs.

Recap: Max pooling picks the winner. Average pooling weights everyone equally. Attention pooling learns whom to listen to, and when. It is a differentiable, context-dependent weighted sum — the foundation of all modern attention mechanisms. Now we look at how the decoder uses these pooled contexts to pick output words — decoding strategies.

Real-World & Domain Connection: Attention pooling is at the heart of set-based deep learning. Deep Sets (Zaheer et al., 2017) use attention pooling to process unordered sets of varying sizes. Applications include particle physics (collision events), astronomy (point clouds of stars), and drug discovery (sets of molecular fragments). Graph Attention Networks (Velickovic et al., 2018) use attention pooling over a node's neighbors to compute node embeddings. They learn which neighbors matter most for each node. This enabled state-of-the-art results on citation networks, protein-protein interaction prediction, and molecular property classification.

Decoding Strategies

Hook: The model gives you a probability distribution over the entire vocabulary — say, 50,000 words. Which one do you pick? The obvious choice (the single highest-probability word) turns out to be a surprisingly bad strategy. Here is why, and what to do instead.

Greedy Decoding

Intuition + Analogy: You are in a maze. At each junction, you take the path marked "best". You reach a dead end. The problem: the locally-best path (greedy) is not the globally-best route. Greedy decoding picks the highest-probability word at every step. This can commit you to a bad sentence early with no way to backtrack.

Where the analogy breaks: In a real maze, you can go back. In greedy decoding, once you pick a word, it feeds into the next step. You cannot undo it without beam search or other look-ahead methods.

Formalize: At each decoder time step $t$ , the model produces a probability distribution over the vocabulary $V$ :

P (Y_{t} = w ∣ Y_{1}, \dots, Y_{t - 1}, C), \forall w \in V

Greedy decoding: Select the word with maximum probability:

Y^_{t} = ar g w \in V max P (Y_{t} = w ∣ history)

The selected $Y^_{t}$ becomes part of the history for step $t + 1$ . This continues until an <EOS> token is generated or a maximum length is reached.

Symbol Registry — Section 14.10

Symbol	Meaning	Dimension
$K$	Beam width (number of hypotheses)	Integer
$p$	Probability threshold for nucleus sampling	Scalar, $[0, 1]$
$Y_{t}$	Output token at time $t$	One-hot, $R^{V}$
$P (\cdot)$	Probability distribution over vocabulary	$R^{V}$

If $P (word_{3} ∣ previous) = 0.8$ and all other words have lower probabilities, the model outputs word 3. Simple and fast.

The weakness: greedily picking the best at each step might not produce the best overall sequence. The model can commit to a suboptimal path early.

Top-K Sampling

Instead of taking just the single best, keep the top $K$ most probable words at each step. Maintain $K$ partial hypotheses and explore from each. This is beam search when $K$ is the beam width. It gives more diversity and often better results than greedy decoding.

Probability Threshold Sampling (Nucleus Sampling)

Setting a fixed $K$ can be arbitrary. In a vocabulary of millions, sometimes 50 words are highly probable. Other times only 10 are. Instead of a fixed count, use a probability threshold. Accumulate word probabilities from highest to lowest until the cumulative probability exceeds a threshold (e.g., 0.9). Select from those words.

This adapts the candidate pool size to each time step's distribution.

Comparison — Decoding Strategies:

Strategy	How it picks	Diversity	Quality	Speed
Beam search ( $K$ )	Keep top- $K$ partial sequences, score by cumulative log-prob	Medium	Better for translation, QA	$K$ × slower
Nucleus sampling ( $p$ )	Sample from the smallest set of words whose cumulative probability exceeds $p$	High	Better for open-ended generation	Similar to greedy

Worked Example — Greedy vs Beam Search

Target vocabulary at step $t$ with probabilities:

P ("the") = 0.4, P ("a") = 0.3, P ("an") = 0.2, P ("some") = 0.1

Greedy: Pick "the" (0.4). Step $t + 1$ proceeds from "the" only.

Beam search (K=2): Keep "the" (0.4) and "a" (0.3) as two hypotheses. At step $t + 1$ , extend both:

From "the": $P ("cat") = 0.6, P ("dog") = 0.3, \dots$
From "a": $P ("cat") = 0.1, P ("dog") = 0.8, \dots$

Cumulative log-probabilities:

"the cat": $lo g (0.4) + lo g (0.6) = - 0.92 + (- 0.51) = - 1.43$
"a dog": $lo g (0.3) + lo g (0.8) = - 1.20 + (- 0.22) = - 1.42$

Even though "the" was locally better, "a dog" scores slightly higher overall. Beam search found it. Greedy would have missed it completely.

Sense-check: Beam search corrects for local greediness by keeping alternatives alive. The cost is $K$ times the compute. For translation (BLEU score objective), beam search with $K = 4$ to $10$ is standard. For creative generation, nucleus sampling is preferred for diversity.

Scope: Greedy decoding works fine for deterministic tasks like speech recognition where one correct transcription exists. Beam search is standard for translation and summarization. In these tasks, a single "best" sequence exists (as measured by BLEU or ROUGE). Nucleus sampling is preferred for open-ended generation. Examples include story writing and chatbots. In these domains, multiple valid continuations exist. Diversity matters. None of these strategies changes the underlying model — decoding is purely an inference-time choice.

Visual Intuition: Picture a decision tree. At the root, the model splits into 50,000 branches (vocabulary), each with a probability. Greedy picks the thickest branch and prunes everything else. Beam search keeps the top $K$ thickest branches and explores their subtrees. Nucleus sampling collects branches from thickest to thinnest. It stops when the collected thickness exceeds 90% of the total. Then it picks randomly from those. The tree illustration shows how greedy can trap you in a subtree with no good leaves. Beam search finds a slightly thinner branch that leads to much better leaves.

Pitfalls:

Thinking greedy decoding is always worst.

For very short answers (question answering: "1947"), greedy is fine — there is only one right answer. The suboptimal-path problem matters most for long sequences.

Setting beam width too large. $K = 100$ is rarely better than $K = 10$ but costs 10× more. Diminishing returns start around $K = 8$ – $16$ .
Forgetting length normalization. Longer sequences accumulate more negative log-probabilities (more terms added), biasing beam search toward shorter outputs. Divide cumulative log-prob by sequence length to fix this.
Confusing beam search with top-K sampling. Beam search maintains $K$ partial sequences and scores them jointly. Top-K sampling samples one word from the top $K$ candidates at each step. Different mechanisms, different use cases.

Q: Is the decoder's output deterministic or probabilistic? A: The output is always probabilistic internally. For every word in the vocabulary, the model computes a probability. What you see as the final output is a sampling choice. By default, most libraries use greedy sampling — take the highest-probability word. But the underlying mechanism is a probability distribution, just like any classification model. In scikit-learn, .predict() gives you the class label (greedy), while .predict_proba() reveals the full probability distribution beneath.

Recap: Decoding strategies control how you pick words from the model's probability distribution. Greedy is fast but shortsighted. Beam search keeps $K$ alternatives alive. Nucleus sampling adapts to the distribution's shape. The choice depends on the task: deterministic (translation) or creative (storytelling). Next we look at what attention weights reveal — visualization and interpretability.

Real-World & Domain Connection: GPT models use nucleus sampling ( $p = 0.9$ typically) for open-ended text generation. This gives the model enough flexibility to produce creative, diverse outputs. It also avoids the low-probability tail. Google Translate uses beam search with $K = 8$ and length normalization to produce fluent, complete translations. OpenAI's Whisper uses greedy decoding with temperature for speech recognition. Simplicity trumps search when the audio-to-text mapping is constrained. The decoding strategy can matter as much as the model itself. Changing from greedy to nucleus sampling can transform a GPT output from repetitive nonsense to coherent prose.

Visualization and Interpretability

Hook: You have trained a translation model. It produces good output, but you have no idea why it chose each word. Attention weights let you peer inside. You can see exactly which source words the model looked at for each output word. It is a rare moment of transparency in deep learning.

Attention Heatmaps

Intuition + Analogy: A heatmap of attention weights is like reading a translator's eye-tracking data. You see that when the translator wrote "pieds" (French for "feet"), their eyes were fixated on the source word "feet". When they wrote "mal" (for "hurt"), their eyes were on "hurt". The heatmap visualizes this fixation pattern — rows are output words, columns are input words, and brightness is fixation duration.

Where the analogy breaks: Eye tracking measures actual cognitive attention. Model attention weights are vector dot products — they may correlate with linguistic alignment but do not guarantee it. A high attention weight can sometimes mean "this position provides useful context" rather than "this is the direct translation". Always interpret with caution.

Attention weights can be visualized as heatmaps. Each row corresponds to a decoder output token. Each column corresponds to an encoder input token. The cell color represents the attention weight. It shows how much the decoder attended to that input word when generating that output word.

A dark cell means high attention. A light cell means low attention. These heatmaps are powerful tools for understanding what the model focused on.

Word Alignment in Machine Translation

When you visualize attention in a machine translation model, a strong diagonal pattern often emerges. This means there is good word alignment between source and target languages. The decoder attends to the source word that directly translates to the target word being generated.

Take a source sentence in one language and a target in another. The heatmap shows which source words got attention when generating each target word. The word "good" in the source might map strongly to its translated equivalent. The word "book" aligns to its translation.

Deviations from the diagonal tell you where the word order differs between languages. This interpretability is a major benefit of attention. Unlike a black-box RNN, you can see why the model made its choices.

Worked Example — Reading a Heatmap

English source: "my feet hurt" French target (generated): "j' ai mal au pieds"

Visualize as a 3×3 grid (target words as rows, source words as columns):

	my	feet	hurt
ai	0.1	0.2	0.7
mal	0.1	0.8	0.1
au	0.05	0.90	0.05
pieds	0.05	0.85	0.10

Interpretation:

"j'" (short for "je") aligns to "my" — subject pronoun mapping.
"ai" (auxiliary verb for past tense) aligns to "hurt" — the verb mapping, though French uses a compound past "ai...mal".
"mal", "au", "pieds" all align to "feet" — French expresses "hurt" as "avoir mal au pieds" (have pain at the feet).
The strong diagonal from "feet" to multiple French words shows a structural difference: one English word maps to three French words.

Sense-check: The heatmap is not purely diagonal because the languages have different grammatical structures. The attention pattern reveals how the model learned to map across these differences. The high weights on "feet" for "pieds" and on "hurt" for "mal" confirm the alignment. The model learned meaningful cross-lingual alignments.

Scope: Attention heatmaps are suggestive but not conclusive. Research by Jain & Wallace (2019) showed that different attention distributions can produce the same model predictions. Attention is not a unique explanation. Use heatmaps for debugging and intuition, but do not treat them as formal proofs of model reasoning. In particular, adversarial attention distributions exist that look plausible to humans but encode nothing about the true decision process.

Visual Intuition: A 5×5 grid of colored squares. Rows labeled with French words ("je", "suis", "etudiant"). Columns labeled with English words ("I", "am", "a", "student"). The grid shows bright squares on the diagonal: "je"→"I" (bright), "suis"→"am" (bright), "etudiant"→"student" (bright). The square for "a" is dim because French drops the article. One off-diagonal bright spot: "etudiant" also lightly attends to "a" (the French word implicitly carries the article). This single image explains the translation in one glance.

Pitfalls:

Over-interpreting attention weights. A high attention weight does not necessarily mean the model "used" that input token. It means the weighted sum gave it more influence. The actual impact depends on the value vectors.
Assuming alignment = translation.

In the "my feet hurt" → "j'ai mal au pieds" example, the heatmap shows strong alignment. But the translation strategy (compound past tense, preposition "au") involves multiple source words. Alignment and translation are not 1:1.

Ignoring the softmax effect. Attention weights always sum to 1 per row. If there are 50 source tokens, each gets ~0.02 on average. A weight of 0.15 on one token is actually quite high (7.5× average). Context matters.
Forgetting that deeper layers have different attention patterns.

In multi-layer Transformers, layer 1 attention often focuses on local syntax (adjacent words). Layer 12 attention captures long-range semantics. The heatmap you plot depends on which layer you look at.

Recap: Attention heatmaps reveal which source words the model focused on for each output word. Diagonal patterns indicate word alignment; off-diagonal patterns reveal structural differences between languages. Use heatmaps for intuition and debugging — not as definitive explanations. Section 14.12 covers how the model learns these attention patterns through training.

Real-World & Domain Connection: Attention visualization was one of the key selling points of the Bahdanau (2014) paper. The authors showed heatmaps where attention weights naturally learned word alignments. No explicit alignment supervision was needed. This was a breakthrough because prior statistical MT systems required separate alignment models trained on word-aligned parallel corpora. Today, attention visualization is used for model debugging. If a translation produces "the cat sat on mat" (missing "the" before "mat"), inspecting the heatmap can help. It can reveal whether the model attended to the second "the". It can show whether the problem is in attention or in the decoder's word choice. Tools like BERTViz and exBERT make these visualizations interactive for Transformer models.

Training and Optimization

Hook: Attention is not magic — it learns through backpropagation, just like every other neural network component. The scoring matrices $W_{Q}, W_{K}, W_{V}$ start random and gradually learn to focus on the right words. But training sequence models has its own tricks. Loss functions, stopping rules, and a pipeline must coordinate encoder, attention, and decoder.

Cross-Entropy Loss

Intuition + Analogy: A teacher grades a translation word by word. "The" was correct — gold star. "Cat" was wrong (should have been "dog") — red mark. At the end, you add up all the red marks and adjust your strategy. Cross-entropy loss does exactly this. It penalizes the model whenever the predicted probability distribution does not put 100% mass on the correct word. The penalty is $- lo g (p)$ — the more confident you were in the wrong word, the bigger the penalty.

Where the analogy breaks: The teacher knows the "right" translation (there may be several). Cross-entropy assumes exactly one correct token at each position. This is called "hard" target training. In practice, label smoothing (assigning 0.1 probability to wrong words) softens this assumption.

Formalize: At each decoder time step $t$ , the output is a probability distribution over the vocabulary:

P (Y_{t} ∣ Y_{< t}, C) = softmax (W_{O} H_{t}^{(d)}) \in R^{V}

where $W_{O} \in R^{V \times d_{h}}$ is the unembedding matrix. The ground truth $Y_{t}^{true}$ is a one-hot vector of dimension $V$ .

The categorical cross-entropy loss at step $t$ is:

L_{t} = - lo g P (Y_{t} = Y_{t}^{true} ∣ Y_{< t}, C)

Because $Y_{t}^{true}$ is one-hot, this simplifies to the negative log of the probability assigned to the correct word:

L_{t} = - lo g (softmax (W_{O} H_{t}^{(d)})_{y_{t}})

where $y_{t}$ is the index of the correct word.

Sum the losses across all $T^{'}$ decoder time steps to get the total sequence loss:

L = t = 1 \sum T^{'} L_{t} = - t = 1 \sum T^{'} lo g P (Y_{t}^{true} ∣ Y_{< t}, C)

Derivation in matrix form: Let $z_{t} = W_{O} H_{t}^{(d)} \in R^{V}$ be the logits. Then:

P (Y_{t} = w ∣ history) = \frac{e ^{z_{t, w}}}{\sum _{v = 1}^{V} e ^{z_{t, v}}}

L_{t} = - lo g \frac{e ^{z_{t, y_{t}}}}{\sum _{v} e ^{z_{t, v}}} = - z_{t, y_{t}} + lo g v = 1 \sum V e^{z_{t, v}}

This is the standard log-softmax form. The gradient is:

\frac{\partial L _{t}}{\partial z _{t, w}} = P (Y_{t} = w ∣ history) - 1 [w = y_{t}]

The gradient flows back through $W_{O}$ , through the decoder RNN. It flows through the attention mechanism, updating $W_{Q}, W_{K}, W_{V}$ . It also flows through the encoder. All of this happens via backpropagation through time.

Symbol Registry — Section 14.12

Symbol	Meaning	Dimension
$L$	Total sequence loss	Scalar
$W_{O}$	Unembedding weight matrix	$R^{V \times d_{h}}$
$V$	Vocabulary size	Integer
$d_{h}$	Hidden state dimension	Integer
$z_{t}$	Logits (pre-softmax scores) at time $t$	$R^{V}$
$y_{t}$	Index of correct word at time $t$	Integer
$T^{'}$	Number of decoder time steps	Integer

Compute the gradient of this loss and backpropagate. The attention weight matrices $W_{Q}, W_{K}, W_{V}$ are updated along with all other parameters.

Stopping Criteria

During inference, the decoder generates tokens one by one. When does it stop? Two common approaches:

Generate until a special end-of-sequence token $⟨ EOS ⟩$ is produced. The model is trained to emit this token when the answer or translation is complete.

Generate until a maximum sequence length is reached (a hard limit to prevent infinite loops).

In question answering, once the model generates the answer and then the end token, it stops. The answer is complete. If the answer is "1947", the decoder generates "1947" followed by $⟨ EOS ⟩$ — two tokens total — and then halts.

Real-world: $W_{O}$ (the unembedding matrix) transforms the decoder hidden state from dimension $d_{h}$ to the vocabulary size $V$ . This matrix is learned during training, and its dimensions are a design choice.

Exam note: When asked about attention scoring, always present the scaled dot-product form with $\frac{Q \cdot K}{d _{k}}$ . The scaling factor prevents softmax saturation and vanishing gradients.

Regularization and Optimizers

Dropout is commonly used as a regularization technique. Specialized optimizers help stabilize training. These use different learning rates and decay schedules. The core training flow has several stages. First, a forward pass through the encoder. Then, attention-weighted context computation. Next, decoder autoregressive generation. Then loss computation via cross-entropy. Finally, backpropagation through time. The full optimization details are covered in mathematical foundations courses.

Worked Example — Training Loop Trace

Consider a minimal translation task: English "cat" → French "chat".

Encoder input: "cat" → embeddings → RNN → $H_{1}, H_{2}$ (plus $H_{3}$ for <EOS>). Decoder target: "chat" <EOS>.

Forward pass (with teacher forcing):

Decoder step 1 (input: <start>): Query against encoder states. Output probabilities over French vocab: $P ("chat") = 0.6, P ("chien") = 0.3, P ("") = 0.1$ .
$L_{1} = - lo g (0.6) = 0.511$
Decoder step 2 (input: "chat"): $P ("") = 0.8$ .
$L_{2} = - lo g (0.8) = 0.223$

Total sequence loss: $L = 0.511 + 0.223 = 0.734$ .

Backward pass: The gradient of $L$ updates:

$W_{O}$ (improves word predictions)
$W_{Q}, W_{K}, W_{V}$ (improves attention focus — step 1 should attend to "cat" more, step 2 should attend to <EOS>)
Encoder and decoder RNN weights (improves representation quality)

Sense-check: A well-trained model would have $P ("chat") \to 0.99$ at step 1. At step 2, $P ("") \to 0.99$ . This gives $L \to 0.02$ . The loss directly measures how surprised the model is by the correct word.

Scope: The training pipeline assumes:

The target sequence is known (supervised learning).
Teacher forcing is used (ground truth tokens as decoder inputs).
The vocabulary is fixed (no out-of-vocabulary words during training).
Loss is summed (not averaged) across time steps, so longer sequences contribute more to the gradient. Some implementations normalize by sequence length to balance short and long examples.

These assumptions hold for standard tasks like translation and summarization. They may need modification for reinforcement learning-based training. This includes using BLEU score directly as a reward. Modifications are also needed when training with partial ground truth (semi-supervised).

Visual Intuition: A pipeline diagram. Box 1: Encoder (reads input → produces hidden states). Box 2: Attention (computes $Q, K, V$ , produces weighted context $C_{t}$ ). Box 3: Decoder (takes $C_{t}$ + previous token → produces output distribution). Box 4: Loss (cross-entropy between predicted and true distribution). Red backward arrows from Box 4, through Boxes 3 and 2, into Box 1. They show backpropagation updating all components. A counter ticks through time steps $t = 1, 2, \dots, T^{'}$ .

Pitfalls:

Forgetting to mask padding in the loss. Padded positions in the target sequence should not contribute to loss. Most frameworks use a loss_mask or ignore_index to skip padding tokens.
Averaging instead of summing the loss. Summing gives higher weight to longer sequences (which may be desirable — longer sequences are harder). Averaging treats all sequences equally. Know which one your framework uses.
Forgetting that $W_{O}$ is a large matrix. For a vocabulary of 50,000 tokens and $d_{h} = 512$ , $W_{O}$ has 25.6M parameters — often the largest matrix in the model. Its gradient computation dominates training time. Techniques like sampled softmax or hierarchical softmax reduce this cost.
Ignoring the interaction between teacher forcing and attention. With teacher forcing, the decoder always sees the correct previous token, so attention can learn clean alignments. Without it, attention must compensate for decoder errors — a much harder learning problem.

Recap: Training a sequence model with attention uses cross-entropy loss summed across decoder time steps. Teacher forcing provides stable gradient flow. Backpropagation flows through all components — encoder, attention matrices, and decoder. The stopping criterion during inference is the <EOS> token. The training pipeline is straightforward but requires careful handling of padding, loss accumulation, and the large $W_{O}$ matrix.

Real-World & Domain Connection: The training pipeline described here is the foundation of all modern sequence-to-sequence models. In large-scale systems like Google Translate, the vocabulary $V$ can be 64,000 subword tokens (using SentencePiece or BPE). The unembedding matrix $W_{O}$ is shared with the input embedding matrix (weight tying) to reduce parameters by half. Training is done on TPU pods with hundreds of devices. Techniques include label smoothing (0.1) to prevent overconfidence. Early stopping is based on BLEU score on a validation set. The basic cross-entropy formulation here — unchanged since Sutskever et al. (2014) — remains the industry standard.

Exam Guidance Summary

Exam note: When asked about attention, present the scaled dot-product form: $score (Q, K) = \frac{Q \cdot K}{d _{k}}$ . This is the standard in all Transformer-based models.

Why scaling is needed: The variance of $Q \cdot K$ grows as $d_{k}$ . Without scaling, large dimensions push softmax into saturation — gradients vanish. Dividing by $d_{k}$ keeps variance at 1 regardless of dimension. If you forget this on an exam, you lose the key insight of Vaswani et al. (2017).

Bahdanau (additive) vs Vaswani (scaled dot-product): Know the scoring formulas, whether extra parameters exist, whether Q and K dimensions must match, and why dot-product is easier to parallelize. On an exam, be ready to write both formulas and explain when each is preferable.

Three bottleneck problems: Fixed capacity (the context vector has limited dimension), information loss over long sequences (decoder influence fades), and uniform encoding (all inputs treated equally). These three problems together motivate attention.

Teacher forcing: Ground truth during training (prevents error cascading), model's own predictions during inference (autoregressive generation). The key insight: teacher forcing creates a train-test mismatch. Be prepared to explain why we accept this trade-off.

QKV framework: Queries come from what you seek (the decoder's need). Keys are what you match against (available context). Values are what you retrieve (the content). Each token projects to all three via learned matrices $W_{Q}, W_{K}, W_{V}$ .

Self-attention vs cross-attention: In self-attention, Q, K, V all come from the same sequence — the sequence attends to itself. In cross-attention, Q comes from one sequence (decoder), K and V from another (encoder). Self-attention builds internal representations; cross-attention connects across sequences.

Decoding strategies: Greedy picks argmax at each step — fast but shortsighted. Beam search (top-K) keeps $K$ hypotheses alive — better for translation. Nucleus sampling uses a probability threshold — better for open-ended generation. Know the trade-off between quality and diversity.

Stopping criterion: The end-of-sequence token $⟨ EOS ⟩$ . The model is trained to emit this when the output is complete. A maximum sequence length provides a hard backup limit.

Cross-entropy loss: $L_{t} = - lo g P (Y_{t}^{true} ∣ Y_{< t}, C)$ . Summed across all decoder time steps. The gradient updates all components: encoder, attention matrices, and decoder.

Attention heatmaps: Rows = output tokens, columns = input tokens. Diagonal = word alignment. Off-diagonal = structural differences. Use for debugging, not formal proof. Know that heatmaps are suggestive — adversarial patterns exist that fool human interpretation.

Key Industry Applications

Transformer models (BERT, GPT, T5): All use attention as their core mechanism. BERT uses bidirectional self-attention for deep language understanding. GPT uses causal (masked) self-attention for autoregressive text generation. T5 casts every NLP task as text-to-text using encoder-decoder cross-attention. These three models cover the attention spectrum: encode-only, decode-only, and encode-decode.

Vision Transformers (ViT): Apply self-attention to image patches (16×16 pixel grids treated as "visual words"). ViT matches or exceeds convolutional networks on image classification (ImageNet) when trained on enough data. This proves attention works beyond text. The patch embedding replaces the word embedding; everything else is identical to the NLP Transformer.

Flamingo (DeepMind): A multimodal model using cross-attention to connect a vision encoder (frozen) with a language decoder. It can answer questions about images, describe scenes, and follow visual instructions — all through attention bridging two modalities.

Whisper (OpenAI): An encoder-decoder Transformer for speech recognition. The encoder processes 80-channel log-Mel spectrogram frames as "audio tokens." Cross-attention connects these to the decoder generating text. The architecture is nearly identical to a translation model, just with audio input instead of text input.

Machine translation (Google Translate GNMT, DeepL): The original killer app for attention. Bahdanau's (2014) attention mechanism solved the long-sentence problem, and Vaswani's (2017) Transformer made training massively parallel. Modern systems use deep Transformer stacks (6+ layers) with multi-head attention and subword tokenization.

Question answering (SQuAD, Natural Questions): Cross-attention is essential — the decoder generates answers by attending to the encoded question + context document. Models can point to exact answer spans through attention weights.

Text summarization (Pegasus, BART): Abstractive summarization uses encoder-decoder attention to read a long document and produce a short summary. The attention learns which sentences carry the core information.

Image captioning (Show, Attend and Tell): A CNN encodes the image into feature maps. An attentive LSTM decoder generates the caption, with cross-attention over the image features. The attention heatmap shows which image region the model "looked at" for each generated word.

Sentiment analysis: Attention helps by focusing on sentiment-bearing words ("disappointed", "not recommended") and ignoring neutral context. Self-attention is now standard in sentiment classifiers (BERT-based).

Time series prediction (stock prices, weather, energy load): Attention weights let the model focus on recent time steps or specific seasonal patterns. Temporal Fusion Transformers and Informer explicitly use attention mechanisms optimized for long-sequence time series.

Protein structure prediction (AlphaFold 2): Attention over amino acid residue pairs captures spatial relationships. The attention weights directly encode pairwise distances and angles — transforming a biological problem into a geometric attention computation.

Code generation (GitHub Copilot, Codex): GPT-style causal self-attention trained on code repositories. The attention mechanism learns programming idioms, variable scoping, and API usage patterns. These all emerge as patterns in the attention weights over token sequences.

Reference: Jurafsky and Martin, Speech and Language Processing (3rd ed., 2024) — Chapter 10 covers attention mechanisms and Transformers in depth. It includes detailed diagrams of the attention computation flow, beam search algorithms, and practical implementation guidance. This is the standard NLP textbook used in most university courses.

DNN Lecture 14 notes · Attention Mechanisms

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

1Encoder-Decoder Foundations

Motivation, architecture, and mathematical formulation of encoder-decoder models for sequence-to-sequence tasks

2The Context Vector and Bottleneck

What the context vector is, three key bottleneck issues, and failure with long sequences

3Teacher Forcing

Training vs inference in decoder networks and why teacher forcing prevents error cascading

4Introduction to Attention Mechanisms

Why attention is needed, dynamic context vectors, and historical development

5Query, Key, and Value Framework

The database analogy, token roles, and QKV projections with worked examples

6Self-Attention

Definition, step-by-step computation, worked example, and linguistic intuition

7Cross-Attention

Definition, when cross-attention applies, and worked example in question answering

8Attention Scoring Mechanisms

Dot-product scoring, softmax normalization, scaled dot-product, and additive attention comparison

9Attention-Based Pooling vs Traditional Pooling

Max, average, and attention pooling with worked comparison

10Decoding Strategies

Greedy decoding, Top-K sampling, and nucleus sampling with comparison

11Visualization and Interpretability

Attention heatmaps and word alignment in machine translation

12Training and Optimization

Cross-entropy loss, stopping criteria, regularization, and training pipeline

Postgraduate students in Deep Neural Networks

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Encoder-Decoder Architecture

Must-know: The encoder reads and compresses the input into a fixed context vector C. The decoder generates output tokens of any length from that vector. This separation is essential when input and output lengths differ.

$H_{t} = f (W_{H X} X_{t} + W_{H H} H_{t - 1} + b), C = H_{T}$

Top pitfall: Thinking the decoder must have the same number of time steps as the encoder. The decoder runs for as many steps as the target needs — a 3-word input can produce a 10-word output.

Self-check: Why can't a standard RNN be used directly for translation tasks?

Connects to: RNN, Sequence-to-Sequence Models, Context Vector

Context Vector Bottleneck

Must-know: The fixed-size context vector creates three problems: fixed capacity (limited dimension), information loss over long sequences (fading influence), and uniform encoding (all inputs treated equally).

$C = H_{T} \in R^{d_{h}}, H_{t}^{(d)} = f (W_{H H}^{(d)} H_{t - 1}^{(d)} + W_{H Y} Y_{t - 1} + W_{H C} C + b^{(d)})$

Top pitfall: Assuming LSTM/GRU fix the bottleneck. Gating helps with vanishing gradients but does not solve the fixed-capacity problem — the final hidden state is still one vector.

Self-check: What three specific weaknesses of the fixed context vector motivate attention?

Connects to: Attention Mechanisms, Encoder-Decoder, Sequence Modeling

Teacher Forcing

Must-know: Teacher forcing feeds the ground truth token as input to the next decoder step during training, preventing error cascading. During inference, the model uses its own predictions (autoregressive generation).

$Without: H_{t + 1}^{(d)} = f (\dots + W_{H Y} Y^_{t} + \dots), With: H_{t + 1}^{(d)} = f (\dots + W_{H Y} Y_{t} + \dots)$

Top pitfall: Forgetting that teacher forcing is training-only. The train-test gap means the model never encounters its own errors during training.

Self-check: Why does teacher forcing create a mismatch between training and inference?

Connects to: Decoder Training, Autoregressive Generation, Scheduled Sampling

Attention Mechanisms

Must-know: Attention replaces the fixed context vector with dynamic, weighted contexts C_t computed fresh at each decoder step. The network learns to weigh input positions by relevance to the current generation step.

$C_{t} = i = 1 \sum T α_{t i} H_{i}, α_{t i} = softmax (score (S_{t - 1}, H_{i}))_{i}$

Top pitfall: Confusing attention weights with model parameters. Attention weights are computed fresh for every input — the model learns the scoring function, not the weights.

Self-check: How does a dynamic context vector solve the three bottleneck problems?

Connects to: Context Vector, Scoring Functions, Bahdanau Attention

Query, Key, Value Framework

Must-know: Every token projects into three spaces via learned matrices: Q (what to look for), K (what to match against), V (what to retrieve). Attention is a differentiable database lookup.

$Q_{i} = X_{i} W_{Q}, K_{i} = X_{i} W_{K}, V_{i} = X_{i} W_{V}, Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V$

Top pitfall: Thinking weight matrices change per input. After training, W_Q, W_K, W_V are fixed — only the attention weights change.

Self-check: What is the role of each projection matrix (W_Q, W_K, W_V)?

Connects to: Self-Attention, Cross-Attention, Transformer

Self-Attention vs Cross-Attention

Must-know: In self-attention, Q, K, V all come from the same sequence. In cross-attention, Q comes from the decoder, K and V from the encoder. Self-attention builds internal representations; cross-attention connects across sequences.

$Self-Attention: Z_{i} = j \sum α_{ij} V_{j}, α_{ij} = softmax (\frac{Q _{i} \cdot K _{j}}{d _{k}})_{j}$

Top pitfall: Forgetting that self-attention has no built-in notion of position order — Transformers add positional encodings.

Self-check: When would you use cross-attention instead of self-attention?

Connects to: Transformer Encoder, Transformer Decoder, Positional Encoding

Scaled Dot-Product Attention

Must-know: The scaling factor 1/√d_k prevents softmax saturation by keeping variance at 1 regardless of key dimension. Unscaled dot products grow large variance with dimension, pushing softmax into vanishing gradient regions.

$score (Q, K) = \frac{Q \cdot K}{d _{k}}, α_{i} = softmax (\frac{Q \cdot K _{i}}{d _{k}})_{i}$

Top pitfall: Using unscaled dot-product for large d_k. Always divide by √d_k.

Self-check: Why does the variance of the dot product grow linearly with d_k?

Connects to: Bahdanau Attention, Softmax, Vaswani et al. 2017

Decoding Strategies

Must-know: Greedy decoding picks the argmax at each step (fast but can miss better sequences). Beam search keeps K hypotheses alive. Nucleus sampling adapts the candidate pool to the distribution shape.

$Y^_{t} = ar g w \in V max P (Y_{t} = w ∣ history), Greedy$

Top pitfall: Forgetting length normalization in beam search. Longer sequences accumulate more negative log-probabilities, biasing toward shorter outputs.

Self-check: When would you use nucleus sampling instead of beam search?

Connects to: Autoregressive Generation, Beam Search, Nucleus Sampling

Cross-Entropy Loss for Seq2Seq

Must-know: Cross-entropy loss is summed across all decoder time steps. The gradient flows back through the decoder, attention mechanism, and encoder. Teacher forcing provides stable gradients.

$L = - t = 1 \sum T^{'} lo g P (Y_{t}^{true} ∣ Y_{< t}, C)$

Top pitfall: Forgetting to mask padding in the loss. Use ignore_index or loss_mask to skip padded positions.

Self-check: What is the gradient of the cross-entropy loss with respect to the logits?

Connects to: Backpropagation, Teacher Forcing, Unembedding Matrix

Previous LectureLecture 13 · Recurrent Neural Networks — Advanced Architectures

Next LectureLecture 15 · Attention Mechanisms and Transformer Architecture

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Attention Mechanisms

Encoder-Decoder Foundations

Motivation and Applications

Architecture and Components

Mathematical Formulation

Why Standard RNNs Fail for Sequence-to-Sequence Tasks

The Context Vector and Bottleneck

What the Context Vector Is

Three Key Bottleneck Issues

Failure with Long Sequences

Teacher Forcing

Training vs Inference

Why Teacher Forcing Matters

Introduction to Attention Mechanisms

Why Attention Is Needed

Dynamic Context Vectors

Historical Development

Query, Key, and Value Framework

The Database Analogy

Each Token Plays Three Roles

Query, Key, and Value Projections

Self-Attention

Definition

Step-by-Step Computation

Worked Example — Scalar Score Computation

Intuition with a Linguistic Example

Cross-Attention

Definition

When Cross-Attention Applies

Attention Scoring Mechanisms

Dot-Product Scoring

Softmax Normalization

Scaled Dot-Product Attention

Additive Attention (Bahdanau, 2014)

Bahdanau vs Vaswani — Comparison

Attention-Based Pooling vs Traditional Pooling

Traditional Pooling

Learned Weighted Aggregation

Decoding Strategies

Greedy Decoding

Top-K Sampling

Probability Threshold Sampling (Nucleus Sampling)

Visualization and Interpretability

Attention Heatmaps

Word Alignment in Machine Translation

Training and Optimization

Cross-Entropy Loss

Stopping Criteria

Regularization and Optimizers

Exam Guidance Summary

Key Industry Applications

DNN Lecture 14 notes · Attention Mechanisms

Sections Breakdown

Exam Revision Notes

Encoder-Decoder Architecture

Context Vector Bottleneck

Teacher Forcing

Attention Mechanisms

Query, Key, Value Framework

Self-Attention vs Cross-Attention

Scaled Dot-Product Attention

Decoding Strategies

Cross-Entropy Loss for Seq2Seq

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat