Deep Neural Networks

Transformer Architectures and Optimization

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Deep Neural Networks

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Attention mechanisms (self-attention, cross-attention, scaled dot-product attention) — covered in Lectures 14 and 15
Encoder-decoder architecture foundations — covered in Lecture 14
Transformer encoder components (layer normalization, residual connections, multi-head attention) — covered in Lecture 15
Positional encoding and sinusoidal embeddings — covered in Lecture 15

Transformer Architectures and Optimization

16.1 Encoder and Decoder — Purpose and Intuition

Hook. How does a machine read a thousand-word article and answer.

A single question about it? And how does that same machine then write an entirely new paragraph.

in French — based on what it read? These two abilities.

understanding and generating — live in two different architectures: the encoder and the decoder.

An encoder is a component that reads input and compresses it..

A decoder is a component that takes compressed information and generates new output from it.

Together they form the backbone of every modern transformer. Let's unpack each one.

16.1.1 What an Encoder Does

Intuition + Analogy. Think of a librarian who reads every book in a library and writes one index card per book.

The card does not store every word. It stores the essence.

the main ideas, key characters, genre. Later, someone can pick up that card and answer questions without re-reading the whole book. The librarian is the encoder. The index card is the context.

a dense numerical vector that captures the compressed meaning of the input.

The analogy maps tightly: the books are input sequences (text, time-series, any sequential data). The librarian's reading process is the encoder's computation.

Each index card is a context vector. The card-holder who answers questions is any downstream task head. Where the analogy breaks.

the encoder does not select "important parts" consciously.

every token contributes to the context through learned attention weights. There is no human-like judgment.

Formalize. An encoder is a function $f_{enc}$ that maps an input sequence of $N$ tokens to a fixed-size context vector $c$ :

$c = f_{enc} (x_{1}, x_{2}, \dots, x_{N})$

Each token $x_{i}$ starts as a discrete symbol (a word or sub-word). It gets embedded into a vector in $R^{d_{model}}$ .

The encoder processes all $N$ vectors through stacked self-attention and feed-forward layers. The final output is either.

A sequence of context-rich token representations (one per input position) or a single aggregated context vector (like the [CLS] token in BERT).

The context $c$ is then plugged into a task head — a small output layer that solves a specific problem:

Classification head: $y = softmax (W_{c} c + b_{c})$ for sentiment analysis, topic classification.
Tagging head: one softmax per token position for part-of-speech tagging, named entity recognition.
Span head: two pointers (start, end) for extractive question answering.

Worked Example. English-to-French Translation (encoder side).

Take the input sentence: "The cat sits."

Step	What happens
1. Tokenize	`["The", "cat", "sits"]` — 3 tokens
2. Embed	Each token becomes a vector of size $d_{model} = 512$ . Shape: $3 \times 512$ .
3. Add positional embedding	Shape stays $3 \times 512$ .
4. Encoder stack	6 encoder blocks process all 3 tokens at once using bidirectional self-attention.
5. Output	3 context-rich vectors, one per input token. Shape: $3 \times 512$ .

These 3 vectors carry the full meaning of the English sentence in a form the decoder can use.

The encoder's job ends here. It has read and understood. Now the decoder will generate the French: "Le chat est assis."

Scope — What the Encoder Is and Is Not.

The encoder does NOT generate text. It only produces representations. If you need a model that writes, you need a decoder.
The encoder sees all tokens at once. It uses bidirectional attention. This makes it powerful for understanding tasks but useless for autoregressive generation (where seeing the future would be cheating).
The encoder output is NOT human-readable. Context vectors are high-dimensional arrays of floats. You decode them into labels or text through a task-specific head.
Assumption: The encoder assumes the whole input is available at once. It cannot process streaming data token-by-token (a decoder can).

Visual Intuition. Picture a funnel. At the wide top, words flow in as 512-dimensional vectors. They pass through the funnel's neck.

the attention layers — where.

Each word's meaning gets refined by looking at every other word. At the narrow bottom, a compact representation emerges.

For a classification encoder, the bottom is a single [CLS] vector. For a translation encoder,.

The bottom is one refined vector per input token. The x-axis is token position (1 to N). The y-axis is richness of representation.

shallow at the top, deep and contextual at the bottom.

Pitfalls — Common Traps.

Confusing encoder output with generated text. The encoder produces vectors, not words.

A student once asked, "Why is the encoder output not French yet?".

because the encoder only understands; generation is the decoder's job.

Thinking the encoder compresses by dropping information. It does not pick and choose tokens.

Every token contributes to the context through weighted sums. The compression comes from learning which patterns matter, not from deleting data.

Assuming a bigger context vector is always better. $d_{model} = 512$ or 768 or 1024.

the right size depends on the task. Bigger is not always better; it costs more compute and may overfit on small datasets.

**Forgetting that the encoder sees.

The entire input.** If your task requires causal (left-to-right) processing, a bidirectional encoder is the wrong choice.

Use a decoder instead.

16.1.2 What a Decoder Does

Intuition + Analogy. You receive a compressed ZIP file. You cannot read it directly. You run an unzip program.

The program reads the compressed bytes and reconstructs the original files..

The decoder is the unzip program. It takes the compressed context and generates a new sequence.

the original text, a translation, a summary, or a conversational reply.

Now extend the analogy: a decoder-only model is like a storyteller who starts with.

A single opening phrase ("Once upon a time") and generates every word that follows.

one at a time — each new word deciding what comes next.

There is no separate compressed input. The prompt itself is the seed.

Formalize. A decoder is an autoregressive function $f_{dec}$ that generates an output sequence one token at a time:

$y_{t} = f_{dec} (y_{1}, \dots, y_{t - 1}, c)$

where $c$ is the context from an encoder (in encoder-decoder models) or $c = \emptyset$ (in decoder-only models).

At each time step $t$ , the decoder outputs a probability distribution over the entire vocabulary:

$P (y_{t} ∣ y_{< t}, c) = softmax (W_{out} h_{t})$

where $h_{t}$ is the decoder's hidden state at position $t$ .

The model picks the next token $y_{t}$ by sampling from this distribution (greedy: pick the highest probability.

top-k: pick from the top $k$ candidates).

For a decoder-only model, the process starts with a special <START> token..

The first generated word gets fed back as input for step 2.

Step 2's output becomes input for step 3..

This loop continues until a <END> token is generated or a maximum length is reached.

Worked Example. Generating Text with a Decoder-Only Model.

Prompt: "The weather today"

Step	Input seen so far	Model predicts	Picked
1	`<START> The weather today`	is (0.42), was (0.31), will (0.18)	is
2	`<START> The weather today is`	sunny (0.55), cloudy (0.23), rainy (0.12)	sunny
3	`<START> The weather today is sunny`	. (0.61), and (0.22), with (0.09)	.
4	`<START> The weather today is sunny .`	`<END>` (0.88)	stop

Sense-check: The model produced a plausible continuation. At each step it used only the words before it.

masked attention prevented peeking ahead. The generation stopped naturally. Final output: "The weather today is sunny."

16.1.3 When You Need Both

Formalize — Encoder-Decoder Architecture. A sequence-to-sequence (seq2seq) task maps one sequence to another. You need both encoder and decoder:

$(y_{1}, \dots, y_{M}) = f_{dec} (f_{enc} (x_{1}, \dots, x_{N}))$

The encoder produces context $c = f_{enc} (x_{1 : N})$ . The decoder generates.

The output token by token, attending to $c$ at every step via cross-attention.

The encoder and decoder are trained end-to-end.

the loss from the decoder's wrong predictions flows back through both components.

Comparison — When to pick which architecture:

Task type	Architecture	Why
Classification, tagging, NER	Encoder-only	Only understanding needed; no generation
Chat, story generation, code completion	Decoder-only	Pure generation from a prompt
Translation, summarization, image captioning	Encoder-decoder	Map one sequence to a different sequence

16.1.4 Student Questions and Answers

Q: Can an encoder be thought of as compressing information into bits, like in information theory?

A: Yes, the analogy holds well. In neural networks, text and sequential data get compressed into numerical vector representations.

those vectors are the compressed data, similar to bits in a communication channel. The encoder compresses.

the decoder decompresses or uses that essence for downstream tasks.

The analogy helps even though the implementations differ across domains.

information theory deals with bits and entropy, while neural networks deal with floating-point vectors and learned representations.

Recap. An encoder reads and compresses input into a context vector. A decoder generates new output from a context or a prompt.

An encoder-decoder pairs them for sequence-to-sequence tasks. Architecture choice depends on one question.

do you need to only understand, only generate, or do both?

Bridge. Now that we know what encoders and decoders do, the next question is how they do it.

The core mechanism inside both is attention.

the topic of Section 16.2.

Real-World Connection. Google Translate uses an encoder-decoder transformer. The encoder reads your English sentence and produces context vectors.

The decoder generates the French (or any of 100+ languages) word by word, attending back to the English encoder output at every step. The same architecture.

with a vision encoder instead of a text encoder.

powers image captioning: the encoder "reads" the image patches; the decoder writes.

The description. Encoder-only BERT powers nearly every search engine's query understanding. Decoder-only GPT powers ChatGPT, Copilot, and Claude.

any interface where you type a prompt and get generated text back.

---

16.2 Attention Mechanisms — Recap

Hook. How does a model know that "cat" in "the cat sat on the mat" is the animal doing the sitting.

and not a person or a piece of construction equipment? It checks every other word in the sentence. This simple idea.

letting each word look at every other word — is called attention. And it powers every modern transformer.

Attention is the core computation inside both encoders and decoders. Before we dive into the architecture details, let's revisit how attention works.

16.2.1 Self-Attention

Intuition + Analogy. You walk into a noisy party. Someone says your name across the room.

Instantly, your brain filters out the background chatter and focuses on that one voice. Your brain just performed attention.

it assigned high weight to one sound source and low weight to everything else.

Now flip it: imagine every person at.

The party simultaneously tunes in to every other person, weighting how relevant each conversation is to them.

That is self-attention. Every word is both a listener (query) and a speaker (key/value) at the same time. The room hears itself.

Formalize — Scaled Dot-Product Self-Attention.

Given an input sequence of $n$ tokens, each with embedding dimension $d$ , we form three matrices from the same input:

$Q = X W_{Q}, K = X W_{K}, V = X W_{V}$

where $X \in R^{n \times d_{model}}$ is the input, and $W_{Q}, W_{K}, W_{V} \in R^{d_{model} \times d_{k}}$ are learned projection matrices.

The attention output is a weighted sum of values, where the weights come from query-key compatibility:

$Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$

Step by step:

Compute scores: $S = Q K^{T}$ .

each query is dotted with every key. For $n$ tokens, shape is $n \times n$ .

Scale: Divide by $d_{k}$ .

prevents the dot products from growing too large, which would push softmax into regions of tiny gradients.

Softmax: Each row becomes a probability distribution over all keys.
Weighted sum: Multiply the softmax weights by the values $V$ .

The output for each token is a mixture of all value vectors.

Because the queries, keys, and values all come from the same sequence, this is called self-attention.

Why scaling matters: With $d_{k} = 64$ , two random vectors have dot product around 8. Without scaling, $softmax (8)$ is near 1 while $softmax (- 8)$ is near 0. The attention becomes nearly one-hot. The $d_{k}$ factor keeps variance at 1.

Worked Example — Self-Attention on a 3-word sentence.

Input: "the cat sat" with $d_{model} = 4$ (tiny for illustration), $d_{k} = 4$ .

Token	Embedding $x_{i}$ (simplified)
the	$[1, 0, 1, 0]$
cat	$[0, 1, 0, 1]$
sat	$[1, 1, 0, 0]$

Assume learned projections produce these query and key vectors (simplified):

Token	Query $q_{i}$	Key $k_{i}$
the	$[1, 0, 1, 0]$	$[1, 0, 1, 0]$
cat	$[0, 1, 0, 1]$	$[0, 1, 0, 1]$
sat	$[1, 1, 0, 0]$	$[1, 1, 0, 0]$

Compute $Q K^{T}$ (scores before scaling):

$101011100010101001011100 = 201021112$

Scale by $4 = 2$ :

$1.0 0.0 0.5 0.0 1.0 0.5 0.5 0.5 1.0$

Apply softmax per row (approximate):

Token	Attends to "the"	Attends to "cat"	Attends to "sat"
the	0.58	0.11	0.31
cat	0.11	0.58	0.31
sat	0.23	0.23	0.54

Sense-check: Each word attends most to itself (diagonal dominance is common in early layers) but also spreads some weight to other words.

"sat" distributes more evenly because it relates to both "the" and "cat."

Key insight: Self-attention ignores word order. If you shuffle "the cat sat on.

The mat" to "mat cat on the sat," the self-attention scores come out the same.

This is a critical insight: the sequence order is lost, so you must inject it back.

16.2.2 Positional Embedding

Intuition + Analogy. Self-attention treats the sentence like a bag of words.

it sees who talks to whom, but not who comes first..

A positional embedding is like writing a small number (1, 2, 3, ...) above each word before handing it to the model.

But instead of plain integers, we use a pattern of sine and cosine waves that lets.

The model figure out both absolute position ("this is word #3") and relative distance ("these two words are 2 positions apart").

Formalize. Positional encoding adds a fixed or learned vector $p_{i} \in R^{d_{model}}$ to each token's embedding:

$x ~_{i} = x_{i} + p_{i}$

In the original transformer (Vaswani et al.), $p_{i}$ uses sinusoidal functions:

$p_{i, 2 j} = sin (\frac{i}{1000 0 ^{2 j / d_{model}}}), p_{i, 2 j + 1} = cos (\frac{i}{1000 0 ^{2 j / d_{model}}})$

where $i$ is the token position and $j$ indexes the dimension (0 to $d_{model} /2 - 1$ ). Different dimensions oscillate at different frequencies.

low dimensions change fast, high dimensions change slow. This creates a unique fingerprint for every position.

If $d_{model} = 512$ , then $p_{i} \in R^{512}$ — same shape as the word embedding, so they add directly.

Worked Example — Positional Encoding for Position 3.

For $d_{model} = 4$ , $j \in {0, 1}$ :

$p_{3, 0} p_{3, 1} p_{3, 2} p_{3, 3} = sin (\frac{3}{1000 0 ^{0/4}}) = sin (3) \approx 0.141 = cos (\frac{3}{1000 0 ^{0/4}}) = cos (3) \approx - 0.990 = sin (\frac{3}{1000 0 ^{2/4}}) = sin (0.03) \approx 0.030 = cos (\frac{3}{1000 0 ^{2/4}}) = cos (0.03) \approx 1.000$

Position 3 gets the vector $[0.141, - 0.990, 0.030, 1.000]$ . Sense-check: The values span $[- 1, 1]$ and are bounded.

they won't explode or vanish as sequence length grows.

16.2.3 Multi-Head Attention

Intuition + Analogy. A single self-attention head is like reading a sentence through one colored lens.

you see one kind of relationship (maybe subject-verb agreement). Multi-head attention gives you several lenses.

a red one for syntax, a blue one for semantics,.

A green one for co-reference. You look through all of them at once, then combine the views into one rich picture.

Formalize. Multi-head attention runs $h$ independent attention heads in parallel, each with its own projections:

$head_{k} = Attention (X W_{Q}^{(k)}, X W_{K}^{(k)}, X W_{V}^{(k)})$

Each head operates in a reduced dimension $d_{k} = d_{model} / h$ . The outputs are concatenated and projected back:

$MultiHead (X) = Concat (head_{1}, \dots, head_{h}) W_{O}$

With $d_{model} = 512$ and $h = 8$ : each head works in $R^{64}$ .

Concatenating 8 heads gives back 512 dimensions. $W_{O} \in R^{512 \times 512}$ mixes information across heads.

The total cost is similar to a single head of full dimension.

the per-head $d_{k}$ is smaller — but the model learns richer representations because each head can specialize.

16.2.4 Bidirectional Self-Attention

Formalize. In bidirectional self-attention, every token attends to every other token.

left and right, simultaneously. This is the attention mode used in encoders.

For a token at position $i$ , the attention distribution covers all positions $j \in {1, \dots, n}$ .

There is no mask. The word "cat" in "the cat sat on.

The mat" sees both "the" (left context) and "sat" (right context). This is why encoders excel at understanding tasks.

they have the full picture.

In contrast, decoders use causal (masked) attention, where position $i$ only sees positions $j \leq i$ .

This prevents the decoder from peeking at future words during autoregressive generation.

16.2.5 Symbol Registry — Attention Mechanisms

$d_{model}$ — embedding dimension per token — scalar (e.g., 512)
$h$ — number of attention heads — scalar (e.g., 8)
$d_{k}$ — dimension per head — $d_{k} = d_{model} / h$ (e.g., 64)
$Q$ — query matrix — matrix in $R^{n \times d_{k}}$
$K$ — key matrix — matrix in $R^{n \times d_{k}}$
$V$ — value matrix — matrix in $R^{n \times d_{k}}$
$W_{Q}, W_{K}, W_{V}$ — projection matrices — each in $R^{d_{model} \times d_{k}}$
$W_{O}$ — output projection matrix — $R^{d_{model} \times d_{model}}$
$n$ — sequence length — scalar

Pitfalls — Common Traps.

Forgetting the $d_{k}$ scaling. Without it, large $d_{k}$ pushes softmax into near-one-hot territory. Gradients vanish. Training stalls.
Confusing keys and queries. The query is "what am I looking for?".

The key is "what do I contain?" In self-attention they come from the same place.

But they play different mathematical roles.

Confusing keys and queries. The query is "what am I looking for?".

The key is "what do I contain?" In self-attention they come from the same place, but they play different mathematical roles.

Thinking positional embeddings are optional for transformers. They are mandatory. Without them, self-attention is permutation-invariant.

"dog bites man" and "man bites dog" produce identical attention patterns.

Using multi-head attention but forgetting the output projection. Concatenating heads without $W_{O}$ means heads cannot share information.

The output projection is not optional.

it mixes and aligns the different head perspectives.

Recap. Self-attention lets every token weigh every other token. Positional embeddings restore lost order.

Multi-head attention runs several attention patterns in parallel. Bidirectional attention gives encoders full context.

causal attention keeps decoders honest during generation.

Bridge. With attention mechanics in place, we now assemble them into a complete encoder.

adding layer normalization, residual connections, and stacking blocks. That is Section 16.3.

Real-World Connection. The scaled dot-product attention formula.

$softmax (Q K^{T} / d_{k}) V$ — is the single most important equation in modern NLP..

It sits inside BERT (Google Search), GPT (ChatGPT), and every other transformer.

The positional encoding trick using sine/cosine waves means a model can handle sequences of any length — 10 words or 10,000.

without learning a separate position vector for each. Multi-head attention is why.

A single BERT model can simultaneously handle part-of-speech tagging, co-reference resolution, and sentiment — each head specializes.

---

16.3 Encoder Architecture Components

Hook. Take ten encoders, stack them on top of each other, and you get a deep understanding machine.

But stacking attention blocks is not enough.

you need two silent helpers to keep the training stable: layer normalization and residual connections.

Without them, a 12-layer encoder would not train at all.

An encoder block is a reusable building block. It contains multi-head self-attention, feed-forward layers, and two key stabilization mechanisms.

Let's build it step by step.

16.3.1 Layer Normalization

Intuition + Analogy. Imagine grading exams where one student writes in tiny 8-point font and another in giant 24-point font.

Before you can compare their answers fairly, you need to resize both to the same scale. Layer normalization does this for neural network activations.

it rescales each token's feature vector to zero mean and unit variance, so no single feature dominates the next layer's computation.

Formalize. Layer normalization (LayerNorm) operates on a single token's feature vector $x \in R^{d_{model}}$ . It computes:

$LayerNorm (x) = γ ⊙ \frac{x - μ}{σ} + β$

where:

$μ = \frac{1}{d _{model}} \sum_{k = 1}^{d_{model}} x_{k}$ — the mean across the features of this token
$σ = \frac{1}{d _{model}} \sum_{k = 1}^{d_{model}} (x_{k} - μ)^{2} + ϵ$ — the standard deviation ( $ϵ$ prevents division by zero)
$γ, β \in R^{d_{model}}$ — learnable scale and shift parameters

The operation is applied independently to each token in the sequence. The relative ordering among features is preserved.

if feature 1 was larger than feature 2 before normalization, it stays larger after (just rescaled).

The purpose is gradient stability.

it prevents any one feature from having disproportionately large or small values that would cause gradients to explode or vanish.

Pre-LN vs. Post-LN: The original transformer (Vaswani et al.) placed LayerNorm after the attention sub-layer (post-LN).

Modern architectures place it before (pre-LN). Why does placement matter?

Post-LN: The large, unscaled values from attention flow through the residual connection before being normalized. This can destabilize training in deep stacks.
Pre-LN: The input is normalized before entering attention. Gradients flow more smoothly through the block. The number of computations is identical.

only the order changes.

Worked Example — LayerNorm on a 4D vector.

Input token embedding: $x = [5, 10, - 5, 25]$

$μ [4 pt] σ [4 pt] \frac{x - μ}{σ} = \frac{5 + 10 + ( - 5 ) + 25}{4} = \frac{35}{4} = 8.75 = \frac{( 5 - 8.75 ) ^{2} + ( 10 - 8.75 ) ^{2} + ( - 5 - 8.75 ) ^{2} + ( 25 - 8.75 ) ^{2}}{4} = \frac{14.06 + 1.56 + 189.06 + 264.06}{4} = 117.19 \approx 10.83 \approx [- 0.346, 0.115, - 1.270, 1.501]$

Assuming $γ = [1, 1, 1, 1]$ and $β = [0, 0, 0, 0]$ (identity, before learning):

$LayerNorm (x) \approx [- 0.346, 0.115, - 1.270, 1.501]$

Sense-check: The output has mean $\approx 0$ and standard deviation $\approx 1$ . The value 25 (which was the largest) is now 1.501.

still the largest, but on a controlled scale. This prevents the next layer from being dominated by that single large value.

16.3.2 Residual Connections

Formalize. A residual connection (skip connection) adds the input of a sub-layer directly to its output:

$Output = SubLayer (x) + x$

This means the sub-layer only needs to learn the residual.

the difference between what the input already provides and what is needed.

If the best thing for a particular layer is to do nothing,.

The residual can be driven to zero and the identity $x$ passes through.

Gradient flow benefit: during backpropagation, the gradient has two paths.

through the sub-layer AND directly through the identity connection. This prevents gradients from vanishing in very deep stacks.

Every encoder and decoder block uses residual connections around both the attention sub-layer and the feed-forward sub-layer.

Scope — When Residual Connections Help and When They Don't.

Help: Very deep networks (12+ layers). Shallow gradients can flow through the skip path directly to early layers.
Help: Training stability. If a layer is initialized poorly, the residual path lets signal bypass it until the layer learns useful features.
Do not help with: The fundamental capacity of the network. Residual connections do not add parameters.

they only change the gradient flow. A 2-layer network with skip connections is still a 2-layer network.

Assumption: The input and output dimensions must match for the addition to work. If a sub-layer changes dimensionality (which does not happen inside transformer blocks), a linear projection is needed on the skip path.

16.3.3 Stacking Encoder Blocks

Formalize — One Complete Encoder Block.

An encoder block consists of two sub-layers:

$x^{'} or x^{'} [4 pt] Output = LayerNorm (MultiHeadAttention (x) + x) — Post-LN style = MultiHeadAttention (LayerNorm (x)) + x — Pre-LN style = FeedForward (LayerNorm (x^{'})) + x^{'}$

A single block detects one level of pattern. Stacking $N$ blocks (e.g., $N = 6$ for BERT-base, $N = 12$ for BERT-large) builds hierarchical features:

Lower blocks (1-3): capture local patterns — word pairs, short phrases, basic syntax.
Middle blocks (4-8): capture sentence-level patterns — subject-verb-object structure.
Higher blocks (9-12): capture abstract, global patterns — sentiment, topic, discourse relations.

After the final encoder block, the output goes to a task-specific head (e.g., softmax classifier).

16.3.4 Dimensionality Throughout the Encoder

Worked Example — Dimensionality Walkthrough.

Given: sequence length $N = 1000$ , embedding dimension $d_{model} = 512$ , heads $h = 8$ .

Component	Input Shape	Output Shape	Learnable Parameters
Input embedding	—	$1000 \times 512$	Vocab $\times$ 512
Positional encoding	—	$1000 \times 512$	0 (if sinusoidal)
LayerNorm #1	$1000 \times 512$	$1000 \times 512$	$γ, β$ : $2 \times 512$
Multi-head attention	$1000 \times 512$	$1000 \times 512$	$W_{Q}, W_{K}, W_{V}$ : $3 \times (512 \times 512)$ ; $W_{O}$ : $512 \times 512$
LayerNorm #2	$1000 \times 512$	$1000 \times 512$	$γ, β$ : $2 \times 512$
Feed-forward	$1000 \times 512$	$1000 \times 512$	2 linear layers

Sense-check: Every component preserves the $1000 \times 512$ shape.

this is by design. The residual connections require identical input/output shapes.

So encoder blocks can be stacked arbitrarily without worrying about dimension mismatches.

16.3.5 Encoder-Only Use Cases

A bidirectional self-attention encoder is enough for any task that needs understanding but not generation:

Sentiment analysis — classify a review as positive, negative, or neutral.
Part-of-speech (POS) tagging — label each word as noun, verb, adjective, etc.
Named entity recognition — identify person names, locations, organizations.
Question answering by span extraction — find the start and end of the answer within a passage.
Semantic similarity — compare embeddings of two sentences.
Grammatical correctness checking.

All of these are classification or pattern-extraction tasks. No sequence generation required.

16.3.6 Symbol Registry — Encoder Components

$x$ — input token embedding — vector in $R^{d_{model}}$
$μ$ — mean across the features of one token — scalar
$σ$ — standard deviation across the features of one token — scalar
$γ$ — scaling factor in layer norm — learnable vector in $R^{d_{model}}$
$β$ — shift factor in layer norm — learnable vector in $R^{d_{model}}$
$ϵ$ — small constant to prevent division by zero — scalar (e.g., $1 0^{- 5}$ )
$N$ — sequence length (number of tokens) — scalar
$W_{Q}, W_{K}, W_{V}$ — query, key, value projection matrices — each in $R^{d_{model} \times d_{model}}$
$W_{O}$ — output projection matrix — $R^{d_{model} \times d_{model}}$
$h$ — number of attention heads — scalar (e.g., 8)
$d_{k}$ — dimension per head — $d_{k} = d_{model} / h$ (e.g., 64)

Pitfalls — Common Traps.

Confusing LayerNorm with BatchNorm. BatchNorm normalizes across the batch dimension (different samples).

LayerNorm normalizes across the feature dimension (same sample). In transformers, batch sizes vary and sequences have different lengths.

LayerNorm is preferred because it operates per-token independently of batch size.

Forgetting that LayerNorm has learnable parameters. $γ$ and $β$ are trained.

They let the network undo the normalization if that turns out to be better. Do not treat them as fixed.

Stacking blocks without residual connections. Without skip connections, a 12-layer network would suffer from vanishing gradients.

The residual path is not optional in deep transformers.

it is what makes deep stacks trainable.

Assuming more encoder blocks always help. Diminishing returns set in. BERT-base (12 layers) to BERT-large (24 layers) shows improvements, but the gains taper off while compute cost doubles.

Choose depth based on your task complexity and data size.

Recap. An encoder block combines multi-head self-attention, layer normalization, and residual connections.

LayerNorm keeps feature scales stable; residual connections keep gradients flowing. Stacking blocks builds hierarchical understanding.

local patterns in lower layers, global patterns in upper layers.

Bridge. Now that we have the encoder fully assembled, let's build its counterpart.

the decoder — which adds masking and cross-attention to the same foundations. That is Section 16.4.

Real-World Connection. Pre-LN (layer norm before attention) is now standard in nearly every transformer implementation.

GPT, LLaMA, BLOOM all use it. The switch from post-LN to pre-LN was.

A key enabler for training models with 100+ layers. Without residual connections, the 175-billion-parameter GPT-3 would simply not converge. These two seemingly small design choices.

where to normalize and whether to add skip connections — are what made the deep learning revolution possible.

---

16.4 Decoder Architecture Components

Hook. You are taking a language exam. The question asks you to translate a sentence from Spanish to English.

You can see the entire Spanish sentence in front of you.

that is fair. But what if you could also see.

The answer key? You would just copy it, learn nothing, and fail any real test.

The decoder faces this exact problem. Its solution: a mask that blinds it to future words.

The decoder shares the encoder's foundations.

self-attention, layer norm, residual connections — but adds two new mechanisms: causal masking and cross-attention. These are what make generation possible.

16.4.1 Masked Multi-Head Attention

Intuition + Analogy. You are writing a sentence word by word.

When you write the third word, you know words one and two. You do NOT know word four.

you have not written it yet. Masked attention enforces this rule inside the model.

It is like putting blinders on a horse.

the horse can see the road ahead (past tokens) but cannot see what is in the adjacent lane (future tokens).

Formalize. During training, the decoder receives the full target sequence as input.

Without masking, when predicting token $t$ , the model could peek at tokens $t + 1, t + 2, \dots$ .

the very answers it should be predicting. Masking prevents this by setting attention scores for future positions to $- \infty$ before the softmax:

$MaskedAttention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}} + M) V$

where the mask matrix $M$ is:

$M_{ij} = {0 - \infty if j \leq i if j > i$

After softmax, every future position gets weight exactly zero. The attention matrix is lower triangular.

tokens can only attend to themselves and earlier positions.

Consequence of not masking: The model produces perfect training loss but fails catastrophically at inference.

because at inference, future tokens truly are not available. The model never learned to generate without peeking.

This looks like overfitting but is actually a data-leakage problem in the causal direction.

Worked Example — Masked Attention Matrix.

Sequence: ["<START>", "I", "love", "NLP"] (4 tokens).

Before masking, $Q K^{T} / d_{k}$ might yield:

$2.1 1.3 0.8 0.2 1.3 3.0 1.5 0.6 0.8 1.5 2.8 1.9 0.2 0.6 1.9 3.2$

Apply mask (set upper triangle to $- \infty$ ):

$2.1 1.3 0.8 0.2 - \infty 3.0 1.5 0.6 - \infty - \infty 2.8 1.9 - \infty - \infty - \infty 3.2$

After softmax per row:

$1.00 0.15 0.09 0.03 0 0.85 0.18 0.05 00 0.73 0.15 000 0.77$

Sense-check: Token 1 ("<START>") can only attend to itself. Token 2 ("I") attends mostly to itself but some to "<START>".

Token 4 ("NLP") attends to.

All previous tokens but with zero weight on future positions (there are none for the last token). The diagonal has high values because each token's self-match is strong.

16.4.2 Cross Attention

Intuition + Analogy. Masked self-attention lets.

The decoder understand its own output so far. But for translation, the decoder also needs to look at the source sentence.

Cross attention is the bridge. Think of it as.

A translator who keeps glancing back at the original text while writing the translation.

each generated word prompts a fresh look at the source.

Formalize. Cross attention is the decoder's connection to the encoder. The key difference from self-attention: where do $Q$ , $K$ , $V$ come from?

$CrossAttention (Q_{dec}, K_{enc}, V_{enc}) = softmax (\frac{Q _{dec} K _{enc}^{T}}{d _{k}}) V_{enc}$

Query $Q_{dec}$ : Comes from the decoder's current hidden state — "what am I looking for in the source?"
Keys $K_{enc}$ and Values $V_{enc}$ : Come from the encoder's output — "here is everything the source means."

Every decoder position can attend to every encoder position.

no masking needed for cross attention because the encoder's output is already fully known (bidirectional).

This pulls source-language context for translation, source-document context for summarization, or image-region context for captioning.

16.4.3 Decoder Stacking

Like encoders, decoder blocks stack. Each block contains three sub-layers (with pre-LN):

Masked multi-head self-attention — processes the decoder's own output so far.
Cross multi-head attention — attends to the encoder's output.
Feed-forward network — transforms each position independently.

Each sub-layer has its own layer normalization and residual connection.

A decoder with $N = 6$ blocks learns to generate through progressively refined representations.

16.4.4 Output Layer

After the final decoder block, the output goes through:

Linear transformation: Maps from $d_{model}$ to vocabulary size $V$ .
Softmax: Converts to a probability distribution over all tokens.

At each time step, the token with the highest probability is picked (greedy decoding).

Alternatively, top-k sampling selects from the top $k$ most probable tokens, adding diversity to the generation.

16.4.5 Parameter Comparison with Encoder

The decoder has about 33% more learnable parameters than the encoder:

Component	Encoder	Decoder
Self-attention weights	$W_{Q}, W_{K}, W_{V}, W_{O}$	$W_{Q}, W_{K}, W_{V}, W_{O}$ (masked)
Cross-attention weights	—	Extra $W_{Q}, W_{K}, W_{V}, W_{O}$
Layer norms	2 per block	3 per block
Feed-forward	1 per block	1 per block

The extra cross-attention weights (4 matrices) and extra layer norm (2 parameters) account for the 33% increase.

This makes the decoder slower to train than the encoder.

16.4.6 Symbol Registry — Decoder Components

$i$ — current decoder position — integer index
$j$ — key token position — integer index
$M$ — mask matrix — $M_{ij} = 0$ if $j \leq i$ , $- \infty$ if $j > i$
$Q_{dec}$ — query from decoder — $R^{n \times d_{k}}$
$K_{enc}$ — keys from encoder output — $R^{n_{enc} \times d_{k}}$
$V_{enc}$ — values from encoder output — $R^{n_{enc} \times d_{k}}$
$V$ — vocabulary size — scalar

16.4.7 Student Questions and Answers

Q: Why do we need masked attention in the decoder? Is.

It to differentiate and not learn too much from the decoder's input?

A: You are very close. The core reason: during training, the decoder has access to the entire target sequence.

Without masking, when predicting word two, it could peek at word three, word four, and so on. The machine would cheat.

it would take shortcuts instead of learning the true conditional dependencies..

It is like knowing the answer key in a math exam and reverse-engineering the steps.

Masking hides future tokens by setting their attention scores to $- \infty$ . That way the decoder learns to generate each word using only what came before.

Pitfalls — Common Traps.

Forgetting to mask during training but not inference. If you apply masking during training but forget to enforce causal generation during inference, the model produces garbage. It expects a mask that is not there.

If you fail to mask during training, the loss will look great but inference will fail.

Confusing cross attention with self-attention. In cross attention, keys and values come from the encoder.

In self-attention, they come from the decoder itself. Mixing them up means the model never looks at the source.

it hallucinates the output.

Teacher forcing trap. During training, the decoder receives the ground-truth previous token (teacher forcing).

During inference, it receives its own prediction..

A small error at step 3 can cascade into nonsense by step 10. This is called exposure bias.

Assuming greedy decoding is always best. Picking.

The single highest-probability token at every step can produce repetitive, boring text (e.g., "I am very very very very...").

Top-k or nucleus sampling adds controlled randomness.

Recap. The decoder extends the encoder with masked self-attention (prevents peeking at future tokens) and cross attention (looks at the encoder's output).

It has ~33% more parameters and is slower to train. The output layer converts hidden states to vocabulary probabilities for autoregressive generation.

Bridge. With both encoder and decoder fully understood, the next question is practical.

which architecture do you pick for your task? Section 16.5 answers that.

Real-World Connection. Masked self-attention is why GPT can generate coherent paragraphs.

it truly does not know word 100 when writing word 10. Cross attention is why Google Translate can translate this paragraph to Japanese.

the decoder generates each Japanese token while attending to the English source..

The teacher-forcing gap (training with ground-truth vs. inference with own predictions) is an active research problem.

techniques like scheduled sampling attempt to bridge it by occasionally feeding the model's own predictions during training.

---

16.5 Architecture Selection

Hook. You have three hammers. One is for pounding nails (classification). One is for pulling them out (generation).

One does both (translation). Picking.

The wrong hammer wastes weeks of training and fails at inference. The rule is simple: know your output.

16.5.1 When to Choose Each Architecture

Formalize — Decision Framework.

The architecture choice follows directly from the task type:

Architecture	Use when...	Examples
Encoder-only	You need to understand and classify the input. Output is a label, not a sequence.	Sentiment analysis, POS tagging, NER, span QA, semantic similarity
Decoder-only	You need to generate text from a prompt. No separate source sequence exists.	Chat, story generation, code completion
Encoder-decoder	You need to map one sequence to a different sequence.	Translation, summarization, image captioning

The decision tree:

Does your output require generating new text? → If no: encoder-only.
Is there a separate input sequence that must be transformed? → If yes: encoder-decoder.
Is the input just a prompt that gets continued? → Decoder-only.

Comparison — Architectural Trade-offs:

Dimension	Encoder-only	Decoder-only	Encoder-decoder
Attention	Bidirectional	Causal (masked)	Both
Parameters	Fewest	Medium	Most
Training speed	Fastest	Medium	Slowest
Inference speed	One pass	Sequential (slow)	Sequential (slowest)
Best for	Understanding	Generation	Transformation

16.5.2 Training vs Inference Behavior

Formalize.

Training (encoder-decoder):

Encoder: processes all $N$ source tokens at once using bidirectional self-attention. Runtime: parallel over all positions.
Decoder: processes target tokens with causal masking. Since all target tokens are known during training, the decoder also runs in parallel.

each position's prediction is computed simultaneously. The mask handles causality.

Loss: cross-entropy between predicted and actual next tokens, averaged over all positions.

Inference (generation):

Encoder: runs once, caches its output. Same as training.
Decoder: runs sequentially. Generate token 1 → feed it back as input → generate token 2 → ... Each step depends on the previous step's output. No parallelism possible in the autoregressive loop. This is why inference latency matters for real-time applications.

Pitfalls — Common Traps.

Using an encoder-only model for generation. BERT cannot generate coherent paragraphs.

It has no autoregressive decoder. If you need text output, you need at minimum a decoder.

Using a decoder-only model when you need bidirectional understanding. GPT sees only left context.

For tasks like "fill in the blank in the middle of.

A sentence," an encoder-only model like BERT is superior because it sees both sides.

Overlooking inference cost. Encoder-decoder models need both the encoder pass AND sequential decoder generation.

For real-time translation, this latency can be unacceptable.

streaming decoder-only models may be preferred.

Recap. Encoder-only for understanding, decoder-only for generation, encoder-decoder for transformation.

The choice is determined by one question: is your output a label, a continuation, or a transformation of a separate input?

Bridge. Now that we know which architecture to use, we need to understand how these models are trained before fine-tuning.

Section 16.6 covers pre-training for encoder-only models.

Real-World Connection. BERT (encoder-only) powers Google Search's query understanding.

GPT-4 (decoder-only) powers ChatGPT. Google Translate uses encoder-decoder transformers. The architecture choice is not academic.

it determines what the model physically can and cannot do. An interesting boundary case.

T5 treats every NLP task as text-to-text, forcing classification into a generation format ("sentiment: positive"). This blurs the line.

but T5 is still encoder-decoder under the hood.

---

16.6 Pre-training Encoder-Only Models — Masked Language Modeling

Hook. How do you teach a model to understand language without giving it labeled data? You give it a book, randomly erase 15% of the words, and ask: "What was here?" The model learns grammar, facts, and reasoning, all from the blank-filling game. This is masked language modeling, and it is how BERT learned to read.

16.6.1 The Masked Language Model (MLM) Objective

Intuition + Analogy. Your teacher hands you a paragraph with words blacked out.

"The [MASK] landed on the runway, its engines roaring." To fill in the blank, you use the surrounding words.

"landed," "runway," "engines." You infer "airplane." BERT learns the same way, but across billions of sentences.

The masking forces it to understand context, not just memorize keywords.

Formalize. MLM is a self-supervised pre-training objective:

Masking: Randomly select 10-30% of tokens in each input sequence. Replace each selected token with:

[MASK] token (80% of the time)
A random token (10% of the time)
The original token unchanged (10% of the time)

Prediction: The encoder processes the entire masked sequence bidirectionally. For each masked position, it outputs a probability distribution over the vocabulary:

$P (w_{i} ∣ x_{1}, \dots, [MASK]_{i}, \dots, x_{n})$

Loss: Cross-entropy between the predicted distribution and the true original token, summed over all masked positions.

The 80-10-10 split has a specific purpose.

if every masked position were [MASK], the model would never see [MASK] during fine-tuning (since downstream tasks don't use it).

The 10% random replacement forces the model to not blindly trust every token.

it must check context. The 10% unchanged keeps the representation aligned with real tokens.

Worked Example — MLM on a Sample Sentence.

Original sentence: "the movie was incredibly boring and poorly acted"

Position	1	2	3	4	5	6	7	8
Original	the	movie	was	incredibly	boring	and	poorly	acted
After masking (15% ≈ 1-2 tokens)	the	movie	was	[MASK]	boring	and	poorly	acted

The encoder sees: [CLS] the movie was [MASK] boring and poorly acted

The model's task: predict "incredibly" at position 4 using bidirectional context. It sees "was" (left) and "boring" (right), and knows the sentence expresses negative sentiment. It outputs probabilities over the full vocabulary. The correct token "incredibly" should receive the highest probability. Even if the model predicts "very" instead of "incredibly" (a semantically similar word), the loss is modest. The model still learned that an intensifying adverb fits in that slot.

Sense-check: The model cannot just memorize "incredibly boring" as a phrase — on other training iterations, a different word (like "boring" itself) might be masked instead. The model must learn the abstract relationship: negative adjectives follow intensifiers.

16.6.2 Why MLM Works

Formalize — Two Key Benefits.

Robustness to missing words: An MLM-trained model handles imperfect input.

spelling mistakes, missing words, unusual phrasing. It learned semantics from the full context, not from individual keywords.

If a sentence reads "The movie was [MASK]," the model still outputs positive/negative sentiment correctly by reading the surrounding words.

Prevents shortcut learning: Without masking, the model might latch onto one diagnostic word.

"difficult" always meaning negative sentiment. With 10-30% masking, many training instances hide that word.

The model must use the broader sentence structure. It learns deeper, more transferable features instead of brittle keyword associations.

16.6.3 The CLS Token

Formalize. A special [CLS] token is prepended to.

The start of every input sequence. Through bidirectional self-attention, every token (including [CLS]) attends to every other token across all encoder layers.

After processing through all layers, the output vector at the [CLS] position carries a fixed-size embedding that aggregates the entire sequence.

For classification tasks, this [CLS] output goes through a softmax classifier:

$P (class ∣ input) = softmax (W h_{[CLS]} + b)$

The [CLS] token has no inherent meaning.

it is a neutral token that learns to accumulate information through training. For.

The input "a very difficult product to use," the sequence becomes [CLS] a very difficult product to use.

The [CLS] embedding at the output layer carries the global sentiment signal.

16.6.4 Padding Masks

When a model expects fixed-length input (e.g., 512 tokens), shorter sequences get padding tokens appended at the end.

These padding positions use a special attention mask that sets their attention weights to zero.

the model ignores them during computation. Padding masks are separate from [MASK] tokens.

padding means "this position is empty, ignore it"; [MASK] means "predict what was here."

16.6.5 Popular Encoder-Only Models

BERT (Bidirectional Encoder Representations from Transformers).

the original, available in BERT-base (110M params, 12 layers) and BERT-large (340M params, 24 layers).

RoBERTa — a robustly optimized BERT variant with more data, longer training, and no next-sentence-prediction objective.
DistilBERT — a distilled version that retains 95% of BERT's performance with 40% fewer parameters.

16.6.6 Student Questions and Answers

Q: What if the most important word — the one carrying the sentiment — gets masked? Would the model fail?

A: It would not fail because you do millions of training iterations with different random masks each time. On some iterations the important word is masked.

on others it is not.

Across epochs, the model sees the same sentence with many different masking patterns..

It learns to rely on holistic context, not just one keyword. Also, real sentences tend to be longer with many words contributing to meaning.

the model aggregates evidence from all of them.

Q: Could the CLS token be applied directly at the softmax layer instead of being prepended at the input?

A: No. The CLS token must be present from.

The very first layer. It propagates through every encoder layer alongside the real tokens.

At each layer, it attends to and accumulates information from.

All other tokens. If you added it only at the softmax, it would have no accumulated context.

it would be an untrained, meaningless vector. Added at the input, its representation evolves layer by layer.

By the output layer, it truly represents the whole sequence.

Pitfalls — Common Traps.

Confusing [MASK] with padding tokens. They are completely different: [MASK] is a prediction target; padding tokens are placeholders for empty positions.
Thinking MLM only works when the exact word is predicted..

The model learns useful representations even when it predicts the wrong word.

the gradient from the error still teaches it about context and word relationships.

Not understanding the 80-10-10 split. If you only ever show [MASK],.

The model will not know what to do with real tokens during fine-tuning.

The 10% random and 10% unchanged tokens bridge the pre-training/fine-tuning gap.

Recap. MLM pre-trains encoders by masking random tokens and predicting them from bidirectional context.

The [CLS] token aggregates sequence-level information. The result is a model that deeply understands language and transfers to any classification task.

Bridge. Encoders learn by filling blanks. Decoders learn by predicting the next word.

Section 16.7 covers the autoregressive pre-training that powers GPT and other generative models.

Real-World Connection. BERT's MLM pre-training was trained on BooksCorpus (800M words) and English Wikipedia (2.5B words).

The cost: about \$7,000 in cloud compute for BERT-base in 2018 (4 TPUs for 4 days). Today,.

A fine-tuned BERT model can classify sentiment, extract answers, or tag entities on a laptop in minutes.

the pre-training cost is amortized across millions of downstream uses. Google uses BERT in nearly every search query processed worldwide.

---

16.7 Pre-training Decoder-Only Models — Autoregressive Task

Hook. Give a model one word.

"The." It predicts "cat." Feed back "The cat." It predicts "sat." Feed back "The cat sat.".

It predicts "on." This simple loop, repeated billions of times across the entire internet, is how GPT learned to write.

16.7.1 Autoregressive Pre-training

Intuition + Analogy. Imagine you are learning a language by reading a massive library, one sentence at a time.

After each word, you pause and guess.

The next one. If you are wrong, you adjust your mental model of how the language works. After reading millions of books this way, you can complete any prompt.

not because you memorized answers, but because you learned the patterns of language. This is autoregressive pre-training.

Formalize. Decoder-only models use a causal language modeling (CLM) objective.

Given a sequence of tokens $x_{1}, x_{2}, \dots, x_{n}$ , the model predicts each token using only the tokens before it:

$L = - t = 1 \sum n lo g P (x_{t} ∣ x_{1}, \dots, x_{t - 1})$

At each position $t$ , the model outputs a probability distribution over the vocabulary.

The loss is the sum of cross-entropy losses across.

All positions. Masked self-attention (causal masking) ensures position $t$ never sees tokens $x_{t + 1}, x_{t + 2}, \dots$

The key difference from MLM: autoregressive models predict every token (not just masked ones) and use only left context (not bidirectional).

This makes them natural generators.

during inference, they can continue any prefix by repeatedly predicting the next token.

16.7.2 Popular Decoder-Only Models

GPT (Generative Pre-trained Transformer) — the series from GPT-1 (117M params) through GPT-4 (estimated 1.7T+), all decoder-only autoregressive models.
ChatGPT — GPT fine-tuned with RLHF (Reinforcement Learning from Human Feedback) for conversational interaction.
Claude — Anthropic's decoder-only model family focused on safety and helpfulness.
LLaMA (Meta) — open-source decoder-only models available in 7B, 13B, 33B, 65B parameter sizes.

16.7.3 Use Cases

Any task where you give a prompt and want generated text: chatbots, creative writing, code generation, dialogue systems, content summarization, and question answering with generated answers.

Pitfalls — Common Traps.

Confusing autoregressive pre-training with MLM. MLM uses bidirectional context and masks random tokens.

Autoregressive uses only left context and predicts every next token. They produce fundamentally different models.

one understands, one generates.

Hallucination. Decoder-only models confidently generate false information because the autoregressive objective optimizes for plausible continuation, not factual accuracy.

A model trained to "predict the next word" has no built-in truth-checking mechanism.

Repetition loops. Without careful decoding strategies, autoregressive models can get stuck in repetitive cycles.

This happens because the model's own outputs reinforce the same high-probability tokens.

Recap. Autoregressive pre-training teaches decoders to predict the next token from previous tokens.

The objective is the sum of next-token cross-entropy losses. These models are the foundation of all modern chatbots and text generators.

Bridge. Having covered encoder-only (MLM) and decoder-only (autoregressive) pre-training, we now look at the combined approach: pre-training encoder-decoder models.

That is Section 16.8.

Real-World Connection. GPT-3 was trained on about 500 billion tokens from Common Crawl, WebText2, Books1/2, and Wikipedia.

The pre-training cost was estimated at \$4.6M-\$12M in compute. The key insight.

the autoregressive objective is so simple that.

It scales to any text data without any labeling. Every word on the internet becomes a training example automatically.

the task is always "predict the next word."

---

16.8 Pre-training Encoder-Decoder Models

Hook. Take a clean document. Scribble on it, tear out a paragraph, shuffle the sentences.

Now hand the mess to a model and say: "Rebuild the original." This game.

corrupting and reconstructing — is how encoder-decoder models like BART and T5 learn to transform sequences.

16.8.1 Sequence-to-Sequence Pre-training

Formalize. Encoder-decoder pre-training corrupts the input, encodes it, and decodes the reconstruction. Common objectives include:

Text infilling: Remove spans of text; the decoder regenerates them.
Denoising: Apply random noise (deletions, masking, shuffling); reconstruct the clean version.
Translation: One language → another (when parallel corpora exist).

16.8.2 BART and the Denoising Autoencoder

Intuition + Analogy. BART is like an archivist who receives water-damaged documents.

The encoder reads the smudged, incomplete version. It produces a context.

a mental model of what the document probably said. The decoder then writes out a clean, restored copy.

The archivist gets better by comparing the restored copy against the original undamaged document stored in the archive.

Formalize. BART uses a denoising autoencoder objective:

Take original input $X$ . Corrupt it: randomly delete tokens, mask spans, permute sentence order.
Pass corrupted $X ~$ through the encoder → context $c = f_{enc} (X ~)$ .
Decoder autoregressively generates $Y^= f_{dec} (c)$ .
Loss: cross-entropy between $Y^$ and the original, uncorrupted $X$ .

The original input serves as its own label.

this is self-supervised. No human annotation is needed. BART's corruptions are more aggressive than BERT's (span masking, sentence shuffling, deletion), forcing the decoder to learn stronger generative abilities.

Worked Example — BART Denoising.

Original input $X$ : "The quick brown fox jumps over the lazy dog."

Corrupt: Delete "quick" and "lazy", mask the span "jumps over".

Corrupted $X ~$ : "The [MASK] brown fox [MASK] [MASK] the [MASK] dog."

Encoder: Processes the corrupted text bidirectionally → produces context $c$ .

Decoder: Autoregressively generates from $c$ :

Step 1: "The"
Step 2: "quick"
Step 3: "brown"
...
Final: "The quick brown fox jumps over the lazy dog."

Loss: Cross-entropy between generated tokens and the original $X$ . The model is penalized for every wrong token. Over millions of such training examples, it learns to restore missing words, reorder sentences, and fill spans.

Sense-check: The more aggressive corruptions (whole span deletion) force BART's decoder to generate multiple coherent words, unlike BERT which only predicts single masked tokens. This is why BART excels at generation tasks like summarization.

16.8.3 T5 and MT5

T5 (Text-to-Text Transfer Transformer).

reformulates every NLP task as text generation. Input: "translate English to French:.

The cat sits." Output: "Le chat est assis." Same architecture handles translation, summarization, QA, and classification — all as text-to-text.

MT5 — multilingual T5 covering 101 languages.

16.8.4 Example Use Cases for Encoder-Decoder

Machine translation: Source language → target language.
Summarization: Long document → short summary.
Abstractive QA: Passage + question → generated answer in natural language.
Structured data to text: Database records → human-readable report.
Image captioning: Image (via vision encoder) → descriptive sentence.

Pitfalls — Common Traps.

Using cross-entropy loss when the professor says MSE. The professor's verbal description mentions "mean squared error loss" for BART's denoising.

In practice, BART and T5 use cross-entropy loss over the vocabulary..

The professor may be using MSE as a simplified illustration. On an exam, know the professor's phrasing.

Assuming all encoder-decoder models use the same pre-training. BART uses denoising, T5 uses span corruption, and some use translation.

The pre-training objective shapes what the model is best at downstream.

Forgetting that encoder-decoder models are slower to train than encoder-only or decoder-only..

They have more parameters (both encoder AND decoder weights) and need both passes.

Recap. Encoder-decoder models pre-train by corrupting input and reconstructing it (BART) or by casting every task as text-to-text (T5).

These models excel at sequence transformation tasks.

Bridge. Pre-training gives us powerful base models. But how do we adapt them to our specific task with limited data? Section 16.9 covers transfer learning and fine-tuning.

Real-World Connection. BART powers Facebook/Meta's content moderation and summarization pipelines.

T5's text-to-text paradigm is elegant: it means a single model checkpoint can do translation, summarization, and classification.

you just change the input prefix..

This "one model, many tasks" approach has influenced prompt engineering in GPT-style models, where the instruction itself becomes the task specification.

---

16.9 Transfer Learning and Fine-Tuning for Transformers

Hook. Training GPT-3 from scratch costs millions of dollars. But you can adapt it to your company's customer support logs for under \$100.

The trick: freeze the giant model and train only a tiny add-on.

a technique called LoRA.

16.9.1 Why Fine-Tune Instead of Training from Scratch

Formalize. Transfer learning reuses a pre-trained model's weights as a starting point.

The decision of how many layers to update depends on three factors:

Task similarity: How different is your task from pre-training? Text classification is close to MLM.

code generation is further from language modeling.

Data size: More labeled data → you can update more layers without overfitting.
Compute budget: Full fine-tuning updates all parameters (e.g., 175B for GPT-3). Parameter-efficient methods update far fewer.

Rule of thumb:

Closely related task + small data → freeze most layers, fine-tune only the top few.
Very different task + large data → full fine-tuning (update all parameters).
Always budget-constrained → use LoRA or similar parameter-efficient methods.

16.9.2 Low-Rank Adaptation (LoRA)

Intuition + Analogy. You borrow a friend's detailed recipe book (the pre-trained model). You want to adapt it to your kitchen's ingredients.

Instead of rewriting every page, you stick a small Post-it note on each recipe. The note says.

"Substitute olive oil for butter" or "Add 5 more minutes baking time." The original book stays untouched. The Post-it notes.

small, cheap, easy to swap — are LoRA adapters.

Formalize. LoRA freezes the original pre-trained weights $W \in R^{d \times d}$ and adds a trainable low-rank update:

$h = W x + B A x$

where the weight update is factorized as:

$Δ W = B A, B \in R^{d \times r}, A \in R^{r \times d}, r ≪ d$

$r$ is the rank — a small number like 4, 8, or 16.
$A$ is initialized with small random values; $B$ is initialized to zero (so training starts at the pre-trained model's behavior).
Only $A$ and $B$ are trained. $W$ stays frozen.

Why low rank works: The weight updates needed for fine-tuning have low "intrinsic dimension".

only a few directions in weight space need to change. The $r ≪ d$ constraint enforces this.

A full $d \times d$ update has $d^{2}$ parameters; LoRA has $2 d r$ parameters. For $d = 4096, r = 16$ .

full update = 16.8M params; LoRA = 131K params — a 128× reduction.

16.9.3 Benefits of LoRA

Reduced memory: Only small adapter matrices are stored and updated. A single GPU can fine-tune a 70B model.
Faster training: Fewer parameters to optimize → fewer gradient computations.
Multi-task serving: Keep one base model. Swap in different LoRA adapters for different tasks.

no need to reload the full model.

Interpretability: $Δ W = B A$ tells you exactly what changed from the base model. The singular vectors of $Δ W$ reveal which feature directions were most important for your task.

16.9.4 Symbol Registry — LoRA

$W$ — original pre-trained weight matrix — $R^{d \times d}$
$Δ W$ — weight update — $R^{d \times d}$
$B$ — first low-rank matrix — $R^{d \times r}$
$A$ — second low-rank matrix — $R^{r \times d}$
$r$ — rank — scalar, $r ≪ d$ (e.g., 4, 8, 16)
$d$ — hidden dimension — scalar
$x$ — input vector — $R^{d}$
$h$ — output — $h = W x + B A x$ — $R^{d}$

Pitfalls — Common Traps.

Setting rank $r$ too high. $r = 64$ or $r = 128$ with LoRA approaches full fine-tuning cost.

Start with $r = 8$ or $r = 16$ .

these values work well for most NLP tasks.

Applying LoRA to the wrong layers. LoRA is typically applied to the attention weight matrices ( $W_{Q}, W_{K}, W_{V}, W_{O}$ ).

Applying it to all layers may not improve results and costs more.

Thinking LoRA is always enough. For tasks radically different from pre-training (e.g., fine-tuning a language model on protein sequences), full fine-tuning may still be needed.
Forgetting that LoRA adapters are task-specific. An adapter trained for sentiment analysis will not help with translation.

Each task needs its own adapter.

Recap. Transfer learning with LoRA freezes the pre-trained model and trains only small low-rank adapter matrices ( $Δ W = B A$ ).

This achieves near full-fine-tuning quality at a fraction of the cost.

Bridge. Transformers are not just for text. Section 16.10 shows how the same architecture.

with a clever input trick — handles images, opening the door to vision transformers.

Real-World Connection. LoRA is the dominant fine-tuning technique in the open-source LLM community.

Fine-tuning LLaMA-7B with full parameters requires ~56GB GPU memory; with LoRA (r=16), it fits in ~16GB.

a single consumer GPU. Companies deploy one base LLaMA model and hundreds of task-specific LoRA adapters.

When a user asks a medical question, the medical LoRA adapter is activated. When.

They ask about code, the coding adapter takes over. Same base model, infinite specializations.

---

16.10 Vision Transformers

Hook. What if you could take the transformer.

designed for words — and feed it an image instead? The trick.

chop the image into little squares, treat each square as.

A "word," and let attention figure out which squares matter most. No convolutions needed.

16.10.1 Image Patches as Tokens

Intuition + Analogy. You have a jigsaw puzzle. Each piece shows a fragment of the big picture.

To understand the whole scene, you must look at how pieces relate.

the blue piece (sky) sits above the green piece (grass), the brown piece connects to the trunk.

A Vision Transformer (ViT) works the same way. It slices.

The image into patches (puzzle pieces) and uses self-attention to discover how they fit together.

Formalize. ViT replaces convolutions with pure attention:

Divide the input image of size $H \times W \times C$ into a grid of $P \times P$ patches.
Flatten each patch into a vector of length $P^{2} \cdot C$ .
Linearly project each flattened patch to dimension $d_{model}$ .
The patch embeddings form the input sequence — exactly like word embeddings in NLP.

For a $224 \times 224 \times 3$ image with $16 \times 16$ patches: there are $(224/16)^{2} = 196$ patches.

Each patch is a $16 \times 16 \times 3 = 768$ -dimensional vector, projected to $d_{model}$ (e.g., 768 or 1024). The sequence length is 196.

the model attends to all 196 patches simultaneously.

16.10.2 Positional Embedding for Patches

Since transformers have no spatial awareness, each patch gets a positional embedding.

In ViT, positional embeddings are typically learned (not fixed sinusoidal). Patch one learns embedding vector one.

patch two learns embedding vector two. These are added to the patch embeddings before the encoder.

Through training, the model learns that patches with similar positional embeddings are spatially close.

16.10.3 ViT for Image Classification

For classification, ViT uses an encoder-only architecture:

Patch embeddings + learned positional embeddings + [CLS] token → encoder stack.
Encoder applies multi-head self-attention across all patches. The model learns which regions matter.

the dog's face gets more attention than the background grass.

After $N$ encoder blocks, the [CLS] token's output goes through an MLP head + softmax → class prediction.

16.10.4 ViT for Image Captioning

For generation tasks, ViT becomes an encoder-decoder:

ViT encoder processes image patches → produces patch-level context vectors.
Text decoder attends to the encoder output via cross-attention → generates the caption word by word.
Architecture choice depends on the task: encoder-only for classification, encoder-decoder for captioning or visual QA.

16.10.5 Loss of Locality in ViT

Formalize — ViT vs. CNN.

CNNs use local receptive fields.

each neuron sees a small neighborhood of pixels. This builds in a strong inductive bias.

nearby pixels are related. ViT has no such bias. Every patch attends to every other patch from the first layer. Spatial relationships must be learned from data through positional embeddings.

Trade-off:

CNNs: strong locality prior → sample-efficient (good with small datasets), but may miss long-range dependencies.
ViT: no locality prior → less sample-efficient (needs more data to learn spatial patterns), but excels at long-range relationships from the start.

This is why ViT outperforms CNNs on large datasets (ImageNet-21k, JFT-300M) but CNNs still win on small datasets.

The inductive bias of convolutions is valuable when data is scarce.

16.10.6 Student Questions and Answers

Q: If the task is to take an image and write three sentences about it, what should the architecture be?

A: The output of the ViT encoder feeds into a decoder.

The decoder generates the sequence of words that forms the sentences. You pick the architecture based on the task.

not on the input modality. For image classification.

encoder-only is enough. For image captioning or visual question answering: you need encoder-decoder. Vision transformers are not limited to encoder-only architectures.

they provide the encoder, and you add the decoder when generation is needed.

Pitfalls — Common Traps.

Using ViT on tiny datasets. ViT needs large datasets because it lacks the locality inductive bias of CNNs.

Training ViT on 5000 images will likely underperform a simple CNN.

Forgetting positional embeddings. Without positional embeddings, the model treats patches as an unordered bag.

it cannot tell if a patch is in the top-left or bottom-right corner.

Assuming patch size doesn't matter. Smaller patches = more tokens = longer sequences = quadratic attention cost.

$14 \times 14$ patches on a $224 \times 224$ image give 256 tokens. $8 \times 8$ patches give 784 tokens.

nearly 10× the attention compute.

Recap. Vision Transformers treat image patches as tokens and apply the same transformer encoder. The key trade-off.

ViT lacks CNN's locality inductive bias, so it needs more data but excels at global relationships.

Architecture choice (encoder-only vs. encoder-decoder) depends on the output type, not the input type.

Bridge. Having covered transformers for both text and vision, Section 16.11 looks ahead at where the field is going.

multimodal models that handle text, images, and audio together.

Real-World Connection. ViT, introduced by Google Research in 2020 (Dosovitskiy et al.), matched or beat state-of-the-art CNNs on ImageNet while requiring fewer computational resources to train.

Today, ViT variants power image search in Google Photos, medical image analysis (detecting tumors in CT scans), and autonomous vehicle perception systems. The patch-as-token trick has been extended to video (tubelets as tokens), 3D point clouds (point patches), and audio (spectrogram patches).

---

16.11 Emerging Trends

Hook. A model that sees a photo, reads the caption in French, and answers in English.

all within one unified brain. This is not three separate pipelines glued together.

It is one transformer processing pixels, words, and sounds as the same kind of thing: tokens.

16.11.1 Multimodal Transformers

Intuition + Analogy. A human does not run three separate brains for text, vision, and sound.

You see a dog, hear it bark, and read the word "dog".

and all three inputs converge to the same concept in your mind. Multimodal transformers work the same way.

text tokens, image patches, and audio spectrogram slices all enter the same transformer.

The attention mechanism learns that the image patch showing fur,.

The sound wave of a bark, and the token "dog" are all connected.

Formalize. Multimodal transformers process different modalities as a unified token stream:

Text → sub-word tokens with learned embeddings.
Images → patch tokens (like ViT).
Audio → spectrogram patch tokens.

These tokens are interleaved into one sequence with modality-specific embeddings and positional encodings. The transformer's self-attention operates across modalities, learning cross-modal relationships.

e.g., the word "bark" attends to the audio token containing a bark sound..

This is fundamentally different from running separate models and combining outputs at the end.

the cross-modal learning happens during feature extraction, not just at the decision level.

16.11.2 Sparse Attention

Intuition + Analogy. When you read a 100-word sentence,.

The meaning of word 47 does not actually depend on all 99 other words.

It depends on maybe 5-10 nearby words and perhaps the subject from.

The beginning. Standard attention computes all 100 × 100 = 10,000 pairwise scores anyway. Sparse attention is like putting blinders on.

you compute only the scores that actually matter, skipping the noise.

Formalize. Standard self-attention has $O (n^{2})$ complexity in sequence length $n$ .

Sparse attention reduces this by restricting each query to attend to only a subset of keys:

Sliding window: Each token attends to $\pm k$ neighboring tokens (window size $2 k + 1$ ). Complexity: $O (nk)$ .
Dilated sliding window: Like dilated convolutions — skip tokens to capture longer-range dependencies while keeping the window sparse.
Global + local: A few "global" tokens attend to everything; all other tokens use local windows.
Random sparse: Each query attends to a random subset of keys.

This research area is active.

sparse attention is critical for processing very long sequences (100K+ tokens) where quadratic attention is computationally infeasible.

Recap. Multimodal transformers unify text, image, and audio processing into one architecture.

Sparse attention reduces the quadratic cost of self-attention for very long sequences by restricting each query to a subset of keys.

Bridge. Every model we have discussed.

from BERT to GPT to ViT.

needs to be trained. And training means optimization. Section 16.12 introduces the algorithms that make all of this possible.

Real-World Connection. OpenAI's GPT-4V (vision), Google's Gemini, and Meta's ImageBind are multimodal transformers.

They can look at a chart, read its axes, and explain the trend in plain language.

all within one model, not separate OCR + analysis pipelines. Sparse attention variants like Longformer and BigBird have enabled transformers to process entire books in one pass. This opens up document-level understanding beyond the typical 512-2048 token limits.

---

16.12 Introduction to Optimization

Hook. You have a model with 175 billion knobs (parameters). You need to find the one setting of those knobs that minimizes error across terabytes of data.

You cannot try every combination.

there are more combinations than atoms in the universe. The solution: follow the slope downhill, one small step at a time.

This is optimization, and every deep learning model depends on it.

16.12.1 Why Optimization Matters

Intuition + Analogy. You are blindfolded on a mountain.

Your goal is to reach the lowest valley. You can only feel the slope under your feet.

steep downhill, gentle slope, or flat ground. You take a step in the steepest downhill direction. Feel again. Step. Repeat.

If your steps are too big, you might jump across the valley entirely. Too small, and you may never reach.

The bottom before nightfall. The learning rate is your step size. The mountain is the loss surface. The blindfolded hiker is the optimizer.

Formalize. Optimization in deep learning seeks weight values $W$ that minimize a loss function $J (W)$ :

$W^{*} = ar g W min J (W)$

The standard approach is gradient descent:

$W_{t + 1} = W_{t} - η \nabla J (W_{t})$

where $η$ (eta) is the learning rate.

the step size. $\nabla J (W_{t})$ is the gradient.

the vector of partial derivatives telling us which direction increases the loss fastest. We move opposite the gradient to decrease the loss.

The quality of the final model depends on:

The shape of the loss surface (convex vs. non-convex, smooth vs. jagged)
Hyperparameter choices (learning rate, decay, momentum coefficient)
Weight initialization (where you start on the mountain)
The optimization algorithm variant (how you process data per update)

16.12.2 The Loss Surface and Its Complexity

The loss function $J (W)$ as a function of the weights is rarely a simple convex bowl. It can have:

Multiple local minima — dips that trap gradient descent.
Saddle points — flat regions where the gradient is zero but it is not a minimum.
Plateaus — large flat areas where gradients are near zero, stalling progress.
Narrow valleys — steep in some directions, flat in others, causing zigzag paths.

The search starts at some initial point (determined by weight initialization) and must navigate this complex terrain to reach a good minimum.

16.12.3 Symbol Registry — Learning Rate

$η$ — learning rate — controls step size — scalar, typically 0.1, 0.01, 0.001
$W_{t}$ — weight vector at step $t$
$J (W)$ — cost/loss function — scalar
$\nabla J (W)$ — gradient — vector of partial derivatives of $J$ with respect to each weight

16.12.4 Gradient Descent Variants

Formalize — Three Fundamental Variants.

Batch Gradient Descent: $W_{t + 1} = W_{t} - η \cdot \frac{1}{N} i = 1 \sum N \nabla J_{i} (W_{t})$ Uses all $N$ training instances per update. Trajectory is smooth.

steady movement toward the minimum. But each update processes the entire dataset, making it slow in wall-clock time. Smoothness: excellent. Speed: slow.

Mini-batch Gradient Descent: $W_{t + 1} = W_{t} - η \cdot \frac{1}{B} i \in B_{t} \sum \nabla J_{i} (W_{t})$ Uses a random batch $B_{t}$ of size $B$ (e.g., $B = 32, 64, 128$ ).

The trajectory oscillates because each batch is a noisy estimate of.

The true gradient. Some batches pull the weights away from the minimum.

others pull back. Smoothness: moderate. Speed: faster per update.

Stochastic Gradient Descent (SGD): $W_{t + 1} = W_{t} - η \cdot \nabla J_{i} (W_{t})$ Uses exactly one random instance $i$ per update. Time per update is $O (1)$ .

extremely fast. But the trajectory is the noisiest. A single outlier instance can push weights wildly off course.

SGD may skip past the global minimum entirely. Needs many more iterations overall, but each is cheap.

Worked Example — Comparing Variants on 1000 Instances.

Dataset: 1000 samples. Batch size $B = 100$ . Learning rate $η = 0.01$ .

Variant	Instances per update	Updates per epoch	Wall-clock time per epoch	Trajectory
Batch	1000	1	Slowest	Smooth
Mini-batch	100	10	Medium	Oscillating
SGD	1	1000	Fastest (per update)	Very noisy

Sense-check: Mini-batch (10 updates per epoch) strikes the best balance for most deep learning.

It is fast enough to handle large datasets and noisy enough to escape local minima, but not so noisy that it never converges.

16.12.5 Understanding Contour Plots

Visual Intuition. Picture a loss function of two weights, $W_{1}$ and $W_{2}$ .

The surface looks like a bowl if the function is convex. Looking from above, you see contours.

concentric rings. Each ring is a set of $(W_{1}, W_{2})$ values that produce the same loss.

Outermost ring: highest loss.
Each inner ring: progressively lower loss.
Innermost point: global minimum — $(W_{1}^{*}, W_{2}^{*})$ yielding the lowest possible loss.

The optimizer traces a path across these rings. Each dot on the path is $(W_{1}, W_{2})$ after one update.

Convergence means the dots move from outer rings to the innermost region..

The path can be smooth (batch), oscillating (mini-batch), or erratic (SGD) depending on the variant.

16.12.6 Student Questions and Answers

Q: What do the contours represent? Are they epochs?

A: The concentric loops are not epochs. They are contours of the loss function.

each ring is a set of $(W_{1}, W_{2})$ values that produce the same loss. The outermost contour has the maximum loss.

The innermost point has the minimum loss. The path traced across contours shows the optimizer's trajectory..

Each dot is $(W_{1}, W_{2})$ after one update. Convergence means the dots move from outer rings to the innermost region.

Q: So optimization is about managing the learning rate — should we not keep it constant?

A: Yes, adapting the learning rate is one key part of optimization..

A constant learning rate may be too high (overshooting the minimum) or too low (convergence is too slow).

Adaptive methods like Adam and RMSProp vary the learning rate per parameter during training.

Q: Is optimization also about making the model generalize better?

A: Yes. A model should neither overfit nor underfit. Good optimization helps reach a minimum that generalizes.

not just a training-set minimum. But many factors beyond optimization affect generalization: cost function choice, data quality, regularization, and hyperparameters.

16.12.7 The Learning Rate Size Effect

With a fixed learning rate:

Too high ( $η = 0.1$ ): The search overshoots, oscillates wildly, may miss the minimum entirely. Like taking 10-meter steps down a narrow valley.

you jump from one wall to the other.

Too low ( $η = 0.00001$ ): The search moves in tiny steps. Convergence is extremely slow. It may never reach the minimum within the training budget.
Just right: Steady progress toward the minimum without wild oscillations or excessive slowness.

Even with an optimal fixed learning rate, mini-batch or stochastic variance introduces oscillations. This is where momentum and adaptive learning rates help.

16.12.8 Momentum

Intuition + Analogy. You are cycling down a hilly road. On.

A steep slope, gravity pulls you down naturally. When you hit a flat patch or a small uphill bump, your speed from the previous downhill carries you through.

That carried-forward speed is momentum. Without.

It, you would stop on every small bump. Gradient descent stops at every local minimum. Momentum rolls through them.

Formalize. Momentum keeps a running average of past gradients and uses it to smooth and accelerate the weight update.

The professor's formulation — the one you should know for the exam:

$v_{t} [4 pt] W_{t + 1} = β v_{t - 1} + (1 - β) \nabla J (W_{t}) = W_{t} - η v_{t}$

where:

$v_{t}$ is the velocity (accumulated gradient history)
$β \in [0, 1)$ is the momentum coefficient — how much of the past velocity to retain (e.g., $β = 0.9$ )
$(1 - β)$ scales the current gradient contribution
$η$ is the learning rate

Relationship to the standard form: Standard texts (e.g., the Deep Learning book by Goodfellow et al.) write momentum as:

$v_{t} = α v_{t - 1} - ε \nabla J (θ_{t}), θ_{t + 1} = θ_{t} + v_{t}$

The professor's form is equivalent but uses a different sign convention and normalization. In the professor's version.

the negative sign is in the weight update ( $W_{t} - η v_{t}$ ).

The $(1 - β)$ factor normalizes the gradient so that $v_{t}$ stays as a weighted average bounded by the gradient magnitude. In.

The standard form, $v_{t}$ grows without bound if gradients are consistent, and the learning rate $ε$ controls the overall scale. Both forms describe the same algorithm.

the professor's version emphasizes the exponential moving average interpretation.

Why momentum helps: At a local minimum, $\nabla J (W_{t}) = 0$ ..

A pure gradient descent would stop. But momentum carries forward the accumulated past gradients ( $v_{t - 1}$ is non-zero).

This accumulated push can propel the search out of the local trap toward.

A better minimum. The velocity acts like a heavy ball rolling through small dips.

Worked Example — Momentum Escaping a Local Minimum.

Suppose the gradient history was strongly downhill for 100 steps ( $\nabla J \approx - 2$ each step). Then it hits a shallow local minimum where $\nabla J = 0$ .

Without momentum ( $β = 0$ ): $v_{t} = (1 - 0) \cdot 0 = 0$ .

Update: $W_{t + 1} = W_{t} - η \cdot 0 = W_{t}$ . Stuck.

With momentum ( $β = 0.9$ , $η = 0.01$ ): $v_{t - 1} [4 pt] v_{t} [4 pt] W_{t + 1} \approx \frac{1 - β}{1 - β} \cdot (- 2) = - 2 (steady state after many steps) = 0.9 \cdot (- 2) + 0.1 \cdot 0 = - 1.8 = W_{t} - 0.01 \cdot (- 1.8) = W_{t} + 0.018$

Sense-check: Even though the gradient is zero at this position, the accumulated velocity $v_{t} = - 1.8$ produces a step of $+ 0.018$ .

The optimizer keeps moving. Over several more steps, the velocity decays as $v_{t + k} = 0. 9^{k} \cdot (- 1.8)$ , eventually slowing to zero.

but by then, it may have escaped the local minimum and found a real downhill slope.

16.12.9 Adaptive Learning Rate

Formalize. Rather than a single fixed learning rate, adaptive methods adjust $η$ per parameter or over time:

Adagrad: Accumulates squared gradients.

divides learning rate by the square root of.

This sum. Parameters with large gradients get smaller updates; parameters with small gradients get larger updates.

RMSProp: Uses a leaky (exponential) average of squared gradients instead of a full sum, preventing the learning rate from decaying to zero.
Adam: Combines momentum (leaky average of gradients) with RMSProp (leaky average of squared gradients), plus bias correction.

Momentum provides acceleration — faster convergence. Adaptive learning rates provide stability — smoother convergence.

Together they form the basis of modern optimizers like Adam, which is the default choice for most transformer training.

Pitfalls — Common Traps.

Learning rate too high. The most common optimization failure.

The loss explodes to NaN. Start small ( $η = 0.001$ or $0.0001$ ) and increase if training is stable.

Confusing momentum coefficient $β$ with learning rate $η$ . They control different things: $η$ controls step size.

$β$ controls how much history is remembered. Common values: $β = 0.9$ or $0.99$ .

Thinking SGD is always worse than Adam. SGD with momentum can generalize better than Adam on some tasks (especially image classification).

Adam converges faster but may find sharper minima that generalize worse. This is an active research topic.

Ignoring the scale of gradients. If gradients are very small (vanishing gradient problem), momentum alone will not help.

the accumulated velocity will be tiny too. You may need better weight initialization or architecture changes (e.g., residual connections, LayerNorm).

Recap. Optimization finds the best weights by following the negative gradient downhill. Batch GD is smooth but slow.

mini-batch GD balances speed and noise; SGD is fast but noisy. Momentum carries past gradients forward to escape local minima.

Adaptive learning rates (Adam, RMSProp) adjust step sizes per parameter. Together, these techniques enable training models with billions of parameters.

Exam note: The examination may require you to connect optimization concepts from DNN with those from Mathematical Foundations for Machine Learning and Machine Learning courses.

Know the gradient descent variant comparison, momentum intuition and formulas, and adaptive learning rate motivation. The recorded lectures and prescribed materials are enough for exam preparation.

Real-World Connection. Adam is the default optimizer for training virtually every large transformer.

GPT, BERT, LLaMA all use Adam or AdamW (Adam with decoupled weight decay). The choice of optimizer hyperparameters can make the difference between a model that converges in 3 days versus 3 weeks on.

A cluster of 1000 GPUs. At the scale of GPT-3 training (\$5M+ compute), even a 10% efficiency improvement from better optimization saves hundreds of thousands of dollars. Learning rate schedules (cosine decay, warmup, linear decay) are standard practice.

the learning rate is typically increased linearly for a few thousand steps (warmup) then decayed, rather than kept constant.

---

Exam Guidance Summary

Exam note: The examination may require you to connect concepts from multiple courses.

Deep Neural Networks, Mathematical Foundations for Machine Learning, and Machine Learning.

Ensure you understand the relationships across subjects. The recorded lecture content and prescribed materials are enough for exam preparation.

no external resources are needed.

Key topics to prepare:

Encoder vs. decoder vs. encoder-decoder architecture selection. Know when to use each.

The decision tree: output is a label → encoder-only. Output is generated continuation of.

A prompt → decoder-only. Output is a transformation of a separate input → encoder-decoder.

Masked attention rationale. Understand why causal masking is necessary in decoder self-attention.

it prevents the decoder from peeking at future tokens during training, which would cause it to cheat and fail at inference.

Know the mask formula: $M_{ij} = 0$ if $j \leq i$ , $- \infty$ if $j > i$ .

Cross attention mechanics. Know that queries come from the decoder's current state, keys and values come from the encoder's output.

This is how the decoder accesses source information during generation.

MLM pre-training objective. Randomly mask 10-30% of tokens.

model predicts the originals using bidirectional context. Know the 80-10-10 split: 80% masked with [MASK], 10% random token, 10% unchanged.

CLS token role. Prepended at input.

its output embedding after all encoder layers encodes a global summary of the sequence, used for classification.

LoRA fine-tuning mechanism. Freeze $W$ , learn $Δ W = B A$ with low-rank factorization.

Forward pass: $h = W x + B A x$ . Know the parameter count advantage: $2 d r$ vs. $d^{2}$ .

Gradient descent variant comparison. Batch (all data, smooth, slow), mini-batch (subset, oscillating, faster per update), SGD (single instance, noisy, fastest per update).

Know trade-offs in smoothness vs. speed.

Momentum intuition and formulas. Know the professor's formulation.

$v_{t} = β v_{t - 1} + (1 - β) \nabla J (W_{t})$ , $W_{t + 1} = W_{t} - η v_{t}$ .

Understand why momentum helps escape local minima (accumulated past gradients provide non-zero velocity even when current gradient is zero).

Adaptive learning rate motivation. Fixed learning rates cannot handle parameters with different gradient scales.

Adaptive methods (Adam, RMSProp) adjust the learning rate per parameter.

The optimization content in this session is drawn from Mathematical Foundations for Machine Learning.

You are expected to know nonlinear optimization concepts from that parallel course, including Adam, RMSProp, and momentum variants.

---

Key Industry Applications

The architectures and techniques covered in this lecture power the vast majority of modern AI systems deployed in production today.

Encoder-Only Models:

BERT (Google, 2018) — powers Google Search query understanding, document ranking, and featured snippets across billions of daily queries. Available in BERT-base (110M parameters, 12 layers) and BERT-large (340M, 24 layers).
RoBERTa (Meta, 2019) — an optimized BERT variant trained on more data with better hyperparameters. Widely used for text classification, sentiment analysis, and content moderation.
DistilBERT (Hugging Face, 2019).

a distilled version retaining 95% of BERT's performance with 40% fewer parameters. Used in latency-sensitive production systems and edge devices.

Decoder-Only Models:

GPT-1 through GPT-4 (OpenAI, 2018-2023).

autoregressive models scaling from 117M to.

An estimated 1.7T+ parameters. GPT-3.5 and GPT-4 power ChatGPT, GitHub Copilot, and the OpenAI API serving millions of developers.

Claude (Anthropic, 2023) — a decoder-only model family focused on safety and constitutional AI, used in enterprise chat and document analysis.
LLaMA (Meta, 2023) — open-source decoder-only models (7B, 13B, 33B, 65B, 70B parameters). Available for research and commercial fine-tuning, powering the majority of open-source LLM applications.

Encoder-Decoder Models:

T5 (Google, 2019) — treats every NLP task as text-to-text. Used in Google's internal systems for translation, summarization, and question answering.
MT5 (Google, 2020) — extends T5 to 101 languages for multilingual applications.
BART (Meta, 2019) — denoising autoencoder for text generation and summarization. Powers Facebook's content understanding and abstractive summarization pipelines.

Vision Transformers (ViT):

Image classification at scale (ImageNet-21k, JFT-300M), object detection, semantic segmentation, medical image analysis, and autonomous vehicle perception.

Fine-Tuning Techniques:

LoRA (Microsoft, 2021) — the dominant parameter-efficient fine-tuning method. Enables fine-tuning of 70B+ models on consumer GPUs. Adapters can be swapped at runtime for multi-task serving. Used by virtually every open-source LLM deployment.

Emerging:

Sparse attention — active research for processing documents with 100K+ tokens (entire books, legal documents, genomic sequences).
Multimodal transformers — GPT-4V, Google Gemini, Meta ImageBind — combining text, image, and audio understanding in unified architectures.

---

DNN Lecture 16 notes · Transformer Architectures and Optimization

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

1Encoder and Decoder — Purpose and Intuition

2Attention Mechanisms — Recap

3Encoder Architecture Components

4Decoder Architecture Components

5Architecture Selection

6Pre-training Encoder-Only Models — Masked Language Modeling

7Pre-training Decoder-Only Models — Autoregressive Task

8Pre-training Encoder-Decoder Models

9Transfer Learning and Fine-Tuning for Transformers

10Vision Transformers

11Emerging Trends

12Introduction to Optimization

13Exam Guidance Summary

14Key Industry Applications

Postgraduate students in Deep Neural Networks

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Encoder vs Decoder

Must-know: An encoder reads input and produces context vectors for understanding. A decoder generates new output token by token using autoregressive generation. The architecture choice depends on whether you need understanding, generation, or both.

$c = f_{enc} (x_{1}, \dots, x_{N}), y_{t} = f_{dec} (y_{1}, \dots, y_{t - 1}, c)$

⚠️ Top pitfall: Confusing encoder output with generated text — the encoder produces vectors, not words. The decoder is what generates text.

Self-check: If you need to classify a sentence as positive or negative, which architecture do you use?

Connects to: Self-Attention, Cross-Attention, Sequence-to-Sequence Models

Self-Attention

Must-know: Self-attention computes a weighted sum of all token values, with weights determined by query-key compatibility. The formula is \text{Attention}(Q,K,V) = \text{softmax}(QK^T/\sqrt{d_k})V.

$Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$

⚠️ Top pitfall: Forgetting the \sqrt{d_k} scaling factor. Without it, large d_k pushes softmax into near-one-hot territory, causing vanishing gradients.

Self-check: Why does self-attention require positional embeddings?

Connects to: Multi-Head Attention, Positional Encoding, Scaled Dot-Product

Positional Encoding

Must-know: Positional encoding adds a sinusoidal vector to each token embedding so the model can use token order. Without it, self-attention is permutation-invariant.

$p_{i, 2 j} = sin (i /1000 0^{2 j / d_{model}}), p_{i, 2 j + 1} = cos (i /1000 0^{2 j / d_{model}})$

⚠️ Top pitfall: Thinking positional embeddings are optional — they are mandatory for any transformer using self-attention.

Self-check: What would happen to 'dog bites man' vs 'man bites dog' without positional encoding?

Connects to: Self-Attention, Embedding Layer

Multi-Head Attention

Must-know: Multi-head attention runs h independent attention heads in parallel, each in a reduced dimension d_k = d_model / h. Outputs are concatenated and projected back.

$MultiHead (X) = Concat (head_{1}, \dots, head_{h}) W_{O}$

⚠️ Top pitfall: Using multi-head attention but forgetting the output projection W_O — concatenating without mixing means heads cannot share information.

Self-check: Why does multi-head attention cost about the same as single-head attention despite having more parameters?

Connects to: Self-Attention, Scaled Dot-Product Attention

Layer Normalization

Must-know: LayerNorm normalizes each token's feature vector to zero mean and unit variance independently. It uses learnable scale \gamma and shift \beta parameters.

$LayerNorm (x) = γ ⊙ \frac{x - μ}{σ} + β$

⚠️ Top pitfall: Confusing LayerNorm with BatchNorm. LayerNorm normalizes across features per token; BatchNorm normalizes across the batch per feature.

Self-check: Why is LayerNorm preferred over BatchNorm in transformers?

Connects to: Residual Connections, Pre-LN vs Post-LN

Residual Connections

Must-know: Residual connections add the input of a sub-layer to its output, so the sub-layer only learns the residual. Gradients flow directly through the skip connection, preventing vanishing gradients in deep stacks.

$Output = SubLayer (x) + x$

⚠️ Top pitfall: Stacking blocks without residual connections — in deep transformers (12+ layers), the skip path is essential for training.

Self-check: How does a residual connection help gradient flow during backpropagation?

Connects to: Layer Normalization, Encoder Stacking

Masked Self-Attention (Causal Attention)

Must-know: Masked attention prevents the decoder from peeking at future tokens by setting attention scores for future positions to -\infty before softmax. The mask matrix M has M_{ij}=0 if j\leq i and M_{ij}=-\infty if j>i.

$MaskedAttention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}} + M) V$

⚠️ Top pitfall: Forgetting to mask during training — the model will have perfect training loss but fail at inference because it never learned to generate without peeking.

Self-check: Why can't a decoder-only model (like GPT) use bidirectional attention?

Connects to: Cross-Attention, Autoregressive Generation

Cross-Attention

Must-know: Cross-attention lets the decoder attend to the encoder's output. Queries come from the decoder, keys and values from the encoder. No masking is needed because the encoder output is fully known.

$CrossAttention (Q_{dec}, K_{enc}, V_{enc}) = softmax (\frac{Q _{dec} K _{enc}^{T}}{d _{k}}) V_{enc}$

⚠️ Top pitfall: Confusing cross-attention with self-attention — in cross-attention, K and V come from the encoder, not the decoder.

Self-check: In an encoder-decoder translation model, where do the keys and values in cross-attention come from?

Connects to: Encoder-Decoder Architecture, Masked Self-Attention

Architecture Selection

Must-know: Encoder-only for understanding tasks (classification, NER). Decoder-only for generation from a prompt (chat, story writing). Encoder-decoder for sequence transformation (translation, summarization).

$Decision: Output is a label \to Encoder-only Output is a continuation \to Decoder-only Output transforms a separate input \to Encoder-decoder$

⚠️ Top pitfall: Using an encoder-only model for generation — BERT cannot generate coherent text. Using a decoder-only model for fill-in-the-blank — GPT cannot see right context.

Self-check: What architecture would you use for image captioning (image → text)?

Connects to: Encoder, Decoder, Encoder-Decoder

Masked Language Modeling (MLM)

Must-know: MLM pre-trains encoders by randomly masking 15% of tokens and predicting them from bidirectional context. The 80-10-10 split (80% [MASK], 10% random, 10% unchanged) bridges pre-training and fine-tuning.

$P (w_{i} ∣ x_{1}, \dots, [MASK]_{i}, \dots, x_{n})$

⚠️ Top pitfall: Confusing [MASK] with padding tokens — [MASK] is a prediction target, padding marks empty positions. Also, not understanding why the 80-10-10 split matters.

Self-check: Why does MLM use 10% unchanged tokens instead of always masking?

Connects to: CLS Token, BERT, Self-Supervised Learning

Autoregressive Pre-training

Must-know: Decoder-only models use causal language modeling: predict every next token using only the tokens before it. The loss is the sum of cross-entropy losses over all positions. Every token on the internet becomes a training example.

$L = - t = 1 \sum n lo g P (x_{t} ∣ x_{1}, \dots, x_{t - 1})$

⚠️ Top pitfall: Confusing autoregressive pre-training with MLM — one uses left context only and predicts every token; the other uses bidirectional context and predicts only masked tokens.

Self-check: Why does autoregressive training have no masking strategy like MLM's 80-10-10?

Connects to: GPT, Decoder-Only, Causal Masking

LoRA (Low-Rank Adaptation)

Must-know: LoRA freezes pre-trained weights and adds trainable low-rank matrices \Delta W = BA with r \ll d. For d=4096, r=16: full update = 16.8M parameters, LoRA = 131K parameters (128x reduction).

$h = W x + B A x, B \in R^{d \times r}, A \in R^{r \times d}$

⚠️ Top pitfall: Setting rank r too high (r=64+) approaches full fine-tuning cost. Start with r=8 or r=16. Also, LoRA adapters are task-specific.

Self-check: Why can a single GPU fine-tune a 70B parameter model with LoRA but not with full fine-tuning?

Connects to: Transfer Learning, Fine-Tuning, Parameter-Efficient Methods

Vision Transformers (ViT)

Must-know: ViT treats image patches as tokens — divides H\times W image into P\times P patches, flattens and projects each to d_model. For 224\times224 with 16\times16 patches: 196 patch tokens. ViT lacks CNN's locality inductive bias.

$Patches = (H / P) \times (W / P), Each patch \in R^{P^{2} \cdot C}$

⚠️ Top pitfall: Using ViT on tiny datasets — without locality bias, ViT needs large data (ImageNet-21k+) to match CNN performance.

Self-check: Why does ViT need more data than a CNN to achieve comparable performance?

Connects to: Encoder, Positional Embedding, Image Classification

Gradient Descent Variants

Must-know: Batch GD uses all N instances per update (smooth, slow). Mini-batch GD uses a batch of size B (balanced). SGD uses one instance (noisy, fast). Mini-batch is the standard for deep learning.

$W_{t + 1} = W_{t} - η \cdot \frac{1}{B} i \in B_{t} \sum \nabla J_{i} (W_{t})$

⚠️ Top pitfall: Thinking SGD is always worse than Adam — SGD with momentum can generalize better on some tasks (e.g., image classification), even though Adam converges faster.

Self-check: Why does mini-batch GD strike the best balance for most deep learning tasks?

Connects to: Momentum, Adaptive Learning Rate, Learning Rate

Momentum

Must-know: Momentum keeps a running average of past gradients (velocity v_t). It helps escape local minima because accumulated past gradients provide non-zero velocity even when the current gradient is zero.

$v_{t} = β v_{t - 1} + (1 - β) \nabla J (W_{t}), W_{t + 1} = W_{t} - η v_{t}$

⚠️ Top pitfall: Confusing momentum coefficient \beta with learning rate \eta. \beta controls how much history is remembered (typically 0.9 or 0.99); \eta controls step size.

Self-check: How does momentum help an optimizer escape a local minimum?

Connects to: Gradient Descent, Adaptive Learning Rate, Optimization

Previous LectureLecture 15 · Attention Mechanisms and Transformer Architecture

Next LectureLecture 17 · Optimization and Regularization

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.