Attention Mechanisms and Transformer Architecture

15.1 Review -- Limitations of RNNs and the Encoder-Decoder Architecture

15.1.1 RNN Limitations

Traditional recurrent neural networks hit three hard walls.

The length mismatch problem. A standard RNN expects the input sequence length to equal the output sequence length. Part-of-speech tagging works because each input word gets exactly one output tag. But translation does not. An English sentence of six words may need a French translation of ten words. A question of several words maps to an answer with an unknown number of words. The RNN has no natural way to handle this mismatch.

The long-range memory problem. Even LSTMs, with their explicit memory cells, start forgetting after about 100 to 200 tokens. The hidden state is a fixed-size vector. You cannot cram an entire paragraph into it. Earlier information gets overwritten by later information.

The sequential bottleneck. RNN processing is strictly step-by-step. You must wait for timestep $t$ to finish before starting timestep $t+1$. There is no way to process all words at once. This kills parallelism and makes training on large datasets painfully slow.

15.1.2 The Encoder-Decoder Solution

Encoder and decoder are not fixed architectures. Each is simply a neural network playing a role. You can build the encoder with an RNN, a CNN, or any other network. The decoder can also be any network type. The names describe what the component does, not how it is implemented.

The encoder extracts the pattern. The decoder uses that pattern to produce a sequence. This clean split handles the input-output length mismatch.

15.1.3 The Context Bottleneck Problem

The encoder-decoder design has its own failure point: the context vector itself.

Two specific problems arise. First, what if the context vector is poor? What if the encoder, being an RNN, focused mostly on the final few words and forgot the early ones? The resulting context misrepresents the input, and the decoder generates a poor output.

Second, in the basic design, the context vector feeds only the first timestep of the decoder. By the time the decoder reaches its tenth or twentieth output word, the original context may have faded entirely from the decoder's internal state.

The deeper fix -- the one that changed everything -- is the attention mechanism. Instead of one fixed context vector, attention lets the decoder consult different parts of the input at every decoding step. This is the topic of the next section.

15.1.4 Student Questions and Answers

Real-world connection: The encoder-decoder pattern shows up far beyond NLP. Image captioning uses a CNN encoder (to understand the image) and an RNN/Transformer decoder (to write the caption). Speech recognition uses an encoder to process the audio waveform and a decoder to produce the text output. The same split -- encode into a latent space, then decode from it -- underpins generative models like variational autoencoders (VAEs) and diffusion models.

15.1.5 Visual Intuition

Picture three diagrams side by side.

Diagram 1 -- Standard RNN: A horizontal chain of boxes, one per input word. Each box connects to exactly one output box below it. Input length equals output length. This is the part-of-speech tagging setup.

Diagram 2 -- Encoder-Decoder: Two separate blocks. On the left, a stack of input words feed into the encoder block, which outputs a single vector (the context). An arrow carries that vector across to the decoder block on the right, which produces output words one at a time. Input length and output length are independent.

Diagram 3 -- Context bottleneck failure: The same encoder-decoder diagram, but now a long input sentence is shown feeding into the encoder, and the single context vector is drawn as a tiny bottle neck. Information spills out the sides -- the vector is too small. This is why attention was invented.

15.1.6 Assumptions, Scope, and Pitfalls

15.1.7 Recap and Bridge

15.2 Attention Mechanism -- Core Intuition

15.2.1 Customized Context Per Timestep

That is the core idea behind attention. Produce a different context vector at every decoder timestep. Each context is customized -- it gives more weight to the input words that are relevant to the current output word and less weight to everything else.

This solves the context bottleneck. You do not need one vector to capture everything. You build a fresh, targeted summary at every step, pulling in exactly the information you need for that step.

15.2.2 Scoring, Keys, Queries, and Values

The scoring process, step by step:

1. Compare the query with every key. Each comparison gives a score -- a number that says how well this input word matches what you are trying to produce right now.
2. Apply Softmax to the scores. This turns them into a probability distribution (the attention weights). All weights sum to 1.
3. Weight the values. Multiply each input word's value vector by its attention weight.
4. Sum everything up. Add the weighted values together. This sum is the customized context vector for the current decoder timestep.

The professor put it plainly: "The machine learns to give more importance, more score, or more weight to some nodes. And it gives lesser weight to all other nodes."

15.2.3 Attention Heatmaps and Interpretability

A valuable side effect of attention is interpretability. You can draw an attention heatmap: a grid where rows are input words, columns are output words, and the color at each cell shows how much attention the output word paid to that input word. Lighter colors mean stronger influence.

15.2.4 Visual Intuition

Draw a matrix where rows are the four input words ["this", "is", "very", "good"] and columns are the three output words ["c'est", "tres", "bien"]. Color each cell by attention weight.

At column 1 ("c'est"), the row for "this" lights up brightest. At column 2 ("tres"), the row for "very" dominates. At column 3 ("bien"), the row for "good" is brightest. The diagonal-like pattern shows the model learned word-level alignment -- exactly what you would expect from translation. The beauty is that it learned this alignment entirely from data, without any explicit alignment supervision.

15.2.5 Assumptions, Scope, and Pitfalls

15.2.6 Recap and Bridge

Real-world connection: Attention mechanisms are now used well beyond NLP. In computer vision, attention helps models focus on relevant image regions when generating captions or answering visual questions. In speech processing, attention aligns acoustic frames with phoneme sequences. In recommendation systems, attention weights capture which past user behaviors are most relevant to predicting the next click.

15.3 Types of Attention

15.3.1 Self-Attention

Application -- Word sense disambiguation. Consider "The bank is near." The word "bank" could be a financial institution or a riverbank. Self-attention helps disambiguate by letting "bank" attend to "the," "is," and "near." The surrounding context words provide the clues that resolve the meaning. Without self-attention, the model would see "bank" in isolation and have no way to know.

The diagonal of the attention matrix (a word attending to itself) is usually the strongest. In practice, these self-scores are often neglected or masked because a word always relates strongly to itself. What matters is the off-diagonal -- the cross-word influence.

Key property: Self-attention eliminates the need for recurrent connections. Every word directly interacts with every other word. All outputs can be computed simultaneously. Parallelism is finally possible.

The professor emphasized: "In self-attention, you take the essence from every input. You compare query, key, and value within the same sequence. The sequential dependency is eliminated."

15.3.2 Multi-Headed Attention

Worked example: Suppose you have 3 heads, each producing a 4-dimensional output vector. Concatenation gives a 12-dimensional vector. The output matrix $W_O$ (shape $12\times 4$) compresses this back to 4 dimensions so the result can feed into the next layer.

The number of heads is a hyperparameter you choose. Typical values: 8 heads (base Transformer), 12 or 16 heads (larger models). This is the architectural centerpiece of the Transformer.

15.3.3 Cross-Attention

Cross-attention is the bridge in any sequence-to-sequence task: machine translation, text summarization, question answering, dialogue systems. Wherever two distinct sequences interact, cross-attention is the link.

15.3.4 Masked Attention

Example: For the sentence "the customer was very happy with the service," consider the word "very" at position 4. When computing attention at position 4, all positions beyond 4 get masked. The words "happy," "with," "the," and "service" are hidden. The model sees only "the," "customer," "was," and "very."

15.3.5 Comparison of Attention Types

When to pick which: Use self-attention when working within one sequence. Add multiple heads when you need to capture different relationship types. Use cross-attention when you have two distinct sequences to align. Apply masking whenever generation must be autoregressive.

15.3.6 Visual Intuition

Draw an attention grid. For self-attention, both axes are the same sentence: ["the", "bank", "is", "near"]. Every cell is filled because every word attends to every other word. The diagonal is brightest (self-attention).

For cross-attention, the y-axis is the target sentence being generated and the x-axis is the source sentence. The pattern is not square -- it shows alignment between languages.

For masked attention, the grid is triangular. All positions above the diagonal are blacked out (masked). Each position can only see itself and everything to its left.

15.3.7 Assumptions, Scope, and Pitfalls

15.3.8 Recap and Bridge

Real-world connection: These attention types are not just theoretical. Every time you use ChatGPT, masked multi-headed self-attention runs across your prompt and the model's ongoing generation. Every time Google Translate processes a sentence, cross-attention aligns source and target languages. In vision Transformers (ViT), self-attention lets every image patch attend to every other patch, replacing convolution with attention entirely.

15.4 Positional Encoding

15.4.1 Why Position Matters

RNNs never had this problem. Words were fed in one by one -- order came for free. Transformers throw away the sequential processing and must add position information back explicitly.

15.4.2 Sinusoidal Encoding Formula

How the frequencies work: The denominator $10000^{2i/d_{\text{model}}}$ controls the wavelength. When $i$ is small (early dimensions), the exponent is near 0, so the denominator is near 1 -- the sine and cosine oscillate slowly across positions. When $i$ is large (later dimensions), the denominator grows large -- the sine and cosine oscillate rapidly across positions.

This creates a pattern where each position gets a unique vector, and positions that are nearby in the sequence have similar encodings in the low-frequency dimensions. The model can learn to interpret these patterns as relative positions.

Key property -- relative position: The sinusoidal encoding has a useful mathematical property. For any fixed offset $\delta$, the encoding at position $pos+\delta$ can be expressed as a linear transformation of the encoding at position $pos$. This means the model can learn to attend to relative positions (e.g., "the word three positions before the current one") by learning simple linear functions of the positional encoding.

15.4.3 Worked Example -- Positional Encoding with Two-Dimensional Vectors

15.4.4 Visual Intuition

Imagine a heatmap where rows are positions (0 to $n-1$) and columns are encoding dimensions (0 to $d_{\text{model}}-1$). Each cell is colored by the sine or cosine value at that position and dimension.

In the first few columns (low frequency), the colors change gradually as you move down rows -- like a slow wave. In later columns (high frequency), the colors alternate rapidly between rows. This gives each row a unique "barcode" pattern. Two nearby rows have similar patterns in the slow-changing columns, encoding their proximity.

15.4.5 Assumptions, Scope, and Pitfalls

15.4.6 Recap and Bridge

Real-world connection: Positional encoding is not unique to text. In vision Transformers, image patches need 2D positional encodings to encode row and column. In audio Transformers, time steps need 1D position information. The same principle -- add a position-dependent signal to the input -- applies across modalities. Modern large models like GPT-4 use learned positional embeddings, while some newer architectures like Llama use rotary position embeddings (RoPE), which encode relative position directly into the attention computation.

15.5 Self-Attention -- Worked Numerical Example (Dot-Product Scoring)

15.5.1 Setup -- Two Words: "playing" and "outside"

Each word starts with a pre-trained embedding. These embeddings are then multiplied by learned projection matrices $W_Q,W_K,W_V$ to produce the query, key, and value vectors. This is step one of any attention computation -- you never use raw embeddings directly.

15.5.2 Computing Query, Key, and Value Representations

For our example, assume the projection step has already been done, yielding these vectors:

• For "playing": $Q_{\text{play}},K_{\text{play}},V_{\text{play}}$
• For "outside": $Q_{\text{out}},K_{\text{out}},V_{\text{out}}$

15.5.3 Scaled Dot-Product Scoring -- First Timestep (query = "playing")

At timestep 1, the query is $Q_{\text{play}}$ (what "playing" is looking for). We compare this query against the key vectors of all words, including "playing" itself.

Why scale by $\sqrt{d_k}$? Assume each element of $Q$ and $K$ is drawn independently with mean 0 and variance 1. Their dot product has variance $d_k$. Without scaling, for large $d_k$, the dot products become very large in magnitude. After exponentiation in Softmax, a few elements dominate, and gradients vanish. Dividing by $\sqrt{d_k}$ keeps the variance at 1 regardless of dimension.

Score computation for timestep 1:

$${\text{score}}_{\text{play, play}}=\frac{Q_{\text{play}}\cdot K_{\text{play}}}{\sqrt{d_k}}$$

$${\text{score}}_{\text{play, out}}=\frac{Q_{\text{play}}\cdot K_{\text{out}}}{\sqrt{d_k}}$$

In this example, $d_k=4$, so each dot product is divided by $\sqrt{4}=2$.

15.5.4 Softmax and Weighted Context Vector

The weights tell you the proportional influence of each word. Then compute the weighted sum of values:

$${\text{context}}_{t=1}={\alpha }_{\text{play, play}}\cdot V_{\text{play}}+{\alpha }_{\text{play, out}}\cdot V_{\text{out}}$$

This context vector is the customized essence for "playing." It captures both what "playing" itself means and how "outside" relates to it. This context feeds into a classification layer that predicts the part-of-speech tag (e.g., verb).

Second timestep (query = "outside"): The process is identical. The query is now $Q_{\text{out}}$. The keys and values are reused -- same vectors, different query. New scores, new Softmax weights, new weighted sum. The resulting context captures the essence for "outside."

Exam note: Expect numerical problems on scaled dot-product self-attention in the EC3 exam. Practice with a small concrete example using explicit vector values. Also practice the same problem using Bahdanau's additive attention (Section 15.6) as an alternative scoring method.

15.5.5 Visual Intuition

Draw a 2x2 grid. Columns are keys ("playing", "outside"). Rows are queries ("playing", "outside"). Each cell holds the raw score between that query-key pair.

Below the grid, show the Softmax-then-weighted-sum pipeline: arrows from each cell to the Softmax nodes, then to the weighted values, then summed into the context vectors. Two context vectors emerge -- one per query -- and each feeds into its own part-of-speech classifier.

15.5.6 Assumptions, Scope, and Pitfalls

15.5.7 Recap and Bridge

Real-world connection: The scaled dot-product attention is the exact mechanism used in every transformer block of GPT, BERT, and all modern LLMs. The scaling factor $\sqrt{d_k}$ is one of those small but critical details -- remove it, and training large models becomes unstable. The same mechanism also appears in recommendation systems (user embeddings as queries, item embeddings as keys/values) and graph neural networks (node features attending to neighbor features).

15.6 Additive (Bahdanau) Attention

15.6.1 Definition and Difference from Dot-Product Attention

How it works: The query and key are each transformed by their own weight matrices into a common hidden space of dimension $h$. They are added together (hence "additive"), passed through $\tanh$, and then projected to a single scalar by $w_v$. This scalar is the attention score. It then goes through Softmax to produce attention weights, just like in dot-product attention.

When to use additive vs. dot-product attention:

The standard form in the literature writes the score as $w_v^{\top }\tanh (W_{q}q+W_{k}k)$. The professor writes the same using $v^{\top }\tanh (W_{Q}Q+W_{K}K)$ -- identical in structure, just with different notation for the weight vector.

15.6.2 Visual Intuition

Picture a two-branch network. The left branch takes the query and multiplies it by $W_Q$. The right branch takes the key and multiplies it by $W_K$. Both branches feed into a sum node -- the "additive" part. The sum passes through tanh (squashing values between -1 and 1), then through a final learned vector $w_v$ that compresses the result into a single number. That number is the score.

15.6.3 Assumptions, Scope, and Pitfalls

15.6.4 Recap and Bridge

Real-world connection: Bahdanau attention was the mechanism that first showed attention's power in machine translation (2014). It inspired the "Attention Is All You Need" paper three years later. While scaled dot-product attention now dominates, additive attention remains an important historical milestone and a useful alternative when dimensionality mismatch makes dot products impossible.

15.7 Introduction to Transformers

15.7.1 Why Transformers

The Transformer architecture was built on three key insights, each fixing a fundamental weakness of RNNs.

Transformers are the basis for all modern AI systems. GPT, ChatGPT, Copilot, Gemini, Claude, Llama, Mistral -- plus vision Transformers, audio Transformers, and video generators. Everything works on Transformer principles. The core component is always attention.

15.7.2 Three Variants of Transformer Architecture

Encoder-Only Models (e.g., BERT):

• Process the entire input bidirectionally -- every word sees every other word, left and right.
• Produce rich contextual representations for each token.
• Best for: classification (sentiment analysis, topic categorization), named entity recognition, question answering where the answer is a span within the text.
• No masking needed -- the model sees full context in both directions.
• Example: BERT (base: 12 encoder blocks, large: 24 encoder blocks).

Decoder-Only Models (e.g., GPT):

• Generate text autoregressively. Given a prompt, predict the next word, then the word after that, and so on.
• Use masked (causal) attention -- each position only sees itself and previous positions.
• Best for: text generation, code generation, conversational AI.
• Examples: GPT-3/GPT-4, Claude, Gemini, Llama, Mistral.

Encoder-Decoder Models (e.g., T5):

• Full architecture with both an encoder and a decoder stack.
• The encoder processes the input sequence. The decoder generates the output sequence, using cross-attention to pull information from the encoder.
• Best for: machine translation, text summarization, question answering with generated answers.
• Example: T5 (Text-to-Text Transfer Transformer).

15.7.3 Visual Intuition

Encoder-only: Draw a stack of arrowed boxes (encoder blocks). An input sentence enters from the left. Every box processes all words bidirectionally. An output vector emerges from the top. The diagram is simple and symmetrical -- just a vertical stack.

Decoder-only: Same vertical stack, but each box has a triangular mask drawn over the attention connections. Input words enter from the left, and output words are generated one at a time from the top. An arrow loops back from the output to serve as the next input -- autoregressive generation.

Encoder-decoder: Two stacks, side by side. The left stack (encoder) feeds into the right stack (decoder) through cross-attention connections -- shown as horizontal arrows between the two stacks at every layer.

15.7.4 Assumptions, Scope, and Pitfalls

15.7.5 Recap and Bridge

Real-world connection: By 2018, Transformers had become the default architecture for NLP. By 2021, vision Transformers (ViT) had matched CNNs on image classification. By 2023, Transformer-based models -- GPT-4, Claude, Gemini -- were driving the generative AI revolution. The architecture that started as a machine translation model now underpins text generation, image generation (via diffusion Transformers), code completion, protein folding prediction, and robotic control. The same core idea -- attention over tokens -- scales across modalities.

15.8 Transformer Encoder Architecture

15.8.1 Input Embedding and Positional Encoding

The embedding alone is not enough. Self-attention loses word order. To inject position information, you compute a positional encoding vector (sinusoidal or learned) of the same dimension and add it element-wise:

$$X^{\prime }=X_{\text{embed}}+PE$$

The result $X^{\prime }\in {\mathbb{R}}^{n\times d_{\text{model}}}$ carries both what the token is (semantic) and where it sits (positional). This combined vector enters the first encoder block.

15.8.2 Layer Normalization

Before entering the attention block, the input passes through layer normalization.

Layer normalization stabilizes learning by keeping values from growing too large or too small as data flows through deep stacks. It is applied at multiple points in each encoder block.

15.8.3 Multi-Headed Attention Block

The professor emphasized: "Each head learns a distinct pattern. One head may learn part-of-speech. Another learns subject-verb agreement. A third learns dependency parsing. One block of multi-headed attention is one pattern-extraction step."

15.8.4 Residual Connections

The residual connection requires the attention output and input to have the same dimension ($d_{\text{model}}$). The projection matrix $W_O$ ensures this by mapping the concatenated head outputs back to $d_{\text{model}}$.

15.8.5 Feed-Forward Network

The professor put it memorably: "Nowhere in the attention block do you see actual classification logic or non-linearity. You are just doing scoring, creating context vectors, and producing an outcome. The feed-forward network is where the intelligence for the task lives."

Why positionwise? The same two-layer MLP is applied independently to each position in the sequence. Every token position goes through the same weights $W_1,W_2$. This is why it is called "positionwise" -- the network is shared across positions.

15.8.6 Stacking Encoder Blocks

15.8.7 Dimension Tracking and Parameter Summary

The professor stated total parameters per block are about $12D^2$. This is reference information -- understand the architecture and how dimensions flow, but do not memorize the formula for the exam.

15.8.8 Visual Intuition

Draw one encoder block as a vertical pipeline:

1. Arrow enters from top: $X^{\prime }$ (N × D)
2. First sublayer: Split into two paths -- one goes through LayerNorm then Multi-Head Attention, the other skips straight ahead (residual). They merge at an "Add" node, then through another LayerNorm.
3. Second sublayer: Same pattern -- one path through LayerNorm then FFN, the other skips through. They merge at an "Add" node.
4. Arrow exits at bottom, same shape (N × D), feeding into the next block.

Stack several such blocks vertically. Data flows downward, preserving shape through every block. This shape-preserving property is what makes stacking possible -- you can add as many blocks as you want without the dimensions changing.

15.8.9 Assumptions, Scope, and Pitfalls

15.8.10 Recap and Bridge

Real-world connection: The encoder architecture described here is not just a theoretical construct -- it is the exact design used in BERT, one of the most influential NLP models ever created. When BERT processes text for Google Search, every word in every webpage is run through these same encoder blocks to produce contextual understanding. The architecture has proven so effective that even multimodal models (text + image) often use a Transformer encoder as their backbone.

15.9 BERT -- A Transformer Application Example

15.9.1 Question Answering with BERT

A feed-forward classifier sits on top of the encoder output. It takes the final representation of each token in the paragraph and predicts two things: (1) is this token the start of the answer? (2) is this token the end? The model outputs the span with the highest combined score.

The [CLS] token, placed at the very beginning of every BERT input, plays a special role. After all encoder blocks have processed the inputs bidirectionally, the output vector at the [CLS] position contains an aggregate of the entire sequence. This aggregate can feed into a classifier for tasks like sentiment analysis.

15.9.2 Sentence Classification with BERT

The professor summarized: "BERT uses multiple encoder blocks stacked together. Every token is processed with the help of all other tokens. The [CLS] token acts as a designated position where the aggregate context can be extracted for classification."

15.9.3 Visual Intuition

For question answering: Draw the input as a long ribbon of tokens: [CLS] + question words + [SEP] + context words. Below it, draw the stack of encoder blocks (the BERT core). From the top, two small classifier blocks descend to the context tokens -- one predicting start positions, one predicting end positions. Highlight two tokens in the context where both scores are highest.

For classification: Same ribbon of tokens but with just one sentence. From the [CLS] position at the top, an arrow leads to a classification head producing labels like "positive" (0.87) and "negative" (0.13).

15.9.4 Assumptions, Scope, and Pitfalls

15.9.5 Recap and Bridge

Real-world connection: BERT variants (RoBERTa, DistilBERT, DeBERTa) power search engines, content moderation systems, and customer support automation worldwide. Google integrated BERT into its search ranking in 2019, affecting roughly 10% of all English-language queries. In healthcare, BioBERT and ClinicalBERT process medical literature and electronic health records. The encoder-only design pattern persists because many real-world tasks -- classifying emails, extracting entities from contracts, detecting hate speech -- are understanding tasks that do not need text generation.

15.10 Exam Guidance Summary

Key exam preparation checklist:

1. Self-attention numerical problems. Practice scaled dot-product scoring with concrete numbers. Know how to compute query-key dot products, divide by $\sqrt{d_k}$, apply Softmax, and compute the weighted context vector. Also practice the same problem using Bahdanau's additive attention ($w_v^{\top }\tanh (W_{Q}Q+W_{K}K)$).
2. Positional encoding computation. Given $d_{\text{model}}$ and a set of positions, compute the sinusoidal positional encoding. Know the formula: $P{E}_{(pos,2i)}=\sin (pos/10000^{2i/d_{\text{model}}})$ and $P{E}_{(pos,2i+1)}=\cos (pos/10000^{2i/d_{\text{model}}})$.
3. Architecture identification. Given a task description (classification, translation, text generation), pick the right Transformer variant: encoder-only (BERT), decoder-only (GPT), or encoder-decoder (T5).
4. Dimension tracking. Be able to follow input dimensions through an encoder block: embeddings ($N\times D$), position encoding ($N\times D$), attention sublayer (preserves $N\times D$), FFN (preserves $N\times D$). Know that attention parameters total $4D^2$ and FFN parameters $8D^2$.
5. Component understanding. Know the role of each piece: positional encoding (adds order), multi-headed attention (captures multiple relationships in parallel), layer normalization (stabilizes training), residual connections (enables deep stacking), FFN (adds non-linearity).
6. Architecture error-finding. Given a diagram or description of a Transformer block with a deliberate error (missing residual, wrong order of operations, missing LayerNorm), identify and correct it.

Study resources: The PPT and reading material on the e-learning portal are sufficient. Go through all webinars and live session recordings. No external sources are needed.

15.11 Key Industry Applications

The Transformer architecture and its attention mechanisms are not just textbook concepts -- they power virtually every modern language AI system. Here are the major families:

Decoder-Only Models (text generation):

• GPT (Generative Pre-trained Transformer): The model family behind ChatGPT. GPT-3 (175B parameters, 96 decoder blocks) and GPT-4 power text generation, code completion, and conversational AI. These models use masked multi-headed self-attention and generate text autoregressively.
• Claude (Anthropic): A decoder-only Transformer focused on safe and helpful conversational AI.
• Gemini (Google): A multimodal decoder-only model that handles text, images, audio, and video.
• Llama (Meta) and Mistral (Mistral AI): Open-weight decoder-only Transformer models for research and deployment.

Encoder-Only Models (text understanding):

• BERT (Google): Encoder-only Transformer for NLP classification, question answering, and named entity recognition. BERT-base: 12 encoder blocks (110M parameters). BERT-large: 24 encoder blocks (340M parameters). Variants include RoBERTa, DistilBERT, and DeBERTa.

Encoder-Decoder Models (text transformation):

• T5 (Text-to-Text Transfer Transformer, Google): Full encoder-decoder Transformer that reframes every NLP task as text-to-text: input text in, output text out. Used for translation, summarization, and question answering.

Beyond Text:

• Vision Transformers (ViT): Apply the Transformer architecture to image data by splitting images into patches and treating them as tokens. Now competitive with CNNs for image classification, object detection, and segmentation.
• Audio Transformers: Process speech waveforms or spectrograms as token sequences for speech recognition and audio generation.
• Video Generators: Diffusion Transformers generate video frames autoregressively or via denoising diffusion in a Transformer backbone.
• Multimodal Models: Models like GPT-4V, Gemini, and Flamingo integrate text and vision by processing both modalities through a shared Transformer architecture.

Why Transformers dominate: The combination of parallel training, direct long-range attention, and scalable deep stacking means Transformers keep improving with more data and more compute. This is the architecture that launched the era of foundation models -- large pretrained models that can be adapted to hundreds of downstream tasks with minimal fine-tuning.