Contextual Word Embedding and Attention Mechanisms

11.1 Dependency Parsing — Edge Weight Learning

11.1.1 Recap: Graph-Based Dependency Parsing and Chu-Liu-Edmonds

The lecture opens with a quick recap of graph-based dependency parsing from the previous session. In graph-based parsing, words in a sentence are treated as nodes and the grammatical relations between them are edges. The goal is to find the best set of edges — the maximum spanning tree (MST) — that connects all words with the highest total edge weight, where every word has exactly one incoming edge (the single-head constraint).

The Chu-Liu-Edmonds algorithm finds this MST greedily. It starts by selecting, for each word, the incoming edge with the highest weight. If the result is a valid tree (no cycles), the algorithm terminates. If cycles exist — for example, if "John" selects "saw" as its head and "saw" selects "John" as its head — the algorithm contracts the cycle into a single mega-node (e.g., "John-saw" becomes WJS), merges the incoming edge weights (adding them), and repeats the greedy selection on the contracted graph. After the final MST is found on the contracted graph, the mega-nodes are unpacked by splitting the merged weights back to their original edges, yielding the final dependency tree.

A worked example from the previous session used the sentence "John saw Mary." The contraction step combined "John" and "saw" into WJS, computed incoming weights from "root" as $10+30=40$, from "Mary" as $11+20=31$, and so on. After greedy selection on the contracted graph, the mega-node was split back, producing the final dependency relations.

11.1.2 Computing Edge Weights Using Feature Vectors

The key question for this section: how are the numerical edge weights (10, 20, 30, etc.) obtained? These weights are computed from features of the word pair and a learned weight vector.

The approach uses an oracle, features, and learned weights. For parsing, the same idea applies: define features for each potential edge, assign initial random weights to those features, and learn better weights from training data.

Feature definition. For any two words $W_i$ (the potential head) and $W_j$ (the potential dependent), a set of binary (0 or 1) features is defined. These features capture properties like POS tags, relative position, and whether a word is root. The lecture uses seven features for the sentence "John saw Mary":

Feature values are Boolean (1 or 0) for simplicity, though in practice they could be real-valued (0.1, 0.2, etc.) and learned from validation data.

Initial weight assignment. The weight vector is randomly initialized:

11.1.3 Worked Example: Computing Feature Vectors and Edge Weights

11.1.4 Training the Weight Vector

The initial weights are random. The training procedure uses a structured perceptron update:

1. Compute feature vectors for all word pairs in the training sentence.

2. Compute edge weights using the dot product with the current weight vector.

3. Apply Chu-Liu-Edmonds to greedily select the maximum spanning tree — this is the predicted graph.

4. Compare the predicted graph with the gold-standard graph from the training data.

5. If the predicted graph matches the gold standard, no update is needed.

6. If they differ, update the weight vector:

Structured perceptron weight update:

$${\mathbf{w}}_{\text{new}}={\mathbf{w}}_{\text{old}}+{\mathbf{f}}_{\text{true}}-{\mathbf{f}}_{\text{predicted}}$$

This adds the feature vector of the correct tree and subtracts the feature vector of the predicted tree. Features that should have been active get their weights increased; features that were incorrectly active get their weights decreased. Over many training iterations, the weights converge to produce correct dependency trees.

Q: How do we programmatically compare the predicted and training data graphs?

A: We compare the feature weights of corresponding edges. If the edge weights match between predicted and gold-standard, no update is needed. The structured perceptron makes this comparison automatic — it operates on the full feature vectors of the trees, not on individual edges.

Q: How do we know which features to use? Are they predefined?

A: Features are given in the problem. They can be either engineered by the developer (in traditional ML) or learned from the data (in deep learning). In deep learning, the architecture learns these features automatically from training data, making the model more complex but domain-adaptive. The current example uses hand-crafted features for explainability.

Q: Are the feature sets predefined and applicable to all domains?

A: Yes, for traditional ML with hand-crafted features. Features like POS tags and word positions are generic across all English sentences, irrespective of domain. In deep learning, these features are learned from training data rather than manually specified.

11.1.5 Student Questions on Dependency Parsing

Exam note: There will not be a final weight-updating question in the exam because it is too time-consuming. However, expect questions on: (a) greedy Chu-Liu-Edmonds for cycle removal, (b) weight learning using deterministic parsing features, or (c) Arc-Eager parsing. Features and initial weights will be given in the problem; students need to compute the dot product to get edge weights.

Dependency parsing gives us the grammatical skeleton of a sentence — but it relies on hand-crafted features and domain-specific weight vectors. The next section asks a deeper question: what if we could learn word representations that capture context automatically, without any hand-engineered features? That motivation leads us to contextual word embeddings.

11.2 Contextual Word Embedding — Motivation and Background

11.2.1 Why Contextual Word Embedding Matters

Contextual word embedding is one of the most important topics not just in NLP but in the broader AIML domain. Every modern application — whether it involves agents, multi-agent systems, MCP, or any other advanced framework — is built on this core fundamental concept of the attention mechanism in transformers.

In NLP specifically, attention is primarily used for contextual word embedding. Language processing is always context-dependent. When someone speaks, the listener understands the meaning because the context of the conversation is maintained in their mind. The "Attention is All You Need" paper (Vaswani et al., 2017) became famous precisely because it captures the relationships among words and encodes that relationship information within the vector, enabling efficient context capture.

11.2.2 Recap: The Evolution of Word Representations

The pre-midsem portion of the course covered word embedding using frequency-based methods (TF-IDF) and dense vector representations (Skip-gram, CBOW, GloVe). The reason for vector representation is that models cannot process raw strings — they need numerical input. Vector representations allow finding semantic relationships even when words share no common characters (unlike Levenshtein distance or character-level matching).

The three types of word representation are:

1. Frequency-based (sparse): TF-IDF. The vector dimensionality equals the vocabulary size (~50,000). Vectors are high-dimensional and sparse.

2. Dense vector, static: Skip-gram, CBOW, GloVe. These produce lower-dimensional dense vectors (e.g., 300 dimensions — Andrew Ng's recommended size). However, each word has exactly one fixed vector regardless of context — "bank" has the same representation whether it means a river bank or a financial bank.

3. Dense vector, contextual: Transformers / BERT. These produce dense vectors that change depending on the surrounding words. "Bank" in "river bank" gets a different embedding than "bank" in "financial bank."

Today, all transformer-based models (GPT, BERT, etc.) use contextual word embedding internally. Different representation types are still used together for different purposes — for example, in retrieval-augmented generation (RAG), sparse representations (TF-IDF) and dense representations are both used for retrieving document chunks.

11.2.3 Static vs Contextual Embeddings — The Critical Difference

This is a key distinction that prompted a student question:

Q: In Skip-gram, we extracted context words and target words, and the exercise ensured similarity between the target word and its positive context words. How is this different from contextual embedding?

A: In Skip-gram, Skip-gram Negative Sampling (SGNS), and CBOW, although we use the concepts of target and context words during training, the final word embedding does not capture word-to-word relationships. The target-context mechanism is treated as a classification problem purely to learn the weights of the network. When we create the embedding vector for a word like "net," we are not calculating it with respect to "star" or "pimples" or other specific context words — we are only using the classification task to learn the network weights. The resulting embedding is static: it does not change from sentence to sentence.

The definitive test: the word "bank" has a single fixed dense vector in GloVe/Skip-gram. In contextual word embedding, "bank" in "I sat on the river bank" produces a different vector from "bank" in "I deposited money in the bank." Both are dense vectors — the difference is that contextual embeddings are dynamic (vary by sentence) while static embeddings are computed once from training data and never change.

Pitfall: Assuming that because Skip-gram uses "context words" during training, it produces contextual embeddings. It does not. The context window is only a training signal — a way to learn good static vectors. Once training is done, each word has exactly one vector, regardless of how it is used in any sentence. Contextual embeddings, by contrast, are computed on-the-fly for each sentence.

Real-world: GPT-3 uses 1024-dimensional embeddings. Today's standard is around 1024 features. In TF-IDF, the vocabulary might be 50,000 — both are dense representations in different senses, but contextual embeddings capture meaning-in-context.

The shift from static to contextual embeddings is the single most important conceptual leap in modern NLP. Every transformer-based model (BERT, GPT, T5) relies on contextual embeddings. Understanding why static embeddings are insufficient — and how contextual embeddings fix this — is the foundation for everything that follows in this lecture.

11.3 Feed-Forward Neural Language Model

11.3.1 Architecture and Limitations

A simple feed-forward network can perform language modeling — predicting the next token given the previous few tokens. The network takes a window of (typically three) previous word embeddings as input, concatenates them into a single vector, passes them through hidden layers with a non-linear activation function, and applies a softmax at the output layer to produce a probability distribution over the vocabulary for the next word.

This limitation is what motivates the next architecture: we need a network that can process sequences of arbitrary length while remembering what came before. That network is the RNN.

11.4 Recurrent Neural Networks

11.4.1 Why RNNs Are Needed

Real-world NLP problems are inherently sequential. POS tagging: given a sequence of words, produce a sequence of POS tags. Sentiment analysis: given a sequence of words (a review), produce a classification. Language translation: given a sentence in one language, produce a sentence in another. Next-word prediction: a sequential problem by definition.

A feed-forward network processes inputs independently — it cannot naturally pass information from one time step to the next. One might argue that positional word embeddings could encode order, but the weights in a feed-forward network are learned once and do not carry forward sequential context.

An RNN (Recurrent Neural Network) addresses this by maintaining a hidden state that is passed from one time step to the next. At each step, the network receives the current input and the hidden state from the previous step, producing a new hidden state and (optionally) an output. This recurrence allows the network to maintain a "memory" of all previous inputs in the sequence.

11.4.2 Types of RNNs

The students covered these variants in their deep neural networks course:

• Simple RNN: The basic recurrent architecture with no gating mechanism. Suffers from vanishing gradients over long sequences.
• Bidirectional RNN: Processes the sequence in both left-to-right and right-to-left directions, capturing context from both sides. The encoder states are the concatenation of the two hidden states: $h_j=(\overleftarrow{h_j},\overrightarrow{h_j})$.
• LSTM (Long Short-Term Memory): Introduces gates (input, forget, output) and a cell state to control information flow, addressing the vanishing gradient problem. The cell state acts as a "conveyor belt" that allows information to flow unimpeded across many time steps.
• GRU (Gated Recurrent Unit): A simplified version of LSTM with two gates (update and reset) and fewer parameters.

Any of these can serve as the building block inside an encoder or decoder. When the lecture says "RNN," it could mean any of these variants — simple RNN, LSTM, GRU, or bidirectional LSTM.

11.4.3 Sequence-to-Sequence Problems

The lecture emphasizes that most NLP applications are sequence-to-sequence problems:

A feed-forward network handles only single-word input and single-word output. An RNN handles sequences — it accepts a sequence of inputs and produces a sequence of outputs by processing one element at a time while maintaining the hidden state.

11.5 Encoder-Decoder Architecture

11.5.1 When and Why Two Networks Are Needed

When the input and output sequences have a one-to-one correspondence (same length, same type), a single RNN suffices. But for problems where the input and output sequences differ in length or structure — like machine translation (English "I eat mango" to Hindi, which may have a different word order and length) — we need to decouple the input processing from the output generation.

The encoder-decoder architecture uses two separate networks:

• Encoder: Reads the entire input sequence and compresses it into a fixed-size hidden representation (a context vector). This is a dense numerical summary of the input's meaning.
• Decoder: Takes the context vector and generates the output sequence one token at a time, autoregressively — each output token is fed as input to the next step.

The encoder and decoder can each be any type of neural network — RNN, LSTM, GRU, bidirectional LSTM, CNN, or even a transformer. For image captioning, the encoder might be a CNN (processing the image) and the decoder an RNN (generating words). For text-to-text tasks like translation, both are typically RNNs of the same type.

Real-world: GPT (Generative Pre-trained Transformer) is a decoder-only architecture. BERT is an encoder-only architecture. T5 and the original transformer use both encoder and decoder.

11.5.2 Encoder-Decoder Mechanism

The encoder processes the input sequence $X_1,X_2,\dots ,X_N$ through a series of hidden states, producing a final context vector that summarizes the entire input. The decoder receives this context vector and generates the output sequence $Y_1,Y_2,\dots ,Y_N$ one token at a time.

Each hidden state in the encoder receives input from:

1. The previous hidden state (recurrent connection)

2. The current input word embedding

Each hidden state in the decoder receives input from:

1. Its own previous hidden state

2. The previous output word (autoregressive feedback)

3. The context vector from the encoder

The decoder produces a probability distribution over the output vocabulary at each time step using a softmax function. The total loss is the sum of individual word losses:

Total loss (sum of negative log probabilities):

$$L=\sum _t-\log P(y_t\mid y_{<t},\mathbf{c})$$

where $\mathbf{c}$ is the context vector. Each term $-\log P(y_t\mid y_{<t},\mathbf{c})$ penalizes the model for assigning low probability to the correct output word at time step $t$. The sum across all time steps gives the total sentence-level loss. Training uses back-propagation through time (BPTT).

Worked example: Suppose we translate "I eat mango" to Hindi with three output words $y_1,y_2,y_3$. The decoder produces probabilities $P(y_1\mid \mathbf{c})=0.8$, $P(y_2\mid y_1,\mathbf{c})=0.7$, $P(y_3\mid y_{<3},\mathbf{c})=0.9$. The total loss is:

$$L=-\log (0.8)-\log (0.7)-\log (0.9)\approx 0.223+0.357+0.105=0.685$$

Lower loss means the model is more confident about the correct translations. The goal of training is to minimize this loss across all sentence pairs in the training data.

11.5.3 The Bottleneck Problem

The traditional encoder-decoder architecture has a critical limitation: the entire meaning of the input sentence — whether it is 5 words or 500 words — must be compressed into a single fixed-size context vector. This is the bottleneck.

For short sentences (10–15 words), this works reasonably well. For longer sentences, information is lost. The decoder cannot selectively focus on different parts of the input at different output steps — it receives the same compressed context vector for every output word.

Scope: The bottleneck is most severe for long sentences and for languages with very different word orders. For short, similar-structure sentence pairs, the simple encoder-decoder works adequately. The attention mechanism (Section 11.7) was invented specifically to solve this bottleneck.

Q: For encoder-only architecture, how do we get output in the desired format without a decoder?

A: Encoder-only architecture is specifically for producing word embeddings, not for generating text output. For machine translation, you need both encoder and decoder. For text generation tasks (like GPT), decoder-only architectures are used where word embeddings are pre-computed and fed directly to the decoder.

Q: For decoder-only models like GPT, isn't an encoder still needed internally for the input?

A: Internally, word embeddings are pre-computed during training. At inference time, only the decoder processes the token embeddings to generate output autoregressively. The encoder step happened during training — the tokens are already embedded when they reach the decoder.

The encoder-decoder architecture decouples input processing from output generation, enabling sequence-to-sequence tasks. But the single context vector creates a bottleneck for long sentences. The next section examines the vanishing gradient problem in RNNs, and then the attention mechanism will solve both problems simultaneously.

11.6 The Vanishing Gradient Problem

11.6.1 The Problem

When training RNNs using back-propagation through time (BPTT), gradients must flow backward through every time step. At each step, the gradient is multiplied by the recurrent weight matrix. If the weights are initialized to small values (e.g., between -0.5 and +0.5, which is standard practice), repeated multiplication causes the gradient to shrink exponentially.

11.6.2 Consequences

When gradients vanish, the weights in early layers receive essentially no update during training. The model infers that features from earlier time steps are unimportant for prediction — only the immediately preceding time step matters. This is problematic because in sentences like "The cats, who were playing in the garden all day long, are hungry," the model needs to remember "cats" (plural) from the beginning of the sentence to correctly predict "are" (plural) near the end.

Programmatically, it has been found that standard RNNs and LSTMs can effectively capture dependencies up to about 7 tokens. Beyond that distance, the vanishing gradient problem causes earlier information to be lost.

11.6.3 The Exploding Gradient Problem

The reverse can also happen: if weights are initialized to large values (e.g., 500), repeated multiplication causes the gradient to grow exponentially — the exploding gradient problem.

11.6.4 Why Attention Solves This

The attention mechanism addresses both the bottleneck problem and the vanishing gradient problem simultaneously:

1. Bottleneck: Instead of compressing the entire input into one vector, attention lets the decoder access all encoder hidden states directly.

2. Vanishing gradient: Attention creates shorter gradient paths — instead of information having to flow through every time step of the encoder, the decoder can attend directly to any input position, providing alternative gradient flow paths.

The vanishing gradient problem is the mathematical reason RNNs forget long-range dependencies. Attention solves this by creating direct connections between any two positions in the sequence, bypassing the need for information to flow through every intermediate step. This is the key insight that led to the transformer architecture.

11.7 The Attention Mechanism

11.7.1 Intuition

Attention allows the model to focus on the most relevant parts of the input when producing each output word. Instead of compressing the entire input into a single vector, the decoder "looks back" at all encoder hidden states and computes a weighted combination, where the weights indicate how relevant each input word is for the current output step.

The professor uses the example of translating "I love NLP" from English to a hypothetical Marathi sentence where the grammar is SOV (Subject-Object-Verb) instead of SVO. In this case, the alignment between input and output words is not sequential — the first output word might correspond to the first input word, but the second output word might correspond to the third input word, and so on. Attention captures these alignments naturally.

11.7.2 Self-Attention vs Cross-Attention

There are two types of attention:

Cross-attention: Compares tokens from the output (decoder) with tokens from the input (encoder). When the decoder wants to produce a word, it computes how relevant each input word is. The query comes from the decoder, and the keys come from the encoder.

Self-attention: Compares tokens within the same sequence. When processing the input "I love NLP," self-attention computes the relationship between "love" and "I," between "love" and "NLP," between "I" and "NLP," and so on. All comparisons happen within the input sequence itself.

For contextual word embedding, only self-attention is needed — we are computing the representation of each input word with respect to all other words in the same input. Cross-attention is used when there is both an encoder and a decoder (like in machine translation).

11.7.3 Computing Attention Scores

The attention mechanism computes three quantities for each word: a query, a key, and a value. Initially, these are all derived from the input word embeddings (static word embeddings from Skip-gram, CBOW, TF-IDF, or positional encoding).

The attention score between two words is computed as follows:

1. Dot product: Compute the dot product of the query vector of one word with the key vector of another word. This gives a scalar score indicating how relevant the two words are to each other.

2. Scaling: Divide by $\sqrt{d_k}$ (the square root of the key dimension) to prevent the dot products from becoming too large as the dimensionality increases.

3. Softmax normalization: Apply the softmax function across all scores for a given query, so they sum to 1. This converts raw similarity scores into probability-like attention weights:

$${\alpha }_{ij}=\frac{\exp (e_{ij})}{{\sum }_k\exp (e_{ik})}$$

4. Weighted sum: Multiply each value vector by its attention weight and sum them to produce the output — the contextual word embedding for that word:

$${\text{output}}_i=\sum _j{\alpha }_{ij}{v}_j$$

Why softmax is essential: Without softmax, the raw dot products yield unbounded values (e.g., 3, 4, 1) that don't indicate relative importance. Softmax converts them to probabilities that sum to 1 (e.g., 0.4, 0.5, 0.1), making importance directly comparable. The word with the highest alpha value contributes the most to the output.

Full attention formula (for reference):

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

This single formula encapsulates all four steps: dot product ($Q{K}^T$), scaling ($÷\sqrt{d_k}$), softmax normalization, and weighted sum ($\times V$).

11.7.4 The Context Vector

The final output of the attention computation for a given word is a weighted sum of all value vectors, where the weights are the attention scores (alpha values). This output vector is the contextual word embedding — it represents the word not in isolation, but in the context of all other words in the sentence.

For self-attention, the representation of word $X_2$ incorporates information from $X_1$, $X_3$, $X_4$, etc. This is bidirectional: $X_2$ is influenced by both words before and after it. This bidirectionality is a key property of BERT and other encoder-only transformer models.

The context vector captures how important each input word is with respect to every other word. This is the information that was lost in the traditional encoder-decoder bottleneck — now every word has direct access to every other word.

Pitfall: Confusing the "context vector" in attention with the "context vector" in the encoder-decoder architecture. In the encoder-decoder, the context vector is a single compressed representation of the entire input. In attention, the context vector is a per-word weighted sum that changes depending on which word is computing attention.

Attention replaces the single compressed context vector with per-word weighted sums, solving the bottleneck problem. The next section explains why we need separate Q, K, V vectors rather than using the raw word embeddings directly.

11.8 Key, Query, and Value Vectors

11.8.1 The Three Roles

The lecture spends significant time explaining the intuition behind the query, key, and value vectors through multiple analogies and a long student discussion.

Query (Q): The word "asking the question" — "How does every other word relate to me?" When computing attention for the word "NLP" in "I love NLP," the query comes from "NLP." It represents what the word is looking for.

Key (K): Each word's "label" or "announcement" to others — "Here is what I am." When comparing "NLP" (query) against all words, each word's key is compared against the query to compute a relevance score.

Value (V): The actual content or meaning of the word. Once the model determines which words are relevant (via the query-key match), it retrieves the value vectors of those words to construct the output.

11.8.2 Why Not Just Use the Raw Word Embeddings?

A student asks an important question: why can't we just multiply the input vectors directly? Why do we need separate Q, K, V vectors?

11.8.3 Matrices vs Vectors

The weight matrices $W_Q$, $W_K$, and $W_V$ are learned during training and are fixed for all input words within a single attention head. The matrices transform each input word embedding into its query, key, and value vectors:

$$Q_i=X_i\cdot W_Q,K_i=X_i\cdot W_K,V_i=X_i\cdot W_V$$

where $X_i$ is the input word embedding (static embedding from Skip-gram, CBOW, etc. combined with positional encoding).

All words share the same $W_Q$, $W_K$, $W_V$ matrices. What differs across words is the input vector $X_i$. The matrices are constant for a given attention head — they are learned parameters that do not change at inference time.

The input to the attention mechanism is the static word embedding. For "I love NLP," the initial inputs are the vector representations of "I" ($X_1$), "love" ($X_2$), and "NLP" ($X_3$) from a static embedding method. These are multiplied by the same $W_Q$, $W_K$, $W_V$ to produce $Q_1,Q_2,Q_3$; $K_1,K_2,K_3$; and $V_1,V_2,V_3$.

11.8.4 Self-Attention Computation Walkthrough

For the sentence "I love NLP" with self-attention:

1. Start with static word embeddings: $X_1$ ("I"), $X_2$ ("love"), $X_3$ ("NLP")

2. Compute queries: $Q_1=X_1{W}_Q$, $Q_2=X_2{W}_Q$, $Q_3=X_3{W}_Q$

3. Compute keys: $K_1=X_1{W}_K$, $K_2=X_2{W}_K$, $K_3=X_3{W}_K$

4. Compute values: $V_1=X_1{W}_V$, $V_2=X_2{W}_V$, $V_3=X_3{W}_V$

Scaled dot-product attention (the complete formula):

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

Breaking this down step by step for each word $i$:

Step 1 — Raw scores: For each word $i$, compute the dot product with every word $j$:

$$e_{ij}=\frac{Q_i\cdot K_j}{\sqrt{d_k}}$$

Step 2 — Normalize: Apply softmax to get attention weights that sum to 1:

$${\alpha }_{ij}=\text{softmax}(e_{i1},e_{i2},e_{i3})$$

Step 3 — Weighted sum: Multiply each value vector by its attention weight:

$${\text{output}}_i={\alpha }_{i1}{V}_1+{\alpha }_{i2}{V}_2+{\alpha }_{i3}{V}_3$$

This output is the contextual word embedding for word $i$. It encodes how important every other word in the sentence is for understanding word $i$.

For cross-attention (in encoder-decoder), the query comes from the decoder side ($Y$) and the keys and values come from the encoder side ($X$). The mathematics are identical — only the source of the query changes.

Pitfall: Thinking the WQ, WK, WV matrices are different for each word. They are the same for all words within a single attention head — only the input embeddings $X_i$ differ. The matrices are learned once during training and remain fixed at inference time.

The Q, K, V projections are not just a mathematical convenience — they are what allows attention to capture different notions of similarity in different subspaces. Without them, the model would be limited to the similarity structure of the original embedding space. The next section shows why a single attention head is not enough.

11.9 Multi-Head Attention

11.9.1 Why Multiple Attention Heads

A sentence can have multiple kinds of relationships between words simultaneously. Consider "The animal didn't cross the street because it was too tired":

• One attention head might link "it" to "animal" (pronoun resolution)
• Another might link "it" to "tired" (adjective-noun relationship)
• Another might capture subject-verb agreement

If there is only one set of Q, K, V, the model must average all these different relationship types into a single attention score. Multi-head attention solves this by running multiple attention computations in parallel, each with its own learned $W_Q$, $W_K$, $W_V$ matrices.

Real-world: Each head can specialize in different types of linguistic relationships — one might capture part-of-speech relationships, another might capture SVO grammar patterns, another might capture compound noun relationships, and so on. This specialization emerges naturally during training.

11.10 BERT — Contextual Word Embedding in Practice

11.10.1 BERT as Encoder-Only Architecture

BERT is an encoder-only transformer. It uses self-attention to compute contextual word embeddings — the representation of each input word incorporates information from all other words in the input, in both directions.

For contextual word embedding, no decoder is needed. The encoder alone produces the word embeddings that capture context. These embeddings are then used as input to any downstream NLP task.

The professor emphasizes that in any NLP application today — including code generation, conversational AI, and agent systems — the first step is always: convert input text to tokens, convert tokens to word embeddings (using contextual embedding), and then feed these embeddings to the model. You never give raw strings as input to a transformer or LLM.

Real-world: GPT-4, GPT-5, and other GPT models are decoder-only architectures. During training, tokens are already embedded into word embeddings. At inference time, the word embeddings are pre-computed and the decoder generates output tokens autoregressively. BERT is encoder-only and produces contextual embeddings used for downstream tasks like classification, NER, and question answering.

11.10.2 When to Use Encoder-Only, Decoder-Only, or Both

Machine translation requires both encoder and decoder — the encoder processes the source language and the decoder generates the target language. Word embedding tasks use encoder-only. Text generation tasks use decoder-only.

Exam Guidance Summary

Key Industry Applications

• GPT-4, GPT-5: Decoder-only transformer architectures using contextual word embedding internally. At inference time, word embeddings are pre-computed and the decoder generates output tokens autoregressively.
• BERT: Encoder-only architecture used for contextual word embedding, widely applied in classification, NER, QA, and search. Fine-tuned for specific tasks with minimal additional training.
• Retrieval-Augmented Generation (RAG): Uses both sparse (TF-IDF) and dense representations for retrieving document chunks, with contextual embedding in the transformer.
• Multi-agent systems, MCP, agentic AI: All built on the core concept of attention mechanism and contextual word embedding. Every modern AI agent uses transformer-based embeddings internally.
• Machine translation: The original application that motivated the attention mechanism ("Attention is All You Need"). Encoder-decoder architecture with cross-attention.
• Google Ngram: Referenced as an available resource for NLP research.
• Hugging Face: Mentioned as a platform for NLP tools and models, providing pre-trained BERT and other transformer models.
• StackQuest (YouTube): Referenced for a good visual explanation of key-query-value matrices.
• Andrew Ng: Referenced for the recommendation of 300 dimensions for word embeddings.