Natural Language Processing

Contextual Word Embedding and Word Sense Disambiguation

Published: 2026-07-27

Level: postgraduate

Audience: Postgraduate students in Natural Language Processing

Contextual Word Embedding and Word Sense Disambiguation

12.1 Attention Score Computation and Contextual Word Embeddings

Hook: The word "bank" appears in "I visited the bank to deposit money" and "the river bank was muddy." A static embedding like Word2Vec gives the same vector for both occurrences. How can a model tell these apart? The answer is a mechanism that lets every word look at every other word in the sentence and build a new representation shaped by context — the attention mechanism.

A contextual word embedding is a vector representation of each word that changes depending on the surrounding words. Static embeddings — Word2Vec (SGNS), GloVe, TF-IDF — assign one fixed vector per word regardless of context. Contextual embeddings fix this by computing a new vector for every occurrence of every word, shaped by the words around it. The transformer architecture, introduced in "Attention Is All You Need" (Vaswani et al., 2017), is the engine that makes this possible.

Intuition — the librarian analogy: Imagine you walk into a library and ask the librarian, "Where are the finance books?" The librarian queries every book's label (its key) to find matches, then hands you the relevant content (the value). The query is what you are looking for; the key is what each item declares about itself; the value is the information you receive if that item is selected. Attention does the same thing for every word in a sentence: each word asks a question (query), every other word advertises its content (key), and the answer is a weighted mix of the values.

12.1.1 Recap: Why Attention Is Needed

Earlier encoder-decoder models built on LSTMs and GRUs suffered from long-term context dependency problems. When an input sequence grew long, the fixed-size hidden state of the recurrent network could not retain information from early tokens — the same information had to squeeze through a bottleneck at every time step. The attention mechanism solves this by letting every output position directly attend to every input position, bypassing the sequential bottleneck. The transformer architecture replaced recurrence entirely with attention, enabling parallel computation across all tokens simultaneously.

A concrete way to see the problem: in a 100-word sentence translated by an LSTM encoder-decoder, the 100th encoder hidden state must carry the meaning of all 100 words. Information from word 1 has to survive 99 sequential updates. With attention, the decoder can look directly at word 1's representation at every decoding step — no sequential compression needed.

12.1.2 Key, Query, and Value Vectors

Every input token produces three vectors when multiplied by learned weight matrices:

Query vector $q_{i} = x_{i} W_{Q}$ — "what am I looking for?"
Key vector $k_{i} = x_{i} W_{K}$ — "what do I contain?"
Value vector $v_{i} = x_{i} W_{V}$ — "what information do I provide if selected?"

Shapes and shared matrices. Let $x_{i} \in R^{1 \times d}$ be the input word embedding for token $i$ . The weight matrices are:

W_{Q} \in R^{d \times d_{k}}, W_{K} \in R^{d \times d_{k}}, W_{V} \in R^{d \times d_{v}}

where $d$ is the input embedding dimension, $d_{k}$ is the dimension of query and key vectors, and $d_{v}$ is the dimension of value vectors. In the standard transformer, $d_{k} = d_{v} = d / H$ where $H$ is the number of attention heads (covered in Section 12.2). In the toy example below, $d = 4$ and $d_{k} = d_{v} = 3$ .

These matrices are learned during training via backpropagation and gradient descent — they are additional parameters of the model, just like the weights in a feedforward layer. Crucially, the same three matrices $W_{Q}$ , $W_{K}$ , and $W_{V}$ are applied to every input word embedding. They do not change from token to token. The query, key, and value vectors vary per token because the input $x_{i}$ varies, but the matrices are shared. In a real model, the entries inside these matrices are small decimal numbers (e.g., 0.023, −0.017), not the clean integers used in the toy example.

Why three separate matrices? You might ask: why not just use the raw embedding for everything? The professor drew an analogy to SVM kernel functions: just as a kernel transforms low-dimensional data into a higher-dimensional space to make separation cleaner, the weight matrices $W_{Q}$ and $W_{K}$ transform the raw embeddings into a space where dot-product similarity is a more meaningful measure of relatedness than it would be on the raw vectors. Without these learned projections, the dot product between raw embeddings might not capture the right notion of "relevance." The three separate matrices let the model learn three different projections: one for asking questions, one for advertising content, and one for providing information.

12.1.3 Mathematical Formulation: Attention Scores

Step 1 — Raw attention score (scaled dot product). The attention score between token $i$ (as query) and token $j$ (as key) is:

α_{ij} = \frac{q _{i} \cdot k _{j}}{d _{k}}

where $d_{k}$ is the dimensionality of the key (and query) vectors. The dot product $q_{i} \cdot k_{j} = \sum_{l = 1}^{d_{k}} q_{i, l} k_{j, l}$ measures how well the query of token $i$ matches the key of token $j$ . A large positive value means "these two tokens are highly relevant to each other."

Why the scaling factor $d_{k}$ ? As $d_{k}$ grows, the dot product grows proportionally to $d_{k}$ (it is a sum of $d_{k}$ terms). Large dot products push the softmax into saturation regions where the gradient is nearly zero, making learning unstable. Dividing by $d_{k}$ keeps the variance of the dot product at roughly 1 regardless of $d_{k}$ , so the softmax operates in a region with usable gradients. This is a variance-stabilising trick, not a conceptual change.

Step 2 — Softmax normalisation. The softmax converts raw scores into a probability distribution over all context tokens:

softmax (α_{ij}) = \frac{e ^{α_{ij}}}{\sum _{m = 1}^{n} e ^{α_{im}}}

where $n$ is the number of tokens in the sequence. The softmax has two key properties: (a) every output is in $(0, 1)$ , and (b) the outputs for a given query token sum to 1. This means the attention weights form a valid probability distribution — token $i$ "spends" its total attention budget of 1.0 across all other tokens.

Step 3 — Contextual embedding (weighted sum of values). The final contextual embedding for token $i$ is:

z_{i} = j = 1 \sum n softmax (α_{ij}) \cdot v_{j}

This $z_{i} \in R^{1 \times d_{v}}$ is the contextual word embedding for token $i$ . It is a weighted average of all value vectors, where the weights come from how relevant each other token is to token $i$ . Tokens that are highly relevant (high attention weight) contribute more to the final representation; tokens that are irrelevant contribute almost nothing.

Scope: The formulas above describe single-head scaled dot-product attention. In a full transformer, this computation runs in parallel across multiple heads (Section 12.2), each with its own $W_{Q}, W_{K}, W_{V}$ matrices. The formulas also assume no masking — every token can attend to every other token. Masked attention (Section 12.4) restricts which tokens are visible.

12.1.4 Worked Example: Attention Score for Three Words

The professor walked through a complete numerical example. Consider three words — "bank," "saving," "account" — with the following (toy) input vectors of dimension $d = 4$ :

x_{1} = [1, 0, 0, 1], x_{2} = [0, 1, 0, 2], x_{3} = [0, 0, 1, 0]

These are the initial embeddings from a static method like SGNS or TF-IDF. In a real system these would be decimal values (e.g., 0.12, −0.03), but the professor used integers for a clean toy demonstration.

Step 1: Compute key, query, and value vectors.

The weight matrices are of size $4 \times 3$ (4 input features, 3 output dimensions). For the key matrix $W_{K}$ , the professor used:

W_{K} = 101101101100

Multiplying $x_{1} = [1, 0, 0, 1]$ by $W_{K}$ (row vector times matrix):

k_{1} = x_{1} W_{K} = [1 \cdot 1 + 0 \cdot 0 + 0 \cdot 1 + 1 \cdot 1, 1 \cdot 0 + 0 \cdot 1 + 0 \cdot 1 + 1 \cdot 0, 1 \cdot 1 + 0 \cdot 1 + 0 \cdot 0 + 1 \cdot 0] = [2, 0, 1]

For $x_{2} = [0, 1, 0, 2]$ :

k_{2} = x_{2} W_{K} = [0 \cdot 1 + 1 \cdot 0 + 0 \cdot 1 + 2 \cdot 1, 0 \cdot 0 + 1 \cdot 1 + 0 \cdot 1 + 2 \cdot 0, 0 \cdot 1 + 1 \cdot 1 + 0 \cdot 0 + 2 \cdot 0] = [2, 1, 1]

For $x_{3} = [0, 0, 1, 0]$ :

k_{3} = x_{3} W_{K} = [0 \cdot 1 + 0 \cdot 0 + 1 \cdot 1 + 0 \cdot 1, 0 \cdot 0 + 0 \cdot 1 + 1 \cdot 1 + 0 \cdot 0, 0 \cdot 1 + 0 \cdot 1 + 1 \cdot 0 + 0 \cdot 0] = [1, 1, 0]

The query matrix $W_{Q}$ and value matrix $W_{V}$ are applied identically to produce $q_{1}, q_{2}, q_{3}$ and $v_{1}, v_{2}, v_{3}$ respectively. The professor gave the query vector for "bank" as $q_{1} = [1, 0, 2]$ .

Step 2: Compute attention scores (dot products).

To find the contextual embedding for the first word ("bank"), we treat its query vector as the query: $q_{1} = [1, 0, 2]$ . We compute dot products with each key vector:

α_{11} = q_{1} \cdot k_{1} = 1 \cdot 2 + 0 \cdot 0 + 2 \cdot 1 = 2 + 0 + 2 = 4

α_{12} = q_{1} \cdot k_{2} = 1 \cdot 2 + 0 \cdot 1 + 2 \cdot 1 = 2 + 0 + 2 = 4

α_{13} = q_{1} \cdot k_{3} = 1 \cdot 1 + 0 \cdot 1 + 2 \cdot 0 = 1 + 0 + 0 = 1

The raw attention scores are $4, 4, 1$ . These measure how much each word's key matches "bank"'s query. "Bank" attends equally strongly to "saving" and "account" (both score 4), and weakly to itself (score 1). The professor rounded these for pedagogical clarity; the exact values depend on the toy matrices used.

Step 3: Apply softmax.

softmax (α_{11}) = \frac{e ^{4}}{e ^{4} + e ^{4} + e ^{1}} = \frac{54.598}{54.598 + 54.598 + 2.718} = \frac{54.598}{111.914} \approx 0.488

softmax (α_{12}) = \frac{e ^{4}}{111.914} \approx 0.488

softmax (α_{13}) = \frac{e ^{1}}{111.914} = \frac{2.718}{111.914} \approx 0.024

Verification: $0.488 + 0.488 + 0.024 = 1.000$ ✓

The professor approximated the weights as $0, 0.5, 0.5$ for simplicity. The exact values show that "saving" and "account" each receive about 49% of the attention, while "bank" itself receives only about 2%. The $e^{4}$ terms dominate the denominator, so the $e^{1}$ term gets a near-zero weight — this is what the professor meant when he said "it must be close to 0."

Step 4: Compute weighted sum of value vectors.

The professor's value vectors were $v_{1}, v_{2}, v_{3}$ . Using the approximate softmax weights (0, 0.5, 0.5):

z_{1} = 0 \cdot v_{1} + 0.5 \cdot v_{2} + 0.5 \cdot v_{3}

The professor showed the resulting vector as $[2, 7, 1, 5]$ . This is the contextual word embedding for the word "bank" — it now encodes information from "saving" and "account" weighted by their relevance.

Sense-check: The output vector is dominated by the value vectors of "saving" and "account," which is exactly what we expect. In the finance context, "bank" should be represented by its financial associates, not by its own static embedding.

Step 5: Repeat for other words.

For the second word ("saving"), the process repeats with $q_{2}$ as the query. New dot products are computed with all three key vectors, new softmax values are obtained, and a different contextual embedding $z_{2}$ results. The same applies for the third word. Each word gets its own unique contextual embedding shaped by its relationships with all other words in the sequence.

12.1.5 Parallelism Advantage

Q: Can attention be computed in parallel, matrix to matrix, instead of vector to vector?

A: Yes. The professor confirmed enthusiastically: "Yes, yes, yes. It is actually done in a parallel fashion. That's the best advantage of this." All query-key dot products become a single matrix multiplication $Q K^{⊤}$ , softmax is applied row-wise, and the result is multiplied by $V$ . This is the core advantage of transformers over RNNs.

Matrix form. In practice, all query-key dot products are computed as a single matrix multiplication. Stack all query vectors into matrix $Q \in R^{n \times d_{k}}$ and all key vectors into matrix $K \in R^{n \times d_{k}}$ . Then:

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

Here $Q K^{⊤} \in R^{n \times n}$ computes all pairwise dot products in one operation, the softmax is applied row-wise, and the result is multiplied by $V \in R^{n \times d_{v}}$ to produce all contextual embeddings simultaneously. This is the core computational advantage of transformers over sequential RNNs: an RNN must process tokens one by one (sequential complexity $O (n)$ per token), while attention processes all tokens in parallel (complexity $O (n^{2} d_{k})$ for the dot products, but fully parallelisable on GPUs).

12.1.6 Encoder-Only Sufficiency for Contextual Embeddings

The professor made an important architectural point: "For an encoder, you are only going to give these words and you are going to get these hidden representations of each of these words. So encoder only architecture is sufficient for your contextual word embedding."

Encoder-only models like BERT and Hugging Face sentence transformers generate contextual word embeddings without any decoder. The decoder is needed only for sequence generation tasks (e.g., machine translation, text generation). This is an important distinction: if your goal is to get a rich vector for each word (for classification, similarity, NER, etc.), you only need the encoder side of the transformer.

12.1.7 Student Questions and Answers

Q: Using dependency graphs like MST, we also computed dependency between words. What is the difference from what we are doing here?

A: In dependency parsing, we find grammatical relations among words — subject-verb, modifier-noun, etc. — but we do not come up with a separate individual vector word embedding for each word. Here in contextual word embedding, we generate a vector representation for each word that captures its meaning in context. These vectors can then be used as input to any downstream deep learning algorithm. Dependency parsing is actually used internally as one of the attention heads in some transformer configurations, so the two ideas are complementary, not competing.

Q: How are the $W_{Q}$ , $W_{K}$ , $W_{V}$ weight matrices learned?

A: They are learned during training using backpropagation and gradient descent, just like all other weights in a neural network. They are additional parameters added to the model. The values inside these matrices will not be small integers like in the toy example — they will be decimal numbers like 0.1, 0.2, learned from data.

Q: In the third step, how does the softmax of $e^{2}$ become zero?

A: The professor clarified that $e^{2} / (e^{2} + e^{4} + e^{4})$ is not exactly zero — it is approximately 0.063, which rounds to 0 for the toy example. The three softmax values must sum to 1, and since $e^{4}$ terms dominate, the $e^{2}$ term gets a near-zero weight. This is a general property of softmax: when one input is much larger than the others, it captures nearly all the probability mass.

Q: The values in the matrices — these will not be such high values in real systems, right?

A: Correct. These are toy values for illustration. In real systems, the weight matrices contain small decimal values, and the input vectors are also decimal-valued embeddings from SGNS or similar algorithms.

Common Pitfalls:

Confusing matrices with vectors. $W_{Q}, W_{K}, W_{V}$ are the same for every token. Only the input $x_{i}$ changes, so the output vectors $q_{i}, k_{i}, v_{i}$ vary per token. Do not think each token has its own weight matrix.

Forgetting the scaling factor. Without $d_{k}$ , the softmax saturates for large $d_{k}$ , and gradients vanish. This is a design choice in the original transformer, not a mathematical necessity — but omitting it causes training instability.

Thinking attention replaces embeddings. Attention produces contextual embeddings from static ones. The static embedding (from SGNS, GloVe, etc.) is the input; the contextual embedding is the output. You still need a good static embedding as the starting point.

Assuming attention scores are probabilities. The raw scores $α_{ij}$ are not probabilities — they can be any real number. Only after the softmax step do they become a valid probability distribution that sums to 1.

Recap: The attention mechanism computes contextual embeddings by (1) projecting each token into query, key, and value vectors via learned matrices, (2) computing scaled dot-product scores between every query-key pair, (3) normalising scores with softmax to get attention weights, and (4) taking a weighted sum of value vectors. The entire operation can be expressed as one matrix formula: $Attention (Q, K, V) = softmax (Q K^{⊤} / d_{k}) V$ . This is the foundation on which multi-headed attention (Section 12.2) and the full transformer architecture (Section 12.4) are built.

12.1.8 Exam Notes

Exam note: The professor confirmed that numerical problems on attention score computation can be expected. "Yes, you can. It's a simple one." The exam is open-book, so students can bring printed slides and notes. Be prepared to: (a) compute $k_{i}, q_{i}, v_{i}$ from given input vectors and weight matrices, (b) compute dot-product scores, (c) apply softmax, and (d) compute the final weighted sum. The worked example in Section 12.1.4 is the template for such questions.

12.1.9 Real-World & Domain Connection

Transformers and contextual embeddings are foundational to all modern NLP: GPT, BERT, T5, Hugging Face sentence transformers, and every large language model in production today use attention as their core mechanism. In industry, contextual embeddings power semantic search (finding documents by meaning, not just keyword match), question-answering systems, chatbots, and machine translation. The ability to compute these embeddings in parallel on GPUs is what makes large-scale NLP practical — but it also means that GPU costs (often several lakh INR for training-grade hardware) are a significant barrier for smaller organisations, making cost-effective variants like DistilBERT important for resource-constrained environments.

12.2 Multi-Headed Attention

Hook: A single attention head can only learn one pattern of relationships at a time. But language is rich: "bank" relates to "deposit" by topic, to "guarantees" by syntax, and to "it" by coreference — all simultaneously. How can the model capture all these relationships in one pass? The answer is to run multiple attention computations in parallel, each specialising in a different type of relationship.

Multi-headed attention runs several attention computations in parallel, each with its own set of $W_{Q}$ , $W_{K}$ , $W_{V}$ matrices, to capture different types of relationships among words simultaneously.

12.2.1 Why Multiple Heads?

A single attention head can only learn one pattern of relationships. But language has many simultaneous relationship types: syntactic dependencies (subject-verb agreement), semantic similarity (synonyms, co-hyponyms), positional patterns (adjacent words), coreference, and more. Multi-headed attention addresses this by giving the model multiple independent "views" of the same input.

Intuition — the committee analogy: Think of multi-headed attention as a committee of specialists reviewing the same document. One specialist highlights named entities, another tracks grammatical structure, and a third identifies sentiment-bearing phrases. Each specialist has their own "lens" (their own $W_{Q}, W_{K}, W_{V}$ matrices), but they all read the same document. The final report is a combination of all their findings.

A student described it accurately: "It learns the varying patterns of the data. One single will learn [only one pattern]." The professor confirmed: "Absolutely right. Very good."

The professor illustrated with a concrete example. Given the sentence "I went to CS 224N class and learned":

Head 1 might attend to named entities: "Stanford University," "CS course" — capturing what the sentence is about.
Head 2 might capture grammatical structure: subject, verb, object — dependency-parsing-like relations.
Head 3 might track part-of-speech patterns: which adjectives modify which nouns.

The "Attention Is All You Need" paper used 8 attention heads. Modern models like GPT-4 use many more (96 or higher), with each head operating on a smaller subspace.

Mathematical formulation. For $H$ attention heads, each head $h$ has its own weight matrices:

W_{Q}^{(h)} \in R^{d \times d_{k}}, W_{K}^{(h)} \in R^{d \times d_{k}}, W_{V}^{(h)} \in R^{d \times d_{v}}

where $d_{k} = d_{v} = d / H$ . Each head computes attention independently:

head_{h} = Attention (X W_{Q}^{(h)}, X W_{K}^{(h)}, X W_{V}^{(h)})

The outputs of all heads are concatenated and projected through a final linear layer:

MultiHead (X) = Concat (head_{1}, \dots, head_{H}) W_{O}

where $W_{O} \in R^{d \times d}$ is an output projection matrix. The concatenation restores the original dimension $d$ , so the multi-head output has the same shape as the input.

12.2.2 Dimensionality Splitting

If the input embedding has $d$ features (dimensions) and there are $H$ attention heads, each head operates on a subspace of dimension $d / H$ . This keeps the total computational cost the same as a single full-dimensional attention.

Example: With $d = 6$ features and $H = 2$ heads:

Each head gets dimension $6/2 = 3$ .
Head 1: $W_{Q}^{(1)}, W_{K}^{(1)}, W_{V}^{(1)}$ are each of size $6 \times 3$ .
Head 2: $W_{Q}^{(2)}, W_{K}^{(2)}, W_{V}^{(2)}$ are each of size $6 \times 3$ .

Each input vector $x \in R^{1 \times 6}$ is multiplied by the $6 \times 3$ matrix to produce a $1 \times 3$ output per head. The two $1 \times 3$ outputs are concatenated back to $1 \times 6$ .

With real model sizes: If the embedding has 1024 features (as in GPT-3.5/4) and there are 8 heads, each head operates on $1024/8 = 128$ features. Each head's weight matrices are $1024 \times 128$ .

A student asked: "If there are 10 words and we use 2 heads, will 5 tokens be given to one head and 5 to the other?" The professor corrected this immediately: "Don't equate the words to the tokens. It is features that I'm talking about."

Common Pitfall — splitting tokens vs. features: All words go through all heads. It is the feature dimensions that are split, not the tokens. If you have 10 words and 2 heads, each head processes all 10 words — but each head sees only half the features (dimensions). Think of it as two photographers taking pictures of the same scene with different lenses: both photograph the entire scene, but each lens captures different details.

12.2.3 Multi-Query Attention and Group Attention

A student mentioned optimised variants: "There can be multi-head attention, there can be multi-query attention, there can be group attention." These reduce memory overhead by sharing key-value matrices across heads.

Multi-query attention (MQA): All heads share the same $W_{K}$ and $W_{V}$ but have different $W_{Q}$ . This means there is only one set of key-value projections, reducing the KV cache size by a factor of $H$ . This is a major optimisation for autoregressive generation, where the KV cache must be stored for every previously generated token.

Grouped-query attention (GQA): A middle ground — heads are divided into groups, and heads within each group share $W_{K}$ and $W_{V}$ . This balances the expressiveness of full multi-head attention with the memory savings of MQA.

These variants are used in production LLMs (e.g., LLaMA, PaLM) to reduce memory bandwidth requirements during inference.

12.2.4 Self-Attention vs Cross-Attention

Self-attention (used in the encoder): the query, key, and value all come from the same input sequence. Each word attends to every other word in the same sentence. This is what produces contextual word embeddings.

Cross-attention (used in the decoder): the query comes from the decoder side, while the keys and values come from the encoder output. This lets the decoder "look at" the input while generating output — essential for tasks like machine translation where the decoder needs to reference the source sentence.

The professor emphasised: "Cross attention is not required for contextual word embedding. Contextual word embedding, you need only the self attention or the encoder side. Only when you have the decoder, you need this concept of cross attention."

Scope: If your goal is to produce contextual word embeddings (for classification, NER, similarity, etc.), you only need self-attention (encoder side). Cross-attention is relevant only for sequence-to-sequence models (encoder-decoder architectures like T5, BART) where the decoder generates output tokens.

12.2.5 Student Questions and Answers

Q: If we have 10 words and use 2 heads, will 5 tokens be given to one head and 5 to the other?

A: No. All words go through all heads. It is the feature dimensions that are split, not the tokens. Each head operates on $d / H$ features, not on a subset of words. If the embedding has 1024 features (as in GPT-3.5/4), each head might operate on 128 features with 8 heads.

Recap: Multi-headed attention runs $H$ parallel attention computations, each with its own learned projection matrices operating on a $d / H$ -dimensional subspace. The outputs are concatenated and projected back to dimension $d$ . This lets the model capture multiple relationship types (syntactic, semantic, positional) simultaneously. Self-attention (encoder) is sufficient for contextual word embeddings; cross-attention (decoder) is needed only for generation tasks.

12.3 Positional Encoding

Hook: Attention treats the input as a set of vectors — it has no notion of "first," "second," or "last." But word order is critical: "dog bites man" and "man bites dog" have the same words but opposite meanings. Positional encoding injects order information into the embeddings so the model can distinguish between them.

Attention is permutation-invariant: if you shuffle the input tokens, the output vectors will be the same (just shuffled). This is because the dot-product scores and weighted sums do not depend on position — they depend only on the content of the vectors. Positional encoding solves this by adding a position-dependent signal to each token's embedding before the attention computation.

12.3.1 Sinusoidal Positional Encoding

The original transformer (Vaswani et al., 2017) uses sine and cosine curves at different frequencies to generate position vectors. For a word at position $p os$ (0-indexed) and feature dimension $i$ :

P E_{(p os, 2 i)} = sin (\frac{p os}{1000 0 ^{2 i / d}})

P E_{(p os, 2 i + 1)} = cos (\frac{p os}{1000 0 ^{2 i / d}})

where $d$ is the embedding dimension. Even-indexed dimensions ( $2 i$ ) use sine; odd-indexed dimensions ( $2 i + 1$ ) use cosine. The number of sinusoidal curves equals the number of feature dimensions — if the vector has 4 dimensions, 4 curves are used (alternating sine and cosine).

What the formula means: Each dimension of the positional encoding is a sine or cosine wave with a different frequency. Low dimensions (small $i$ ) have high frequency (rapid oscillation); high dimensions (large $i$ ) have low frequency (slow oscillation). The base frequency is 1, and each successive pair of dimensions divides the frequency by 10000. This creates a hierarchy of "clock speeds" — some dimensions change rapidly between adjacent positions, others change slowly across the entire sequence.

The professor's intuition: "If you look at any sine or cosine curve, it's a time dimensional curve. Based on the time on the x-axis, you have values of cosine and sinusoidal curves. Because of this time factor, we are considering it as a position." A word appearing first gets the value at timestamp 1; a word appearing second gets the value at timestamp 2.

Intuition — the clock hands analogy: Imagine a clock with many hands, each spinning at a different speed. The fastest hand completes a full rotation every few ticks; the slowest hand barely moves. At any given moment, the combination of all hand positions uniquely identifies the time. Positional encoding works the same way: each dimension is a "hand" spinning at a different frequency, and the combination of all dimension values uniquely identifies the position.

Example: For $d = 4$ and position $p os = 1$ :

P E_{(1, 0)} = sin (\frac{1}{1000 0 ^{0/4}}) = sin (1) \approx 0.841

P E_{(1, 1)} = cos (\frac{1}{1000 0 ^{0/4}}) = cos (1) \approx 0.540

P E_{(1, 2)} = sin (\frac{1}{1000 0 ^{2/4}}) = sin (\frac{1}{100}) \approx 0.010

P E_{(1, 3)} = cos (\frac{1}{1000 0 ^{2/4}}) = cos (\frac{1}{100}) \approx 1.000

So the positional encoding for position 1 is approximately $[0.841, 0.540, 0.010, 1.000]$ . For position 2, the values shift — the fast dimensions (0, 1) change noticeably, while the slow dimensions (2, 3) barely change. This creates a unique "fingerprint" for each position.

12.3.2 Why Multiple Curves Prevent Repetition

A student raised a sharp concern: "If it is sine and cosine, it goes periodically. If we have long sentences, it can repeat the value." The professor acknowledged this as valid but explained the mitigation: "Everything will not be repeated because you are taking multiple curves. The starting point of these curves can be different. So you can get multiple [unique values]."

Scope and practical limits: With 1024 or more features (as in real models like GPT or BERT), the combination of many different-frequency sinusoids produces effectively unique position vectors for practical sequence lengths (up to a few thousand tokens). However, for extremely long sequences (millions of tokens), the high-frequency components do repeat. This is one reason why modern models often use learned positional embeddings (trainable vectors for each position) or relative positional encodings (which encode the distance between tokens rather than absolute position) instead of sinusoidal encoding. The sinusoidal approach was chosen for the original transformer because it generalises to sequence lengths not seen during training — a learned embedding for position 5001 does not exist if the model was trained on sequences up to length 5000, but the sinusoidal formula can compute it.

12.3.3 Combining Positional Encoding with Word Embeddings

The positional encoding vector is added to the original word embedding (not concatenated):

h_{i} = x_{i} + P E_{i}

where $x_{i}$ is the static word embedding and $P E_{i}$ is the positional encoding vector for position $i$ . The result $h_{i}$ encodes both the word's meaning and its position in the sequence. This combined representation is what enters the attention mechanism.

Why add and not concatenate? Concatenation would increase the dimensionality (from $d$ to $2 d$ ), doubling the computational cost of all subsequent layers. Addition keeps the dimension at $d$ and is empirically just as effective. The model can learn to separate the positional signal from the semantic signal because they occupy different "subspaces" of the embedding — the positional encoding has a fixed, structured pattern (sinusoidal), while the semantic embedding is learned and data-driven.

Common Pitfall — positional encoding is not a replacement for embeddings: The positional encoding is an additive correction to the word embedding, not a substitute. Without the word embedding, the positional encoding tells you where a word is but not what it is. Both signals are necessary.

12.3.4 Student Questions and Answers

Q: Since sine and cosine are periodic, won't the positional values repeat for long sentences?

A: Valid concern, but multiple curves at different frequencies prevent repetition for practical sequence lengths. With 1024 features, the combination of many different-frequency sinusoids produces effectively unique position vectors for sequences up to several thousand tokens. For very long sequences, alternative methods like relative positional encoding are preferred.

Recap: Positional encoding injects word-order information into the transformer by adding a sinusoidal signal to each token's embedding. Each dimension uses a sine or cosine wave at a different frequency, creating a unique "fingerprint" for each position. The signal is added (not concatenated) to the word embedding before attention. This is necessary because attention is permutation-invariant — without it, "dog bites man" and "man bites dog" would look identical.

12.4 Masked Self-Attention, Residual Connections, and Normalisation

Hook: A transformer with 16–32 stacked attention layers is very deep. Without special techniques, gradients vanish during backpropagation — the same problem that killed early RNNs. The transformer solves this with two elegant tricks: residual connections (skip connections) and layer normalisation. Together, they let information and gradients flow cleanly through dozens of layers.

12.4.1 Masked Self-Attention (Decoder Side)

On the decoder side, the model generates output one word at a time, left to right. At position $t$ , it cannot see future tokens at positions $t + 1, t + 2, \dots$ — those words have not been generated yet. Masked self-attention enforces this by setting the attention scores for all future positions to $- \infty$ before the softmax.

How masking works. Given the raw attention score matrix $A = Q K^{⊤} / d_{k}$ , the masked version replaces entries where $j > i$ (future positions) with $- \infty$ :

A_{ij}^{'} = {A_{ij} - \infty if j \leq i if j > i

Since $e^{- \infty} = 0$ , the softmax assigns zero weight to all future positions. Token $i$ can only attend to tokens $1, 2, \dots, i$ — it sees the past and present, but not the future. This is called causal or autoregressive attention.

The professor noted: "Masked attention is not useful for self attention [encoder side]. It is more useful for the decoder side." On the encoder side, every token can attend to every other token — there is no information asymmetry.

Scope: Masked self-attention is used only in the decoder. The encoder uses unmasked (bidirectional) self-attention, where every token sees every other token. If you are building an encoder-only model like BERT for contextual embeddings, you do not need masking.

12.4.2 Residual Connections

Each attention layer has a residual (skip) connection: the input vectors are added directly to the output of the attention computation, bypassing the attention and feedforward layers.

Mathematical form. For a sublayer function $f$ (which could be attention or feedforward), the residual connection computes:

output = f (x) + x

The input $x$ is added to the output of $f (x)$ . This means even if $f$ produces poor output (e.g., due to random initialisation), the input passes through unchanged — the network starts as an identity function and gradually learns to make useful adjustments.

Why: When transformers stack 16 to 32 layers, gradients can vanish during backpropagation — the same problem that motivated LSTMs. A residual connection provides a "gradient highway" directly from output to input, ensuring gradients can flow even through very deep networks.

The professor connected this to prior knowledge: "In the process of backpropagation through time, the problem which you saw in LSTM — vanishing gradient — can occur for this attention mechanism." Residual connections are the solution.

Intuition — the highway analogy: Imagine you need to deliver a package from floor 1 to floor 32 of a building. Without a residual connection, the package must pass through every floor's security checkpoint — if any checkpoint is broken, delivery fails. With a residual connection, there is an express elevator that skips all floors. The package can still visit intermediate floors if useful, but it always has a direct path to the destination.

Reference from textbook (R2, Ch. 7): The self-attention layer in the transformer combines self-attention with residual connections explicitly:

h^_{j} = layer-norm (self-attention (h_{j}) + h_{j})

Then applies a feed-forward step with another residual:

output_{j} = layer-norm (ReLU (W h^_{j} + b) + h^_{j})

Each sublayer (attention, feedforward) has its own residual connection and normalisation.

12.4.3 Layer Normalisation

After adding the residual, the values may not be on a consistent scale. Layer normalisation rescales the vector values to have zero mean and unit variance within each layer.

How layer norm works. For a vector $x \in R^{d}$ :

LayerNorm (x) = γ ⊙ \frac{x - μ}{σ ^{2} + ϵ} + β

where $μ = \frac{1}{d} \sum_{i = 1}^{d} x_{i}$ is the mean, $σ^{2} = \frac{1}{d} \sum_{i = 1}^{d} (x_{i} - μ)^{2}$ is the variance, $ϵ$ is a small constant for numerical stability, and $γ, β \in R^{d}$ are learned scale and shift parameters. The normalisation ensures that the values are centred around 0 with unit variance, regardless of how large or small the raw sums were.

The professor explained: "Because of the residual connection and because you are doing add and sum, sometimes you may have very large values which are not normalised. To keep them in a definite value — real-world values are generally normalised, falling in a certain curve — we try to normalise all the values."

12.4.4 Complete Encoder Block

The full encoder block processes data through these stages in sequence:

Input embedding — the static word embedding (e.g., from SGNS or GloVe)
+ Positional encoding — add sinusoidal position vectors (Section 12.3)
Multi-head self-attention — compute attention scores across all tokens (Section 12.1–12.2)
+ Residual connection — add the original input directly
Layer normalisation — rescale to consistent range
Feedforward network — fully connected layers for further transformation (typically two linear layers with ReLU in between)
+ Residual connection — another skip connection
Layer normalisation

The output at the end is the contextual word embedding for each input token. This block can be repeated 16–32 times (configurable as a hyperparameter). Each repetition refines the representations — early layers capture surface-level patterns, middle layers capture syntactic structure, and later layers capture semantic relationships.

12.4.5 Complete Decoder Block

The decoder follows the same structure but with two additions:

Masked multi-head self-attention — on the output sequence (prevents looking ahead, see Section 12.4.1)
+ Residual connection and layer normalisation
Cross-attention — queries from the decoder, keys and values from the encoder output (lets the decoder "read" the input)
+ Residual connection and layer normalisation
Feedforward network with residual connection and normalisation
Softmax output layer — to predict the next token

12.4.6 Model Variants

Encoder-only (e.g., BERT, sentence transformers): generate contextual word embeddings only. Use unmasked self-attention. Used for classification, similarity, NER, and any task that needs rich token representations.
Decoder-only (e.g., GPT, Claude): generate text autoregressively. Use masked self-attention. The dominant architecture for modern LLMs.
Encoder-decoder (e.g., BART, T5): used for sequence-to-sequence tasks like machine translation. The encoder reads the input with unmasked attention; the decoder generates the output with masked self-attention and cross-attention to the encoder.

Smaller variants like DistilBERT are "cheaper and faster if you want to do it on lower resources, if you don't have GPUs."

Common Pitfalls:

Using masked attention in the encoder. The encoder uses unmasked (bidirectional) self-attention. Masking is only for the decoder.

Forgetting residual connections. Without them, deep transformers (16+ layers) suffer from vanishing gradients and fail to train. Residual connections are not optional — they are a core architectural component.

Confusing layer norm with batch norm. Layer normalisation normalises across features for a single token; batch normalisation normalises across tokens in a batch. Transformers use layer norm because it works better for variable-length sequences.

12.4.7 Student Questions and Answers

Q: Did you cover transformer architecture in any of the courses?

A: The students confirmed partial coverage in the Deep Neural Networks course — a basic example with simple values, but not in detail.

Q: You mentioned it can be done in parallel. Is this how it's actually implemented?

A: Yes. All query-key dot products become a single matrix multiplication $Q K^{⊤}$ , softmax is applied row-wise, and the result is multiplied by $V$ . This is the core advantage over RNNs, which must process tokens sequentially.

Recap: The complete transformer encoder block chains: embedding → positional encoding → multi-head self-attention → residual + layer norm → feedforward → residual + layer norm. Masked self-attention is decoder-only (prevents looking at future tokens). Residual connections solve vanishing gradients in deep networks. Layer normalisation stabilises training by keeping values on a consistent scale. This block is stacked 16–32 times to build deep representations.

12.5 Word Sense Disambiguation — Introduction

Hook: If a user asks a chatbot about "bank transactions," the system must not retrieve information about river banks. If a machine translator sees "bass," it must decide: is this a fish (Spanish: lubina) or a musical instrument (Spanish: bajo)? Word sense disambiguation (WSD) is the detective work of figuring out which meaning of a word is intended in a given context.

Word sense disambiguation (WSD) is the task of determining which meaning of a word is intended in a given context. Every NLP application — machine translation, conversational AI, search, information retrieval — requires resolving ambiguity. The professor framed this as the bridge between the transformer-based contextual embeddings (Sections 12.1–12.4) and practical NLP systems: once you have rich word representations, you need to know which sense they represent.

12.5.1 Lexemes and Word Senses

A lexeme is the technical NLP term for a word entry in the mental dictionary (the lexicon). Each lexeme has:

A lemma (the root form or dictionary headword — "carpet" is the lemma of "carpets"; "sing" is the lemma of "singing, sang, sung")
One or more senses (discrete meanings)

The sense of a word in context is its specific meaning in that sentence or document. For example, the lexeme "bank" has at least two senses: "financial institution" and "sloping land beside a river." The task of WSD is to select the correct sense for each occurrence.

12.5.2 Types of Lexical Relations

Understanding how words relate to each other is essential for disambiguation. The four primary lexical relations are:

Homonymy — same spelling or pronunciation, completely different meanings with no etymological connection. "Bat" (wooden sports implement) vs. "bat" (flying mammal). These are "identical strangers" — they share a face (spelling) but have no shared DNA (meaning).

Polysemy — same spelling, related but distinct meanings. "Wood" as material from a tree vs. "wood" as a small forest. Both relate to trees, but the senses differ. Polysemy is harder to handle than homonymy because the senses overlap and the boundaries between them are fuzzy.

Synonymy — different words, same (or very similar) meaning. "Water" and "H₂O," "big" and "large." Context determines which is appropriate — you would say "give me water" at home, not "give me H₂O." True synonymy is rare; even "big" and "large" differ in connotation ("big sister" ≠ "large sister").

Hyponymy / Hypernymy — class hierarchy ("is-a" relationship). "Car" is a hyponym (child, more specific) of "vehicle"; "vehicle" is a hypernym (parent, more general) of "car." These relationships help disambiguation: if the parent class in context is "animal," "bat" means the mammal; if it is "sports equipment," "bat" means the wooden implement.

Why these relations matter for WSD: Homonymy requires a forced choice between unrelated meanings (easy to distinguish with context). Polysemy requires distinguishing closely related meanings (harder — even humans disagree). Synonymy helps because knowing synonyms of the correct sense can match context words. Hyponymy/hypernymy provides hierarchical clues: if the context mentions "fishing," the hypernym "aquatic creature" helps select the fish sense of "bass."

12.5.3 WordNet

WordNet is a lexical database developed by Princeton University, freely available online. It is structured as a graph where every word is connected to other words through typed relationships: synonymy, hypernymy, hyponymy, meronymy (part-of), entailment, antonymy, and more.

Each word entry in WordNet includes:

Multiple synsets (synonym sets) — one per sense
Part-of-speech tags
Glosses (definitions)
Usage examples
Relationships to other words

For example, the word "bass" has 8 noun senses and 1 adjective sense (9 total meanings). The synset for "bass" as a singer includes related words: "basso, adult male singer with lowest voice." The hierarchy goes: bass → singer → musician → performer → entertainer → person → organism → living thing → entity.

WordNet functions as a sophisticated graph-structured dictionary that not only defines words but maps their relationships. It is domain-independent (generic English) and is the foundation for knowledge-based WSD approaches.

WordNet in practice (preview of Section 12.8): Using NLTK, you can explore WordNet programmatically:

from nltk.corpus import wordnet as wn
for syn in wn.synsets("bass"):
    print(syn.name(), syn.definition())

This prints all 9 senses of "bass" with their definitions — the raw material for disambiguation.

12.5.4 SemCor Dataset

SemCor is a standard WSD evaluation dataset. It consists of Brown University text documents where each ambiguous word is manually tagged with its correct WordNet sense. For example, the word "find" might be tagged with its 9th verb sense in one sentence and its 1st sense in another. This labelled "gold data" is used to train and evaluate both machine learning and knowledge-based WSD systems.

Scope: SemCor covers only a subset of English words and senses. Words or senses not in SemCor cannot be directly evaluated using this dataset. For low-resource languages or domain-specific terms, alternative datasets or unsupervised methods are needed.

12.5.5 All-Words vs Lexical Sample Tasks

Two variants of the WSD task exist:

All-words task: disambiguate every word in the input text (including unambiguous ones). Computationally expensive — every word needs a sense lookup and a disambiguation decision.

Lexical sample task: disambiguate only words that are likely to have multiple meanings. This is the standard in industry — you focus resources on ambiguous words.

Practical optimisation: Function words (articles like "the," "a"; conjunctions like "and"; prepositions like "to") typically have a single meaning and can be skipped. Focus on open-class words: nouns, verbs, adjectives, adverbs. This is similar to how POS tagging focuses on structural cues for function words but uses broader context for content words.

12.5.6 Simple Heuristics

Two practical shortcuts reduce computation:

Most frequent sense (MFS) fallback: If a word is not in the training data or no context signal is available, select the first (most common) sense listed in WordNet. This baseline is surprisingly effective — for many words, the most common sense accounts for 60–80% of all occurrences. Always compare your WSD system against this baseline; if it cannot beat MFS, it is not learning anything useful.

One-sense-per-discourse: If a word like "bass" has been disambiguated once in a document as "musical instrument," all subsequent occurrences in the same document can use the same sense without re-computation. This heuristic holds about 90% of the time in practice (Gale et al., 1992).

Common Pitfall — ignoring the MFS baseline: Many fancy WSD systems fail to outperform the simple "always pick sense 1" heuristic. Always establish this baseline before investing in more complex approaches.

Recap: WSD determines which meaning of a word is intended in context. Lexical relations (homonymy, polysemy, synonymy, hyponymy) provide the theoretical framework. WordNet is the primary knowledge base with synsets, glosses, and hierarchical relations. The lexical-sample task focuses on ambiguous words; the all-words task covers everything. The most-frequent-sense heuristic is a strong baseline that any serious WSD system must beat.

12.6 Supervised Machine Learning for WSD

Hook: We have WordNet with its senses and glosses, and we have labelled training data (SemCor). Can we treat WSD as a standard classification problem — train a classifier on labelled examples, then predict the sense for new occurrences? Yes, and this is the most accurate approach when labelled data is available.

12.6.1 Approach

Treat WSD as a classification problem. For each ambiguous word, train a classifier to predict its sense given the surrounding context.

Requirements:

Labelled training corpus (like SemCor) with each occurrence tagged by sense
Feature engineering (for traditional ML) or automatic feature learning (for deep learning)
A classification algorithm

Features can include:

Bag-of-words of surrounding context (window of ±2 words) — which words appear near the target
Part-of-speech tags of the target word and neighbours — structural cues
Collocation features — which words frequently co-occur with each sense
Word embeddings (SGNS, GloVe) of context words — dense vector representations

Algorithms: Naive Bayes, SVM, decision trees, gradient boosting (XGBoost, AdaBoost), feedforward networks, LSTMs. LSTMs and transformers learn features automatically and do not require manual feature engineering.

Limitation: whatever senses were present in the training data, only those can be predicted at test time. Applying the model to a new domain or new words requires retraining on new labelled data.

Scope — the supervised bottleneck: Supervised WSD requires labelled training data for every ambiguous word. This is expensive to produce — each word needs dozens of labelled examples per sense. For words not in the training set, the system cannot make predictions. This is why knowledge-based approaches (Section 12.7) are important as a complement.

12.6.2 Worked Example: Naive Bayes Classifier for WSD

The professor walked through a complete Naive Bayes computation for disambiguating "bass" as either "fish sense" or "guitar sense."

Training data (4 documents, labelled):

Doc	Words (relevant)	Sense
D1	line, fish, ocean	fish
D2	jazz, guitar, music	guitar
D3	fish, ocean, water	fish
D4	line, water, fish	fish

Step 1: Compute prior probabilities.

The prior probability of each sense is the fraction of training documents with that sense:

P (fish) = \frac{3}{4} = 0.75, P (guitar) = \frac{1}{4} = 0.25

Step 2: Build the vocabulary and count words.

Unique vocabulary across all 4 documents: ${) l in e, f i s h, oce an, j a z z, g u i t a r, m u s i c, w a t er (}$ — size $V = 7$ .

Total word occurrences in fish class (D1 + D3 + D4): D1 has 3 words (line, fish, ocean), D3 has 3 words (fish, ocean, water), D4 has 3 words (line, water, fish). So $N_{fish} = 9$ .

Total word occurrences in guitar class (D2): D2 has 3 words (jazz, guitar, music). So $N_{guitar} = 3$ .

Step 3: Compute likelihoods with Laplace (add-1) smoothing.

The likelihood of word $w$ given class $c$ with Laplace smoothing is:

P (w ∣ c) = \frac{count ( w , c ) + 1}{N _{c} + V}

Likelihoods for fish class ( $N_{fish} = 9$ , $V = 7$ , denominator = $9 + 7 = 16$ ):

P (line ∣ fish) = \frac{2 + 1}{16} = \frac{3}{16}

(line appears in D1 and D4 — 2 times total)

P (guitar ∣ fish) = \frac{0 + 1}{16} = \frac{1}{16}

(guitar does not appear in any fish document)

P (jazz ∣ fish) = \frac{0 + 1}{16} = \frac{1}{16}

Likelihoods for guitar class ( $N_{guitar} = 3$ , $V = 7$ , denominator = $3 + 7 = 10$ ):

P (line ∣ guitar) = \frac{0 + 1}{10} = \frac{1}{10}

P (guitar ∣ guitar) = \frac{1 + 1}{10} = \frac{2}{10}

P (jazz ∣ guitar) = \frac{1 + 1}{10} = \frac{2}{10}

Step 4: Compute posteriors for the test document.

Test document (D5): words = "line, guitar, jazz, jazz" (note: "jazz" appears twice).

The Naive Bayes posterior for class $c$ given document $D$ is:

P (c ∣ D) \propto P (c) \times w \in D \prod P (w ∣ c)

Posterior for fish:

P (fish ∣ D 5) = P (fish) \times P (line ∣ fish) \times P (guitar ∣ fish) \times P (jazz ∣ fish)^{2}

= \frac{3}{4} \times \frac{3}{16} \times \frac{1}{16} \times (\frac{1}{16})^{2} = 0.75 \times 0.1875 \times 0.0625 \times 0.00390625

\approx 0.75 \times 0.1875 \times 0.0625 \times 0.00391 \approx 3.43 \times 1 0^{- 5}

Posterior for guitar:

P (guitar ∣ D 5) = P (guitar) \times P (line ∣ guitar) \times P (guitar ∣ guitar) \times P (jazz ∣ guitar)^{2}

= \frac{1}{4} \times \frac{1}{10} \times \frac{2}{10} \times (\frac{2}{10})^{2} = 0.25 \times 0.1 \times 0.2 \times 0.04

= 0.25 \times 0.1 \times 0.2 \times 0.04 = 2.0 \times 1 0^{- 4}

Step 5: Compare and decide.

P (fish ∣ D 5) \approx 3.43 \times 1 0^{- 5} < P (guitar ∣ D 5) \approx 2.0 \times 1 0^{- 4}

Prediction: guitar sense. The classifier predicts that "bass" in D5 refers to the musical instrument, not the fish.

Sense-check: The key factor is that "jazz" appears twice in the test document and is strongly associated with the guitar class (appears in D2), while "jazz" has zero count in the fish class. The word "line" slightly favours fish (appears in 2 fish documents), but it is overwhelmed by the jazz signal. Laplace smoothing ensures that zero-count words (like "jazz" in fish class) get a small non-zero probability rather than zeroing out the entire product.

Common Pitfall — forgetting to handle repeated words: In the test document, "jazz" appears twice. The likelihood $P (jazz ∣ c)$ is raised to the power of 2 in the posterior computation. Do not count repeated words only once — each occurrence contributes independently to the probability.

Common Pitfall — zero probabilities without smoothing: Without Laplace smoothing, any word that does not appear in a class's training data would have $P (w ∣ c) = 0$ , which zeroes out the entire product regardless of how strong the other evidence is. Laplace smoothing (add-1) prevents this by giving every word a minimum probability of $1/ (N_{c} + V)$ .

12.6.3 K-Nearest Neighbours

KNN is a "lazy algorithm" — it computes distances at runtime to all training examples, which is expensive for large datasets. For WSD, KNN finds the $k$ most similar training contexts to the test context and votes on the sense. Naive Bayes and other eager learners (SVM, decision trees) are more popularly used for WSD because they precompute a model during training and are faster at prediction time.

Recap: Supervised WSD treats sense selection as classification. Features include bag-of-words, POS tags, collocations, and embeddings. The Naive Bayes classifier computes posterior probabilities using priors and likelihoods with Laplace smoothing. The key limitation is the need for labelled training data per word per sense.

12.7 Knowledge-Based WSD: The Lesk Algorithm

Hook: Supervised WSD works well — if you have labelled training data. But labelling is expensive, and most words in most languages do not have labelled examples. Can we disambiguate words using only a dictionary, with no training data at all? Yes — the Lesk algorithm does exactly this, and it is still used in production systems today.

12.7.1 Motivation

The Lesk algorithm requires no training data. It uses an existing knowledge base (WordNet, Wikipedia, or any dictionary with sense definitions) and the overlap between context words and sense definitions to select the correct sense. It is transparent, requires minimal computational resources, and is still used in educational platforms, low-resource languages, small enterprise search systems, and embedded systems.

The GPU cost reality: The professor noted: "When you talk about the transformer, immediately you have to think about GPUs, which cost about 30 lakhs of rupees. Only if there is a return on investment is it worth buying these GPUs. For startup IT companies or even mid-level companies, it may not be advisable to use transformers for each and every problem." A hybrid approach combining simple algorithms like Lesk with transformers for high-value tasks is the pragmatic production strategy.

12.7.2 Algorithm Steps

The Lesk Intuition: The correct sense's definition (gloss) probably shares words with the context. If the context says "deposit" and "mortgage," and one sense of "bank" has a gloss containing "financial institution that accepts deposits," that sense is likely correct.

Input: A sentence containing an ambiguous word $w$ .

Step 1: Preprocess — remove stop words ("the," "can," "will," "in"), keep open-class words (nouns, verbs, adjectives). Build a context vector $C$ from the remaining words.

Step 2: For each sense $s_{i}$ of word $w$ in WordNet:

Build a signature $S_{i}$ from the words in the definition (gloss) and example sentence for sense $s_{i}$ .

Step 3: Compute the overlap between $C$ and each $S_{i}$ :

score (s_{i}) = ∣ C \cap S_{i} ∣

Step 4: Select the sense with maximum overlap:

s^= ar g s_{i} max score (s_{i})

Step 5 (optimisation): If no sense has significant overlap with the context, fall back to the most frequent sense in WordNet.

Intuition — the detective analogy: Imagine you find a note with the word "bank" and surrounding clues: "deposit," "mortgage," "investments." You open a dictionary and compare each sense's definition against your clues. Sense 1 (financial institution) mentions "deposits" and "loans" — two matches. Sense 2 (river bank) mentions "sloping land" and "water" — zero matches. Case closed: it is the financial sense.

12.7.3 Worked Example: Lesk Algorithm

The professor demonstrated with: "The bank can guarantee deposit will eventually cover future tuition cost because it invests in adjustable rate mortgage securities."

Target word: "bank."

Step 1: Remove stop words.

Stop words: "the," "can," "will," "eventually," "it," "in."

Context vector $C = {) bank, g u a r an t ee, d e p os i t, co v er, f u t u r e, t u i t i o n, cos t, in v es t, a d j u s t ab l e, r a t e, m or t g a g e, sec u r i t i es (}$ .

Step 2: Get WordNet senses for "bank" and their signatures.

Sense 1 (financial institution): gloss — "a financial institution that accepts deposits and channels the money into lending activities." Signature $S_{1} = {) f inan c ia l, in s t i t u t i o n, a cce pt s, d e p os i t s, c hann e l s, m o n ey, l e n d in g, a c t i v i t i es (}$ . Example: "bank offers savings accounts." Additional words: ${) o f f er s, s a v in g s, a cco u n t s (}$ .

Sense 2 (river bank): gloss — "sloping land (especially the slope beside a body of water)." Signature $S_{2} = {) s l o p in g, l an d, s l o p e, b es i d e, b o d y, w a t er (}$ . Example: "he sat on the bank of the river." Additional words: ${) s a t, r i v er (}$ .

Step 3: Compute overlap.

Overlap with Sense 1: $C \cap S_{1} = {) d e p os i t, m or t g a g e (}$ — score = 2 (or more if other gloss words like "money" match context words — the exact count depends on how broadly you define the signature).

Overlap with Sense 2: $C \cap S_{2} = {}$ — score = 0

Step 4: Select sense with maximum overlap.

Sense 1 (financial institution) wins with score 2 > 0.

Sense-check: The words "deposit" and "mortgage" appear both in the test sentence and in WordNet's definition of the financial sense, but no context words match the river-bank sense. The algorithm correctly identifies the financial meaning without any training data.

Common Pitfalls:

Not removing stop words. If you keep "the," "in," "it," these will match glosses for every sense (all glosses use common words), diluting the signal from content words.

Using only the gloss, not the example sentence. WordNet provides both a definition and an example for each sense. Including the example sentence in the signature increases the overlap surface and improves accuracy.

Not falling back to MFS. If the context has no overlap with any sense (e.g., the sentence uses unusual vocabulary), the algorithm should fall back to the most frequent sense rather than returning a random or zero-overlap sense.

12.7.4 Using Wikipedia as a Knowledge Source

Instead of WordNet, Wikipedia can serve as the sense inventory. Wikipedia has disambiguation pages (e.g., the "bank" disambiguation page lists: financial institution, river bank, blood bank, etc.). Each Wikipedia article for a specific sense provides context words that can be used as the signature for overlap computation. Wikipedia is widely used in both academic research and industry for WSD because it covers more words and senses than WordNet, including proper nouns and domain-specific terms.

Recap: The Lesk algorithm disambiguates words using dictionary glosses — no training data needed. It builds a context vector from the sentence (minus stop words) and a signature from each sense's gloss, then selects the sense with maximum word overlap. It is simple, transparent, and practical for resource-constrained environments. Wikipedia can replace WordNet as the knowledge source for broader coverage.

12.8 Practical Implementation: WordNet with NLTK

Hook: We have learned the theory of WSD — supervised (Naive Bayes) and knowledge-based (Lesk). Now let us see how to actually implement WSD using Python's NLTK library, which provides a direct interface to WordNet. This section bridges theory to code.

12.8.1 NLTK WordNet Interface

The NLTK library provides a Python interface to WordNet via nltk.corpus.wordnet. Key capabilities include exploring senses, navigating the hierarchy, and computing similarity between words.

Exploring senses:

from nltk.corpus import wordnet as wn

# All synsets for "education"
for syn in wn.synsets("education"):
    print(syn.name(), syn.definition(), syn.examples())

The word "education" has multiple synsets, each with a definition and examples. The POS tag is part of the synset name (e.g., education.n.01 for noun sense 1, education.n.02 for noun sense 2).

Hyponyms and hypernyms:

# Child classes of "cat"
cat = wn.synset("cat.n.01")
print(cat.hyponyms())   # domestic cat, wild cat, etc.

# Parent classes of "cat"
print(cat.hypernyms())  # feline, carnivore, etc.

Synonyms and antonyms:

for syn in wn.synsets("good"):
    for lemma in syn.lemmas():
        if lemma.antonyms():
            print(lemma.name(), lemma.antonyms()[0].name())

12.8.2 Path Similarity and WUP Similarity

WordNet provides built-in similarity measures based on the graph structure:

Path similarity: inverse of the shortest path length between two synsets in the taxonomy graph. Returns a value in $[0, 1]$ , where 1 means identical synsets.

car = wn.synset("car.n.01")
automobile = wn.synset("automobile.n.01")
boat = wn.synset("boat.n.01")

print(car.path_similarity(automobile))  # 1.0 (same node)
print(car.path_similarity(boat))        # 0.125

Car and automobile are identical (path similarity = 1.0), while car and boat are distant (0.125), despite both being modes of transportation.

WUP (Wu-Palmer) similarity: a depth-based measure that considers the depth of the least common subsumer (nearest common ancestor) in the taxonomy:

WUP (s_{1}, s_{2}) = \frac{2 \cdot depth ( LCS )}{depth ( s _{1} ) + depth ( s _{2} )}

where LCS is the lowest common subsumer. WUP gives a better similarity score for car and boat (reflecting that both are vehicles) compared to path similarity, because their common ancestor "vehicle" is at a meaningful depth.

The professor advised: "It depends on your application which kind of similarity measure you want to use."

Comparison of similarity measures:

Word pair	Path similarity	WUP similarity	Interpretation
car – automobile	1.0	1.0	Identical synsets
car – boat	0.125	~0.67	Both are vehicles (WUP captures this)
car – banana	~0.07	~0.29	Very distant in taxonomy

Path similarity is strict (only close synsets score high); WUP is more forgiving for words that share a meaningful ancestor.

12.8.3 WSD Using WordNet Similarity: Worked Example

The professor demonstrated WSD for the sentence "river bank is beautiful."

Step 1: POS-tag the sentence. Keep only nouns: "river" (noun), "bank" (noun).

Step 2: Get all synsets for both words.

"river": senses related to flowing water, body of water
"bank": senses include financial institution, river bank, blood bank, etc.

Step 3: Compute similarity between each synset of "river" and each synset of "bank" using path similarity.

Step 4: Select the pair with maximum similarity. The river-water sense of "river" and the sloping-land sense of "bank" (i.e., river bank) have the highest similarity score.

Result: "Bank" is disambiguated as "sloping land beside a body of water" in this sentence.

Additional example: "plant" in different sentences is disambiguated as manufacturing plant vs. flowering plant based on context words ("industrial" vs. "garden"). The maximum-similarity pairing correctly identifies the intended sense in each case.

12.8.4 Error Note

The professor encountered a minor error in the code: "I think I did not run the first line, WN, so probably it gave an error, but it's a running code. I'll share it later." This was a missing import statement (from nltk.corpus import wordnet as wn), not a logic error. Always ensure the import is present before using WordNet functions.

12.8.5 Student Questions and Answers

Q: After identifying the correct sense in a sentence, where and how is that information used downstream?

A: The correct sense can serve as an additional dimension in the word embedding, or feed directly into downstream applications:

Conversational AI / chatbots for correct query interpretation (e.g., "bank transaction" should not trigger river-bank information)
Search engines for accurate retrieval (filtering documents by the intended sense)
Workflow management systems for decision support
Machine translation for selecting the right target-language word (e.g., "bass" as fish vs. musical instrument translates to different words in most languages)

Knowing the correct sense is a foundational step that improves every subsequent NLP task.

Recap: NLTK provides a Python interface to WordNet for exploring synsets, navigating hierarchies, and computing similarity. Path similarity uses shortest graph distance; WUP similarity uses depth-based distance and captures hierarchical relationships better. WSD using WordNet similarity works by finding the synset pair (one from each word) with the highest similarity score. Always include the import statement.

12.9 Exam Guidance Summary

12.9.1 Expected Question Types

Exam note:

A numerical problem on attention score computation can be expected. The professor confirmed: "Yes, you can. It's a simple one." Use the worked example in Section 12.1.4 as the template. Be prepared to: (a) compute key/query/value vectors from input vectors and weight matrices, (b) compute dot-product scores, (c) apply softmax, (d) compute the final weighted sum.
Word sense disambiguation is part of the syllabus and likely examinable, including both the ML approach (Naive Bayes with Laplace smoothing — Section 12.6.2) and the knowledge-based approach (Lesk algorithm — Section 12.7.3).

12.9.2 Exam Format and Study Advice

Exam note:

The exam is open-book: students can bring printed slides and notes. No need to memorise relations or formulas.
A review session will be held before the end-semester exam to discuss expected mathematical problems and question patterns.
Focus your preparation on: (1) the step-by-step attention computation, (2) the Naive Bayes WSD worked example, and (3) the Lesk algorithm worked example. These three cover the most likely exam questions.

12.10 Key Industry Applications

12.10.1 Transformer-Based NLP in Production

Transformers and contextual embeddings are foundational to all modern NLP: GPT, BERT, T5, Hugging Face sentence transformers, and every LLM in production today.
GPU costs (about 30 lakh INR for a training-grade GPU) make transformer-based solutions impractical for small enterprises and startups without clear ROI.
DistilBERT and smaller transformer variants provide cost-effective contextual embeddings for resource-constrained environments.

12.10.2 Knowledge-Based and Hybrid WSD Systems

Lesk algorithm and knowledge-based WSD are used in educational platforms, low-resource Indian languages, small enterprise search systems, and embedded systems where computational resources are limited.
Hybrid approaches combining simple algorithms (Lesk, Naive Bayes) with transformers are the pragmatic production strategy — using the right tool for the right problem.
WordNet is the foundational knowledge graph underlying many NLP systems, including graph RAG, agentic RAG, and conversational AI systems.
Wikipedia is widely used as a knowledge source for WSD in both industry and academic research.

12.10.3 Application Domains

Conversational AI / chatbots require WSD to correctly interpret user queries (e.g., "bank transaction" should not trigger river-bank information).
Machine translation requires WSD to select the correct target-language word for an ambiguous source word (e.g., "bass" as fish vs. musical instrument translates to different words in most languages).
Search and information retrieval use contextual embeddings to match documents by meaning, not just keyword overlap. WSD improves precision by filtering out documents using the wrong sense.
Knowledge graphs and semantic search build on WordNet-style sense inventories to link entities and concepts across documents.

NLP Lecture 12 notes · Contextual Word Embedding and Word Sense Disambiguation

Natural Language Processing· postgraduate· 2026-07-27

Sections Breakdown

1Attention Score Computation and Contextual Word Embeddings

Scaled dot-product attention, query/key/value vectors, softmax, and the matrix form of attention.

2Multi-Headed Attention

Running multiple attention computations in parallel with separate weight matrices to capture different relationship types.

3Positional Encoding

Injecting word-order information via sinusoidal sine/cosine signals added to word embeddings.

4Masked Self-Attention, Residual Connections, and Normalisation

Decoder-only causal masking, skip connections for gradient flow, and layer normalisation for stable training.

5Word Sense Disambiguation — Introduction

Lexical relations, WordNet synsets, SemCor, and the most-frequent-sense baseline.

6Supervised Machine Learning for WSD

Naive Bayes classification with Laplace smoothing for sense selection.

7Knowledge-Based WSD: The Lesk Algorithm

Dictionary-gloss overlap method requiring no training data.

8Practical Implementation: WordNet with NLTK

Python interface to WordNet for synset exploration, hierarchy navigation, and similarity computation.

9Exam Guidance Summary

Expected question types, exam format, and study advice.

10Key Industry Applications

Transformers in production, knowledge-based WSD, and hybrid approaches.

Postgraduate students in Natural Language Processing

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Attention Score Computation and Contextual Word Embeddings

Must-know: Compute attention scores step-by-step: multiply input by W_Q/W_K/W_V to get vectors, dot-product to get scores, softmax to get weights, weighted sum of V to get contextual embedding. Know the matrix form: softmax(QK^T/sqrt(d_k)) V.

$Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V$

Top pitfall: Confusing shared weight matrices (same for all tokens) with per-token output vectors. Forgetting the scaling factor sqrt(d_k).

Self-check: Given x1=[1,0], W_K=[[1,0],[0,1]], what is k1?

Connects to: 12.2, 12.3, 12.4

Multi-Headed Attention

Must-know: All words go through all heads — features are split, not tokens. Each head operates on d/H dimensions. Outputs are concatenated and projected.

$MultiHead (X) = Concat (head_{1}, \dots, head_{H}) W_{O}$

Top pitfall: Thinking tokens are split across heads. In reality, feature dimensions are split.

Self-check: If d=512 and H=8, what is the dimension per head?

Connects to: 12.1, 12.4

Positional Encoding

Must-know: Positional encoding uses sine (even dims) and cosine (odd dims) at different frequencies. It is added to word embeddings (not concatenated). Necessary because attention is permutation-invariant.

$P E_{(p os, 2 i)} = sin (\frac{p os}{1000 0 ^{2 i / d}}), P E_{(p os, 2 i + 1)} = cos (\frac{p os}{1000 0 ^{2 i / d}})$

Top pitfall: Thinking positional encoding replaces word embeddings. It is added to them, not a substitute.

Self-check: Why is positional encoding necessary for transformers?

Connects to: 12.1, 12.4

Masked Self-Attention, Residual Connections, and Normalisation

Must-know: Encoder block: embedding + positional encoding → self-attention → residual + layer norm → feedforward → residual + layer norm. Masked attention is decoder-only. Residual connections prevent vanishing gradients.

$output = LayerNorm (f (x) + x)$

Top pitfall: Using masked attention in the encoder. Forgetting that residual connections are essential, not optional.

Self-check: Why are residual connections necessary in deep transformers?

Connects to: 12.1, 12.2, 12.3

Word Sense Disambiguation — Introduction

Must-know: Four lexical relations: homonymy (unrelated meanings), polysemy (related meanings), synonymy (same meaning), hyponymy/hypernymy (is-a hierarchy). WordNet synsets group synonyms per sense. MFS baseline picks the most common sense.

Top pitfall: Ignoring the MFS baseline. Confusing homonymy with polysemy.

Self-check: What is the difference between homonymy and polysemy? Give an example of each.

Connects to: 12.6, 12.7, 12.8

Supervised Machine Learning for WSD

Must-know: Naive Bayes for WSD: compute priors from class frequency, likelihoods with Laplace smoothing (count+1)/(N+V), multiply to get posterior. Handle repeated words by raising likelihood to power of count.

$P (w ∣ c) = \frac{count ( w , c ) + 1}{N _{c} + V}$

Top pitfall: Forgetting to handle repeated words (raise to power). Not using Laplace smoothing (zero probabilities kill the product).

Self-check: Given P(fish)=0.75, P(line|fish)=3/16, P(guitar|fish)=1/16, compute P(fish) x P(line|fish) x P(guitar|fish).

Connects to: 12.5, 12.7

Knowledge-Based WSD: The Lesk Algorithm

Must-know: Lesk algorithm: remove stop words, build context vector C, build signature S_i from each sense's gloss, score by |C ∩ S_i|, pick argmax. Falls back to MFS if no overlap.

$s^= ar g s_{i} max ∣ C \cap S_{i} ∣$

Top pitfall: Not removing stop words. Not including gloss examples in signature. Not falling back to MFS.

Self-check: Apply Lesk to 'He played bass in the jazz band' for the word 'bass'. Which sense would win?

Connects to: 12.5, 12.6, 12.8

Practical Implementation: WordNet with NLTK

Must-know: NLTK WordNet interface: wn.synsets(), .hyponyms(), .hypernyms(), .path_similarity(), .wu_palmer_similarity(). WSD by finding max-similarity synset pair across context words.

$WUP (s_{1}, s_{2}) = \frac{2 \cdot depth ( LCS )}{depth ( s _{1} ) + depth ( s _{2} )}$

Top pitfall: Forgetting the import statement. Confusing path similarity (strict) with WUP similarity (depth-based, more forgiving).

Self-check: What is the path_similarity between car.n.01 and automobile.n.01? Why?

Connects to: 12.5, 12.7

Exam Guidance Summary

Must-know: Attention score numerical, Naive Bayes WSD numerical, Lesk algorithm — these three are the expected exam question types. Open-book exam.

Connects to: 12.1, 12.6, 12.7

Key Industry Applications

Must-know: GPU costs (~30 lakh INR) limit transformer adoption. Lesk + WordNet for low-resource settings. Hybrid approaches are pragmatic.

Connects to: 12.1, 12.7

Previous LectureLecture 11 · Contextual Word Embedding and Attention Mechanisms

Next LectureLecture 13 · Knowledge Graphs and Semantic Web

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Contextual Word Embedding and Word Sense Disambiguation

12.1 Attention Score Computation and Contextual Word Embeddings

12.1.1 Recap: Why Attention Is Needed

12.1.2 Key, Query, and Value Vectors

12.1.3 Mathematical Formulation: Attention Scores

12.1.4 Worked Example: Attention Score for Three Words

12.1.5 Parallelism Advantage

12.1.6 Encoder-Only Sufficiency for Contextual Embeddings

12.1.7 Student Questions and Answers

12.1.8 Exam Notes

12.1.9 Real-World & Domain Connection

12.2 Multi-Headed Attention

12.2.1 Why Multiple Heads?

12.2.2 Dimensionality Splitting

12.2.3 Multi-Query Attention and Group Attention

12.2.4 Self-Attention vs Cross-Attention

12.2.5 Student Questions and Answers

12.3 Positional Encoding

12.3.1 Sinusoidal Positional Encoding

12.3.2 Why Multiple Curves Prevent Repetition

12.3.3 Combining Positional Encoding with Word Embeddings

12.3.4 Student Questions and Answers

12.4 Masked Self-Attention, Residual Connections, and Normalisation

12.4.1 Masked Self-Attention (Decoder Side)

12.4.2 Residual Connections

12.4.3 Layer Normalisation

12.4.4 Complete Encoder Block

12.4.5 Complete Decoder Block

12.4.6 Model Variants

12.4.7 Student Questions and Answers

12.5 Word Sense Disambiguation — Introduction

12.5.1 Lexemes and Word Senses

12.5.2 Types of Lexical Relations

12.5.3 WordNet

12.5.4 SemCor Dataset

12.5.5 All-Words vs Lexical Sample Tasks

12.5.6 Simple Heuristics

12.6 Supervised Machine Learning for WSD

12.6.1 Approach

12.6.2 Worked Example: Naive Bayes Classifier for WSD

12.6.3 K-Nearest Neighbours

12.7 Knowledge-Based WSD: The Lesk Algorithm

12.7.1 Motivation

12.7.2 Algorithm Steps

12.7.3 Worked Example: Lesk Algorithm

12.7.4 Using Wikipedia as a Knowledge Source

12.8 Practical Implementation: WordNet with NLTK

12.8.1 NLTK WordNet Interface

12.8.2 Path Similarity and WUP Similarity

12.8.3 WSD Using WordNet Similarity: Worked Example

12.8.4 Error Note

12.8.5 Student Questions and Answers

12.9 Exam Guidance Summary

12.9.1 Expected Question Types

12.9.2 Exam Format and Study Advice

12.10 Key Industry Applications

12.10.1 Transformer-Based NLP in Production

12.10.2 Knowledge-Based and Hybrid WSD Systems

12.10.3 Application Domains

NLP Lecture 12 notes · Contextual Word Embedding and Word Sense Disambiguation

Sections Breakdown

Exam Revision Notes

Attention Score Computation and Contextual Word Embeddings

Multi-Headed Attention

Positional Encoding

Masked Self-Attention, Residual Connections, and Normalisation

Word Sense Disambiguation — Introduction

Supervised Machine Learning for WSD

Knowledge-Based WSD: The Lesk Algorithm

Practical Implementation: WordNet with NLTK

Exam Guidance Summary

Key Industry Applications

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat