Word Embeddings and Word2Vec

3.1 Word Embedding Recall and Motivation

3.1.1 Why Words Become Vectors

A word embedding (how a word is placed into a numeric space) is the act of converting a word into a vector — a list of real numbers. The reason is two-fold:

1. Similarity becomes computable. With vectors you can measure how alike two words are using a dot product or cosine similarity. Two strings cannot be compared this way at all.
2. Meaning becomes encoded. A vector built the right way is a semantic representation: the numbers reflect what the word means, not how it is spelled. "Apple" and "mango" should land near each other; "apple" and "xylophone" should not.

Why does "built the right way" work? The guiding idea is the distributional hypothesis, the oldest idea in this field: a word is known by the company it keeps. Words that appear in similar surroundings (say, both preceded by "ripe" and "juicy") are used to mean similar things, so a good training algorithm places their vectors close together. The words themselves carry the signal — nobody writes down meanings by hand.

Where this session is heading: the star of today is word2vec and its workhorse variant, skip-gram with negative sampling (SGNS), the algorithm industry used everywhere before transformers arrived. Contextual embeddings built with attention inside transformers come in a later module, after the mid-semester break.

3.1.2 Recap: Frequency-Based Approaches and Their Problems

The previous session covered frequency-based embeddings. These are counting methods: TF-IDF weights, count vectors, and co-occurrence counts gathered inside a window of plus/minus 4 words around each target word. Every one of them counts how often words occur — inside a document, or inside a window, or across the whole collection. Counting is simple and cheap, and TF-IDF in particular still powers real search engines today.

The shared weakness is sparsity and size.

• Sparse means mostly zeros. A sparse vector representation is one where most entries are 0. A TF-IDF vector for one document has a non-zero only for the handful of words that actually appear in that document; the other tens of thousands of slots sit empty.
• Size means the vocabulary. Each vector's length equals $V$, the number of unique words in the corpus. A training corpus with 100,000 unique words gives every word a vector with 100,000 entries, almost all zero.

Imagine storing 100,000 numbers per word just to hold a handful of useful values — and then computing dot products over those mostly-empty lists, millions of times. Long vectors full of zeros are a real computational burden. They also waste their dimensions: slot 47,213 of the vector means "one specific word occurred", which tells you nothing about meaning.

That burden is the motivating question of this whole session: can we improve this vector format?

3.1.3 Dense Vector Representation

The improvement is the dense vector representation: a short vector where most entries are non-zero. Instead of one slot per vocabulary word, a dense vector has a fixed small number of features $D$ — and the feature dimension is much smaller than the vocabulary size:

$$D\ll V$$

$V$ (the vocabulary size) is the number of unique words in the training corpus; $D$ (the embedding dimension) is the length of the dense vector you choose. The word2vec family — skip-gram, skip-gram with negative sampling (SGNS), CBOW — and also GloVe, all produce dense vectors. They come out of the neural language model line of research: the vectors are learned by a small network instead of counted from a table.

Two design points about $D$ matter:

• $D$ must stay well below $V$. If $D$ were as large as $V$, the dense format saves nothing — you would be back to one-hot-sized vectors with no compression benefit.
• $D$ should not be enormous either. Typical values: 300 is the commonly recommended starting point; GPT-2 uses 768, GPT-4 about 1024, and reported sizes reach about 4000, rarely beyond 5000. Bigger $D$ costs more memory and compute for diminishing returns.

3.1.4 Static Versus Contextual Embeddings

There are two families of dense embeddings, and the difference is when the vector gets built.

Static word embeddings (word2vec, GloVe) do not depend on context. Each word gets one vector, learned once from the training data, and it stays fixed forever after. The word "bank" gets the same vector whether the sentence means a river bank or a financial bank. The vector is a compromise over every use of the word the training data ever saw.

Contextual word embeddings (BERT, GPT) are also dense vectors, but they are computed per occurrence. Inside a transformer, attention mixes each word with its actual surroundings, so "bank" sitting next to "river" produces one vector and "bank" sitting next to "loan" produces another. Contextual embedding gets full treatment after the mid-semester break.

The link between the two is direct, and that is why this session matters beyond itself: the intuition and several core ideas of static embedding algorithms carry over into contextual word embedding. Both are dense representations; the difference is only context dependence. The transformer's input is still an embedding table of exactly the kind word2vec builds.

Real-world placement. In next semester's NLP applications and conversational AI topics, practical systems — transformers, LLMs, agentic pipelines — usually run a hybrid: frequency-based sparse vectors (for exact keyword hits and fast first-pass retrieval) combined with dense vectors (for meaning-level matching). Neither family replaced the other; they stack.

3.2 Vocabulary, One-Hot Encoding, and the Embedding Matrix

3.2.1 From Training Data to Token IDs

Start from the training corpus — all the text you collected. The vocabulary is the set of unique words that appear in it, and its size is written $V$ (the running example in this course uses $V=100,000$). The vocabulary also includes one special entry: the unknown token (UNK), a bucket that catches any word not in the vocabulary.

Sort the vocabulary alphabetically (lexicographic order). Each word then gets a token ID — its position in that sorted list, numbered from 0.

3.2.2 One-Hot Representation

A one-hot vector is a vector of length $V$ where exactly one position is 1 and every other position is 0. "One hot" is literal: one position is hot (1), the rest are cold (0).

For a word with token ID $j$, the one-hot vector $o$ of shape $1\times V$ has:

$$o_k=\{\begin{array}{ll}1& \text{if }k=j\\ 0& \text{otherwise}\end{array}$$

That is the whole formula — one non-zero entry, at the word's own slot.

3.2.3 Why One-Hot Fails

One-hot is a perfect identifier and a terrible representation. Three failures stand out:

1. No similarity. Any two distinct one-hot vectors have dot product 0 — "apple" and "mango" are exactly as different as "apple" and "xylophone". All pairs of distinct words sit the same distance apart. Orthogonality is fine for an index, useless for meaning.
2. No context knowledge. A one-hot vector for "orange" says nothing about "juice", even if the corpus pairs them constantly. The spelling of one word cannot tell you anything about another word.
3. No prediction. Because each word is an isolated axis, you cannot compute "which word comes next" from one-hot vectors — there is no shared coordinate system to make the comparison in.

On top of all that, one-hot keeps the exact problem we are trying to escape: it is the sparse format again, $V$ slots per word with almost all zeros. Every one-hot limitation pushes toward the same conclusion — we need dense vectors.

3.2.4 The Embedding Matrix: One-Hot Times a Weight Matrix

The mechanism that converts one-hot to dense is a weight matrix of size $V\times D$. Multiply the one-hot row by this matrix and out comes a dense $1\times D$ vector.

There is a neat geometric fact hidden in this multiplication: a one-hot vector picks out one row of the weight matrix. Multiplying by the zeroes contributes nothing; the only survivor is the row whose index is hot. So the embedding of a word is that row of the matrix.

$$o_{j}W=\text{row }j\text{ of }W$$

That is why, once the matrix is learned and stored, the matrix and the embeddings are the same thing. At run time you never multiply anything — you just look up row $j$. The one-hot is only a pedagogical stepping stone.

3.2.5 The Features Are Learned, Not Named

A key shift in perspective separates machine learning from deep learning:

• In machine learning, you do feature engineering: you invent features by hand and hand them to the model, which learns weights for them.
• In deep learning, the features themselves are learned from the training data through hidden nodes and interconnected neurons.

So what are the "features" in our dense vector? Nothing more than $F_1,F_2,\dots ,F_{300}$ — abstract slots in a numeric space. The names are immaterial; only the numbers matter.

3.3 Word2Vec and Self-Supervised Learning

3.3.1 What word2vec Is

word2vec is the conversion of words to vector format. It comes from a pioneering paper by Google — Mikolov et al. 2013, "Efficient Estimation of Word Representations in Vector Space". The idea is simple and elegant, which is itself a lesson: research does not always need fancy ideas; simple ideas can give very good solutions.

Two properties made word2vec famous:

• Speed. It is a very fast algorithm. The skip-gram with negative sampling (SGNS) variant became the most-used word embedding method in industry precisely because it trains fast on huge corpora.
• Portability. Today transformers and attention-based models dominate, but the input to a transformer is still a word embedding of the kind produced by skip-gram. Static embeddings did not die; they became the front door of the models that replaced them.

Real-world placement. Implementations of all the static embeddings are freely available online, including the embedding projector — an interactive tool where you can drag the vector space around and watch similar words huddle together. TF-IDF and word2vec need little compute: they train on a laptop, no GPU cluster or H100 servers required. That is a major practical advantage over training a transformer from scratch.

3.3.2 Self-Supervised Data Generation

Skip-gram training data needs no human labeler. Here is how the labels generate themselves, step by step:

Two practical points:

• The window adapts to corpus size. With a huge training corpus, a smaller window is fine — there are plenty of context words. With a small corpus, use a longer window so each target still collects enough context words.
• Each word plays both roles. Every word in the vocabulary gets two representations: one as a target word and one as a context word. A word can be a target in one example and a context word in another.

3.3.3 Supervised, Unsupervised, Self-Supervised — Terminology

The three terms differ only in where the labels come from:

The supervised row is the typical ML model you already know: features $X_1,X_2,X_3$ go in, parameters $w_1,w_2,w_3$ get learned, and the dot product against a threshold decides the class. Self-supervision changes only the label story — the model itself still trains the same way.

The learning algorithm then uses gradient descent on this self-supervised data to learn the weights of the model — and those weights become the embeddings. That closes the loop from section 3.2: the matrix you multiply a one-hot by is trained on data that labeled itself.

3.4 Skip-Gram Training Data: Context and Out-of-Context Words

3.4.1 The Context Window Around a Target

3.4.2 Negative Examples

Every pair whose context word lies inside the window is a positive example. For each positive example, sample $K$ negative examples — words that do not lie in the window of the target.

Why more negatives than positives? Because learning to say "no" is what shapes the space. The same idea appears in contrastive learning, used today when training transformers and visual language models: positive and negative examples, with more negatives to sharpen the contrast so the model learns the weights better.

Negative selection needs care:

• A negative word must not fall inside the target's plus/minus 2 window.
• If a word is a positive context for a target, it must not appear as a negative for that same pair.
• The same word can be a negative for a different target — negativity is judged per pair, not globally.
• Some noise still slips through the samples, which is why you take more training examples — quantity washes out the occasional bad sample.
• You sample randomly, not exhaustively: if "apple" occurs 1000 times across 10,000 articles, you may pick just 3 or 4 occurrences.

3.4.3 Each Word Has Two Embeddings

Every word in the vocabulary carries two vectors:

• a target embedding — used when the word acts as the target (center) word, and
• a context embedding — used when it acts as a context word.

SGNS learns both, and they remain distinct through training. At the end, the final vector used for a word is the sum of its two embeddings:

$$\text{final embedding}=\text{target embedding}+\text{context embedding}$$

3.5 The Classification Framework and Its Math

3.5.1 Word2Vec as Binary Classification

The training problem gets reframed as classification. Positive pairs (target + a word inside the window) form the positive class with label $t=1$; negative pairs (target + an out-of-context word) form the negative class with label $t=0$. Two classes only: context and out of context.

The training goal follows straight from the labels:

• maximize the similarity between the target and its context words, and
• minimize the similarity between the target and the out-of-context words.

Training the classifier learns the weights — and those weights are the embeddings. The classifier is a means to an end: we do not care about the classification task itself, only about the vectors the network builds while solving it.

Why frame it this way? The original skip-gram asked a 100,000-way question — "which word out of the whole vocabulary is the context?" — using a softmax over all $V$ words. Every single training step had to compute a dot product against the entire vocabulary, which is brutally expensive at $V=100,000$. Negative sampling swaps that for a far cheaper binary question: "is this specific pair real or fake?" That is the whole trick.

Any classification algorithm could be used — decision tree, SVM, naive Bayes — but skip-gram uses logistic regression, the same classifier from the ML course. That is convenient: everything below reuses the logistic machinery you already know.

3.5.2 Similarity and the Sigmoid Probability

To measure similarity we take the dot product of the two vectors. The dot product is not normalized, so raw values are hard to interpret — is 0.508 big? Small? That is why the dot product is pushed through the sigmoid function to get a probability (or we use cosine similarity for retrieval, where interpretability is less important).

The identity $1-\sigma (x)=\sigma (-x)$ is worth memorising — it says the sigmoid is symmetric around the point $(0,\frac{1}{2})$ in the sense that mirroring the input flips the output around one half. It is the step that lets the same function handle both classes.

Why sigmoid? Its derivative is elegant. Verify it once with the chain rule, because the result is used constantly:

$$\begin{array}{rlr}\sigma (z)& ={\left(1+e^{-z}\right)}^{-1}& \\ {\sigma }^{\prime }(z)& =-{\left(1+e^{-z}\right)}^{-2}\cdot \left(-e^{-z}\right)& \text{(chain rule on the outer power)}\\ & =\frac{e^{-z}}{{\left(1+e^{-z}\right)}^2}& \\ & =\frac{1}{1+e^{-z}}\cdot \frac{e^{-z}}{1+e^{-z}}& \text{(split the square into two factors)}\\ & =\sigma (z)\left(1-\sigma (z)\right)& \text{(second factor is }1-\sigma \text{)}\end{array}$$

$${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$$

The derivative of the sigmoid is just "sigmoid times one minus sigmoid" — no exponentials left in the expression, which makes gradient math cheap. That nice property is why sigmoid shows up in feedforward networks, transformers, and softmax-style variants across the field.

Visual intuition. Plot $z$ on the horizontal axis and $\sigma (z)$ on the vertical: the curve is an S. It hugs 0 on the far left, rises through exactly $0.5$ at $z=0$, and hugs 1 on the far right, never quite touching either end. Landmarks: at $z=0.508$ the curve reads $0.62$; at $z=-0.508$ it reads $1-0.62=0.38$ — mirror inputs give mirror distances from the ends. The takeaway: the sigmoid is a soft switch — large positive dot products read as "same context" (near 1), large negative ones as "out of context" (near 0), and everything in between is graded smoothly.

3.5.3 Multiple Context Words: Product Becomes Sum

With $L$ context words $c_1,\dots ,c_L$, the probability of the whole set being positive is the product of the individual probabilities:

$$P(+\mid w,c_1,\dots ,c_L)=\prod _{l=1}^L\sigma (c_l\cdot w)$$

Products of small probabilities shrink fast: $0.005\times 0.005=0.000025$, and ten such terms would be unreadable — numbers that small also play badly with floating-point arithmetic on a computer. Probability theory has a standard fix: take the log. The log converts the product into a summation:

$$\log \prod _{l=1}^L{P}_l=\sum _{l=1}^L\log P_l$$

3.5.4 The SGNS Loss and Its Gradients

For one target word $w$, one positive context word $c_{pos}$, and $K$ negative words $c_{neg,1},\dots ,c_{neg,K}$, the loss is:

$$L(w,c_{pos},c_{neg})=-\log \sigma (c_{pos}\cdot w)-\sum _{k=1}^K\log \sigma (-c_{neg,k}\cdot w)$$

Each term is the usual logistic log loss. With labels $t=1$ for the positive and $t=0$ for each negative, a single term reads:

$$-[t\log \sigma (c\cdot w)+(1-t)\log (1-\sigma (c\cdot w))]$$

Put $t=1$ and the bracket keeps only $-\log \sigma (c\cdot w)$ — the positive term. Put $t=0$ and it keeps $-\log (1-\sigma (c\cdot w))=-\log \sigma (-c\cdot w)$ — the negative term. So the SGNS loss is logistic regression's cross-entropy written out for one positive plus $K$ negatives. Minimizing it raises the positive probability and lowers the negative probabilities, exactly the goal from 3.5.1.

Deriving the gradient with respect to the target $w$. Use the chain rule one term at a time. For a term of the form $-\log \sigma (x)$ with $x=c\cdot w$:

$$\frac{d}{dx}[-\log \sigma (x)]=\sigma (x)-1\text{because}\frac{d}{dx}\log \sigma (x)=1-\sigma (x)$$

and $dx/dw=c$. For the positive term, $x=c_{pos}\cdot w$. For a negative term, $x=-c_{neg,k}\cdot w$, so $dx/dw=-c_{neg,k}$, and the minus sign flips the error back:

$$\begin{array}{rlr}\frac{\partial L}{\partial w}& =(\sigma (c_{pos}\cdot w)-1)c_{pos}+\sum _{k=1}^K(1-\sigma (-c_{neg,k}\cdot w))c_{neg,k}& \text{(chain rule, all }K+1\text{ terms)}\\ & =(\sigma (c_{pos}\cdot w)-1)c_{pos}+\sum _{k=1}^K\sigma (c_{neg,k}\cdot w)c_{neg,k}& \text{(identity }1-\sigma (-x)=\sigma (x)\text{)}\end{array}$$

In words: the positive term is sigmoid minus its label 1 times the positive context vector; each negative term is sigmoid minus its label 0 times that negative vector; add all the vectors. The unified pattern: (predicted probability minus true label) times the other vector.

The same chain rule gives the derivatives with respect to the context vectors — now the inner derivative of $x$ is $w$ itself, so $w$ multiplies the error:

$$\frac{\partial L}{\partial c_{pos}}=(\sigma (c_{pos}\cdot w)-1)w\frac{\partial L}{\partial c_{neg,k}}=\sigma (c_{neg,k}\cdot w)w$$

The pattern to remember: differentiating with respect to the target pulls in the context vector as the multiplier; differentiating with respect to a context word pulls in the target vector $w$, because $w$ appears in both the positive and negative terms of the loss.

The update rule is plain gradient descent — new weight = old weight minus $\eta$ times the derivative of the loss:

$$w_{new}=w_{old}-\eta \frac{\partial L}{\partial w},c_{new}=c_{old}-\eta \frac{\partial L}{\partial c}$$

The full derivation of these gradients appears in the appendix of the Jurafsky and Martin textbook. The exam does not require deriving the loss function itself.

3.5.5 Gradient Descent Variants and the Learning Rate

A quick recap of the SGD variants from the ML course, applied to the 12 training examples from the apricot example:

• Batch gradient descent: update the weights after all training examples (after all 12).
• Mini-batch: divide into chunks (say, two chunks of 6) and update after each chunk.
• Stochastic gradient descent: update after every single training example.

SGNS uses stochastic gradient descent: update after every example, where one example = one positive pair together with its $K$ negative pairs.

The learning rate $\eta$ is the step size — how fast you go down the loss curve. If $\eta$ is very large, you move fast and can overshoot and miss the minimum; if very small, convergence takes very long. Typical values are 0.05 or 0.01. At the minimum of the loss curve the derivative equals 0 — the update stops changing the weights, which is the signature that you have arrived. Stopping criteria are the same as any gradient descent: stop when the loss reaches a minimum, or after a fixed number of iterations (say 10), or after going through the training examples (say 12).

3.6 SGNS Worked Example: Ned Stark, One Full Iteration

3.6.1 Setup, Embedding Dimensions, and Initialization

Toy corpus: "Ned Stark the honorable man" — five words after stop-word removal. The vocabulary has 5 words; every word gets a target embedding and a context embedding of dimension

$$D=3$$

All vectors start from random values. Initialization guidelines: values between $-0.5$ and $+0.5$, avoiding zeros and avoiding too-high values. (Small random values keep the dot products tame at the start, so the sigmoids begin near $0.5$ where the gradient is largest — the network learns fastest from a neutral start.)

The slide table for this iteration — target word ned ($w$), positive context stark ($t=1$), and three negative context words pimples, zebra, idle ($t=0$, so $K=3$):

One target, one positive, three negatives — this single set is one training example.

3.6.2 Forward Pass: Dot Products and Sigmoid

Take the dot product of the target vector with each context vector — multiply component-wise and add. For the positive pair:

$$c_{pos}\cdot w=(0.500)(-0.018)+(0.518)(0.500)+(0.518)(0.498)=-0.009+0.259+0.258=0.508$$

Then push it through the sigmoid:

$$\sigma (0.508)=\frac{1}{1+e^{-0.508}}=\frac{1}{1+0.6017}=0.62$$

A note on the numbers read aloud in class: the lecture quotes "you get this 0.55 value" for the positive sigmoid, while the slide's dot product $0.508$ gives $\sigma (0.508)=0.62$ — the two differ because the slide table rounds its own entries. When you solve a problem, the table you are given is the authority: compute each sigmoid from the dot products in your table and carry one consistent value through every later step. The worked computation here uses $0.62$ so every number lines up end to end.

For the negatives, use the negative formula — the same dot product but with $1/(1+e^{+c\cdot w})$, which equals $1-\sigma (c\cdot w)$:

$$\sigma (-c_{neg,k}\cdot w)=\frac{1}{1+e^{c_{neg,k}\cdot w}}$$

The three negative probabilities should be 0, but the random initialization gives around 0.55 for each:

$$\begin{array}{rlr}{c}_{neg,1}\cdot w=(-0.500)(-0.018)+(0.300)(0.500)+(0.100)(0.498)& =0.209& \Rightarrow \sigma =0.55\\ c_{neg,2}\cdot w=(-0.400)(-0.018)+(0.200)(0.500)+(0.200)(0.498)& =0.207& \Rightarrow \sigma =0.55\\ c_{neg,3}\cdot w=(-0.300)(-0.018)+(0.300)(0.500)+(0.100)(0.498)& =0.205& \Rightarrow \sigma =0.55\end{array}$$

3.6.3 Updating the Target Embedding

The derivative with respect to the target $w$ (from 3.5.4):

$$\frac{\partial L}{\partial w}=(\sigma (c_{pos}\cdot w)-1)c_{pos}+\sum _{k=1}^3\sigma (c_{neg,k}\cdot w)c_{neg,k}$$

Compute each term — the positive error $-0.38$ times the stark vector, plus each negative sigmoid times its vector:

$$\begin{array}{rl}({\sigma }_{pos}-1)c_{pos}& =-0.38\times [0.500,0.518,0.518]=[-0.190,-0.197,-0.197]\\ {\sigma }_{neg,1}{c}_{neg,1}& =0.55\times [-0.500,0.300,0.100]=[-0.275,0.165,0.055]\\ {\sigma }_{neg,2}{c}_{neg,2}& =0.55\times [-0.400,0.200,0.200]=[-0.220,0.110,0.110]\\ {\sigma }_{neg,3}{c}_{neg,3}& =0.55\times [-0.300,0.300,0.100]=[-0.165,0.165,0.055]\end{array}$$

Add the four vectors:

$$\frac{\partial L}{\partial w}=[-0.850,0.243,0.023]$$

Then update with $\eta =0.05$ (the worked slide uses 0.05; 0.01 is the other typical value — the lecture's spoken "0.5 by default" is the slide's 0.05):

$$w_{new}=w_{old}-\eta \frac{\partial L}{\partial w}$$

$$\begin{array}{rl}{w}_{new}& =[-0.018,0.500,0.498]-0.05\times [-0.850,0.243,0.023]\\ & =[-0.018,0.500,0.498]-[0.0425,-0.0122,-0.0012]\\ & =[0.0245,0.4878,0.4968]\approx [0.02,0.49,0.50]\end{array}$$

The headline result matches the slide: the first component of ned moves from $-0.018$ to about $+0.02$. The old value was negative and the update pushes it positive — toward stark's positive first component. That is exactly the aim: the positive context embedding moves closer to the target, while the negative pulls drag ned away from pimples, zebra, and idle.

3.6.4 Updating the Context Embeddings

The same iteration also updates the context word vectors. For the positive context stark, reuse the error $-0.38$ from the table — but multiply it by the target vector $w$ (ned), not by the context vector:

$$\frac{\partial L}{\partial c_{pos}}=(\sigma (c_{pos}\cdot w)-1)w,c_{pos,new}=c_{pos,old}-\eta \frac{\partial L}{\partial c_{pos}}$$

$$c_{pos,new}=[0.500,0.518,0.518]-0.05\times (-0.38\times [-0.018,0.500,0.498])=[0.4997,0.5275,0.5275]$$

For each negative, use its own sigmoid from the table:

$$\frac{\partial L}{\partial c_{neg,k}}=\sigma (c_{neg,k}\cdot w)w,c_{neg,new}=c_{neg,old}-\eta \frac{\partial L}{\partial c_{neg,k}}$$

$$\begin{array}{rl}{c}_{neg,1,new}& =[-0.500,0.300,0.100]-0.05\times (0.55\times [-0.018,0.500,0.498])=[-0.4995,0.2863,0.0863]\\ c_{neg,2,new}& =[-0.400,0.200,0.200]-0.05\times (0.55\times [-0.018,0.500,0.498])=[-0.3995,0.1863,0.1863]\\ c_{neg,3,new}& =[-0.300,0.300,0.100]-0.05\times (0.55\times [-0.018,0.500,0.498])=[-0.2995,0.2863,0.0863]\end{array}$$

Each negative context word is updated individually with its own sigmoid. Geometrically the update is pure force: stark is pulled slightly toward ned (its error is negative, so $-\eta \cdot (\text{negative})$ adds a touch of ned to stark), and each negative is pushed away (positive error times the target vector, subtracted). Words that appear together attract; sampled words that do not appear together repel.

3.6.5 Iterations and Stopping

One pass through one positive pair and its negatives is one iteration — and since SGNS updates after every example, it is also one stochastic update. In practice you repeat over all training examples: stark will later act as target with its own context window, pimples as target with other negatives, and so on. Each word keeps a single target embedding and a single context embedding, but a word serving as context for many targets gets updated many times — and that is fine, that accumulation is the learning.

When to stop? Stop when the loss reaches a minimum, or after a fixed number of iterations, or after passing through the training examples. The gradient descent stopping logic applies unchanged.

3.7 Hyperparameters, OOV Words, and Practical Training Choices

3.7.1 The Context Window and K

The two main knobs are the context window $L$ (words on each side) and $K$ (negative samples per positive pair):

• Window $L$: plus/minus 2 in the worked examples, but plus/minus 4 or 5 works too. Small corpus → larger window, because you need more context words per target; huge corpus → smaller window is fine, since context is plentiful.
• $K$: 2 to 5 for larger datasets; the apricot example used $K=2$ (8 negatives from 4 positives, 12 examples total) and the ned example used $K=3$. More negatives generally give better learning but slow down training, since each negative costs a dot product and a vector update.

3.7.2 Out-of-Vocabulary Words

The OOV gap is a direct consequence of the static design: the vocabulary is frozen at training time, and the matrix has a row only for words in that vocabulary. Retraining is the only way to add a brand-new word's embedding.

3.7.3 Static Means Learned Once

The weight matrix is learned once from the training data and then stays fixed for every word. This is why these algorithms are called static word embeddings. Given the same corpus, the learned matrix is the same; a different training corpus yields a different matrix. There is no per-user or per-sentence computation — the vectors are precomputed and stored.

3.7.4 Choosing D and Initialization

• $D$ (embedding dimension) is a hyperparameter: 300 is the common recommendation, GPT-2 uses 768, GPT-4 about 1024, and typical practice keeps $D$ below 5000. $D$ must be smaller than $V$, otherwise there is no benefit over one-hot.
• Initialization: random values in $[-0.5,+0.5]$, no zeros, nothing too large. Small symmetric random values start every sigmoid near $0.5$ — the point of steepest gradient — so learning starts fast. A reference document with initialization guidelines was shared in class.

3.7.5 Final Embedding = Target Plus Context

After training, the vector used for a word is the sum of its target embedding and its context embedding:

$$\text{word embedding}=\text{target embedding}+\text{context embedding}$$

SGNS learns two sets of embeddings; the final representation adds them. This is a specific detail to remember — the lecture states it flat out: "the word embedding for each word is going to be the summation of the target and the context."

3.7.6 How Negatives Are Sampled: The Noise Distribution

A supporting detail from the reference material, useful for knowing why real implementations do not just pick uniformly: negative words are drawn from a noise distribution $P_n(w)$ rather than uniformly. Drawing from the raw frequency distribution would flood the samples with ultra-frequent words like "the" and "is", which teach the model little. The standard fix raises the unigram distribution $U(w)$ to the power $3/4$:

$$P_n(w)=\frac{[U(w){]}^{3/4}}{{\sum }_k[U(k){]}^{3/4}}$$

The $3/4$ power is a smoothing trick: it shrinks the gap between frequent and rare words, so rare words get sampled a bit more often and frequent words a bit less. ("$w$ to the three quarters" sits between the square root $\frac{1}{2}$ and the identity power 1 — milder than a square root, enough to rebalance.) In this course, unless a problem says otherwise, you can treat negatives as randomly drawn from outside the window; the $3/4$ detail is context, not exam arithmetic.

3.7.7 Subsampling Frequent Words

One more standard trick: subsampling. Very frequent words like "the" and "is" appear everywhere and carry little meaning — the pair (France, the) teaches almost nothing compared to (France, Paris). Implementations randomly discard such words before generating pairs, with probability

$$P(\text{discard }{w}_i)=1-\sqrt{\frac{t}{f(w_i)}}$$

where $t$ is a threshold (for example $10^{-5}$) and $f(w_i)$ is the word's frequency. Frequent words get dropped often, rare words almost never. This speeds up training and improves the embeddings of rarer words, because they stop being swamped by "the".

Exam Guidance Summary

• A numerical SGNS problem is very likely. The statement will give you: the target word, the positive context words, the negative context words, and the table of initial embeddings (with $D=2$, 3, or at most 4). Expect a numerical on skip-gram negative sampling — it may be tweaked, not exactly as shown in class.
• What you must compute: dot products, a sigmoid table (positive and negative rows), one SGD weight update for the target, and one update for each context word. Exam note: show your work in a table — it is easier to grade, and reusing the table rows in the derivative step saves you from recomputing sigmoids.
• The labels are given, not computed: $t=1$ for the positive context, $t=0$ for each negative. The sigmoid is the prediction; the difference between the two is the error that drives the update.
• Values stay simple: numbers like 0.1 and 0.01. The exam tests understanding of the concept, not calculation skill. No big numbers.
• You do not have to derive the loss function. Know the update formulas — $w_{new}=w_{old}-\eta \cdot \partial L/\partial w$ — and the derivative pattern: sigmoid minus label, times the other vector.
• Learning rate: typically 0.05 (or 0.01). Stopping criteria: minimum loss, or a fixed number of iterations.
• Remember: final word embedding = target embedding + context embedding.
• Theory background: the full content — skip-gram, TF-IDF, contextual embeddings — is in the Jurafsky and Martin textbook, and the derivation is in its appendix. Andrew Ng's tutorial video and slides cover the intuition.
• Practice: the instructor shares the Excel sheet with the correct worked calculations. Go through the sheet, the slides, and the recording again — practice is required, because this math is the exam's centrepiece.

Key Industry Applications

• SGNS was the industry standard. Skip-gram negative sampling was the most popular word embedding method in industry before transformer-based embeddings (BERT, GPT) took over — and it trained fast enough to run on enormous corpora.
• LLMs still begin with word vectors. GPT-2 uses 768-dimension word vectors and GPT-4 about 1024; every large language model consumes dense word embeddings as its input layer.
• Transformers build on static embeddings. Attention inside a transformer produces contextual embeddings, but the input to the transformer is still a static-style word embedding table of the kind this lecture describes.
• Hybrid retrieval is production practice. Real systems combine sparse retrieval (TF-IDF) with dense retrieval (word vectors); the sparse side also covers out-of-vocabulary words that the dense side cannot.
• Similarity search is a whole industry. Dense vectors let a user prompt be converted to vectors and matched against a stored vector data store using cosine similarity — the machinery behind modern search, recommendation, and retrieval-augmented generation (RAG) systems.
• Sentence embeddings power classification. Averaging or max-pooling word vectors builds sentence embeddings, which drive sentiment classification and other lightweight text classifiers without any deep model.
• Negative sampling became contrastive learning. The positive/negative pair idea is exactly the mechanism of contrastive learning used today in transformer training and visual language models.
• Static embeddings run anywhere. Tools like the interactive embedding projector (which lets you visually explore the vector space) run on a laptop without GPUs — no H100 servers needed.
• Named references: the Google word2vec paper (Mikolov et al. 2013, "Efficient Estimation of Word Representations in Vector Space"), Andrew Ng's tutorial, and the Jurafsky and Martin textbook (skip-gram, TF-IDF, contextual word embeddings).