Energy-Based Models and Natural Language Processing Applications

15.1 Energy-Based Models: The Core Formulation

An energy-based model is a way to learn the probability distribution that generated a set of training data. As with every generative technique covered so far, we call the training data $x$. The data could be anything — for images, the pixel values are simply the dimensions of $x$, so an image with a million pixels is a point in a million-dimensional space. The model assigns a probability to every possible $x$ through a scoring function $f_{\theta }(x)$, where $\theta$ stands for the parameters of a neural network.

Think of $f_{\theta }(x)$ like a credit score, but for data points. A bank's scoring function takes an applicant and returns one number summarizing how "loan-worthy" they are; here the network takes a data point and returns one number saying how much the point looks like genuine training data. High score means plausible; low score means implausible.

15.1.1 Why the Exponential Shape

The defining equation of an energy-based model reads:

$$p_{\theta }(x)=\frac{\exp (f_{\theta }(x))}{Z_{\theta }}$$

Spoken aloud: "the probability of x is e raised to the power of f theta of x, divided by z theta." Each symbol has a job:

• $p_{\theta }(x)$ is the model probability of a data point $x$ — a non-negative number, and across all $x$ these values behave like true probabilities.
• $f_{\theta }(x):{\mathbb{R}}^D\to \mathbb{R}$ is the neural network output, called the energy up to sign. In physics-flavored texts the energy is defined as $E_{\theta }(x)=-f_{\theta }(x)$, so high probability corresponds to low energy; in this lecture we work directly with the score $f_{\theta }(x)$, where high score means high probability.
• $Z_{\theta }$ is the normalizing constant, also called the partition function. It is a number that rescales the exponentials so everything fits into valid probability range. Formally, for continuous data it is the integral of the unnormalized scores over the whole input space:

$$Z_{\theta }=\int \exp (f_{\theta }(u))du$$

and for discrete data (such as binary pixels) the same object is a sum over every possible configuration. This integral is what makes EBMs both powerful and painful, and Section 15.2 returns to that cost.

Why insist on the exponential shape? Two structural properties follow directly from it:

1. Positivity. Because $e$ raised to any power is always positive, $p_{\theta }(x)$ is automatically greater than or equal to zero — no matter what values the network produces. Even if the network outputs a wildly negative score like $f_{\theta }(x)=-1000$, then $\exp (-1000)$ is a tiny positive number, never a negative probability. The speaker's own restatement: "since the output is written as exp of f theta x, the p theta x values are always greater than or equal to zero, because e to the power of anything is greater than or equal to zero."
2. Normalizability. Dividing by the appropriate normalizing constant squeezes the values so they sum (or integrate) to one across the whole space, which is the defining property of a probability distribution.

15.1.2 The Flexibility That Defines EBMs

The key distinguishing feature of energy-based models, compared with every model discussed before them, is flexibility. Because $f_{\theta }$ is realized by a neural network, it can be arbitrarily complex in nature. You are not locked into a parametric family such as a Gaussian, whose density is forced into a specific bell shape; the network can carve out whatever shape the data demands — several islands of probability, long thin ridges, sharp spikes.

Picture the difference this way. Fitting a Gaussian to data is like insisting every landscape must be a single smooth hill; fitting an EBM is like letting the sculptor decide. The training process itself learns the distribution from one single requirement:

The flexibility is real but not free. The next section prices it honestly: sampling and likelihood evaluation become hard problems — yet a simple comparison trick rescues most practical uses.

In the broader field, this scoring view is the shared skeleton behind Boltzmann machines, contrastive divergence, and modern score-based diffusion models — LeCun and colleagues formalized energy-based learning in 2006 precisely to give all of these one mathematical home. Wherever you later see a model described as "learning an energy," you are looking at this equation wearing different clothes.

15.2 Strengths and Trade-offs of Energy-Based Models

15.2.1 Three Hard Problems

Extreme flexibility is attractive, but a bunch of not-so-nice properties come with it:

1. Sampling from $p_{\theta }$ is hard. Drawing a realistic data point out of the model is not a simple operation. For a Gaussian you have closed-form recipes; for $\exp (f_{\theta }(x))/Z_{\theta }$ with an arbitrary neural network inside, no such recipe exists, and Section 15.5 has to build one from statistical physics.
2. Evaluating and optimizing the likelihood $p_{\theta }(x)$ is hard because of the presence of $Z_{\theta }$ in the denominator. You cannot even report how probable a single training point is without knowing that number.
3. Computing $Z_{\theta }$ numerically becomes very hard whenever the dimensionality of $x$ is large — think of images, where the pixel count runs into the thousands or millions. The normalizing constant involves a sum (or integral) over every possible configuration of pixels. A single grayscale image with just 100 binary-valued pixels already forces $2^{100}$ terms — more than the number of seconds since the Big Bang — and real images have thousands to millions of pixels. The count grows exponentially with dimension, so brute force is hopeless.

These are the defining trade-offs of the family: maximum expressive freedom, paid for with intractable normalization.

15.2.2 Relative Probabilities Are Enough

Why bother with EBMs at all? Because many tasks do not require knowing the explicit value of $Z_{\theta }$. You only need to compare two points. When you compare $p_{\theta }(x)$ against $p_{\theta }(x^{\prime })$ for two points $x$ and $x^{\prime }$, the normalizing constants cancel:

$$\frac{p_{\theta }(x)}{p_{\theta }(x^{\prime })}=\frac{\exp (f_{\theta }(x))/Z_{\theta }}{\exp (f_{\theta }(x^{\prime }))/Z_{\theta }}=\exp (f_{\theta }(x)-f_{\theta }(x^{\prime }))$$

Spoken: "this is nothing but an exponential of f theta x minus f theta x prime." Walk through why the cancellation happens: both numerator and denominator contain exactly the same $Z_{\theta }$, because the partition function depends on the model — not on which point you are evaluating. Dividing the two fractions therefore divides the two exponentials and cancels the shared factor. You calculate the network output at both points and decide which one is more probable — no $Z_{\theta }$ evaluation needed anywhere.

This mirrors a familiar trick from classical machine learning: the naive Bayes classifier. There you compare the joint probability of the attributes given rival classes, and whatever sits in the denominator — the overall data distribution $P(\text{attributes})$ — is a constant across all the comparisons being made. Whichever class gives the higher value wins. The same analogous reasoning applies here: once the model is trained, any point that comes from the training data will be associated with a higher probability than points coming from elsewhere. That is the beauty of energy-based models.

15.2.3 Applications: Denoising and Anomaly Detection

Because comparisons are cheap, EBMs fit tasks built on relative probabilities — the ratio of probabilities matters more than the probabilities themselves.

Real-world: Anomaly detection. Feed a trained EBM a stream of points — for example, sensor readings from a factory line or network traffic records in a security operations center. An anomalous outlier will be deemed to have essentially no probability compared with all other points that belong to the training distribution, so outliers pop out immediately. Fraud detection teams exploit exactly this: transactions far outside the learned high-probability region get flagged before a human ever looks.

Real-world: Denoising. A corrupted input sits far from the high-probability region; the model can judge how "training-like" a repaired candidate is by direct comparison. Candidate repairs can be ranked against each other, and Section 15.3.1 shows the Ising model turning this idea into a full denoising procedure.

15.3 Basic Energy-Based Model Families

Before training methods, recall the basic EBM types introduced previously: the Ising model, product of experts, the Boltzmann machine, and its restricted cousin. This section is a guided tour of four stops on one family tree — all four are energy-based models, differing only in how they wire their compatibility terms together.

15.3.1 The Ising Model for Corrupted Images

In the Ising model (a name borrowed from statistical physics, where it describes interacting magnetic spins), the observed pixels are noisy pixels — call them $x_1,x_2,x_3,\dots$. Each observed pixel is a noise-corrupted version of the real data, but the real data is not known to us. What we have is only the corrupted observations. The model thinks in terms of a small neighborhood — for instance a 3 by 3 patch of pixels around each location.

The joint distribution of the true pixels $Y$ given the observed pixels $X$ is written as a particular function with two ingredients:

$$P(y\mid x)\propto \prod _i{\psi }_i(x_i,y_i)\prod _{(i,j)\in \mathcal{N}}{\psi }_{ij}(y_i,y_j)$$

Here ${\psi }_i$ and ${\psi }_{ij}$ are the compatibility functions you choose — each returns a non-negative number saying how compatible its two arguments are — and $\mathcal{N}$ denotes the set of neighboring pixel pairs. The symbol $\propto$ means "proportional to": the right side has the right shape, and a normalizing constant over all candidate clean images would turn it into an exact probability.

The formula deserves a full derivation rather than an announcement. It follows from two modeling assumptions plus Bayes' rule:

Both ingredients carry meaning you can feel. First, there is dependence between each noisy pixel and its underlying true pixel: $x_i$ and $y_i$ are tied together for all values of $i$, because the corrupted pixel you view depends on the $i$-th original pixel — and that original pixel is exactly what you want to recover by running the model. Second, neighboring pixels tend to hold similar values: $y_i$ and $y_j$ tend to be alike. Physically, imaging data shows strong correlation between nearby points; a normal image has a small number of edges, so away from edges the local correlation is quite high.

Your goal is to maximize this posterior probability with respect to the unknown $y_i$ values — search over candidate clean images and keep the one whose product of compatibility scores is largest. Doing so yields the recovered, denoised image. As a quick sanity check of the structure: if you deleted every smoothness term ${\psi }_{ij}$, the model would still trust each observed pixel individually but would produce speckly results, because nothing would encourage neighbors to agree; delete the data terms instead and the model would return an over-smoothed blur. Denoising quality lives in the balance between the two products.

15.3.2 Product of Experts

A product of experts means you have multiple different trained models — the "experts" — and each expert makes its judgment independently. Pass any data point through all of them, then multiply the individual probability outputs together and divide by a fixed quantity:

$$p(x)=\frac{1}{Z}\prod _{m=1}^M{f}_m(x)$$

Here $f_1,\dots ,f_M$ are the $M$ experts, each a separate trained function returning a non-negative score for $x$, and $Z$ is the normalizer that makes the product behave like a probability. Once the experts are trained and fixed, that divisor remains constant, so you can again compare the resulting products between two points and decide which one is more likely:

$$\frac{p(x)}{p(x^{\prime })}=\prod _{m=1}^M\frac{f_m(x)}{f_m(x^{\prime })}$$

It is the same relative-comparison logic as Section 15.2.2, applied across several independent judges. The intuition for why multiplication helps: each expert can enforce one property — one expert might demand "edges look like edges," another "textures look like textures," another "colors are plausible" — and a data point passes only if it satisfies every expert simultaneously. Multiplication acts like a logical AND on probabilities: a point that pleases three experts at $0.9$ each scores $0.9^3\approx 0.73$, while a point that displeases even one expert at $0.01$ collapses to near zero no matter how good the others are.

15.3.3 Boltzmann Machines and Their Deep Version

The Boltzmann machine is another member of the energy-based family. The observed data occupies the bottom layer of variables — pixel values — and the layers above represent progressively higher-level features such as corner and edge detectors, moving toward genuinely semantic information as you go up.

Structurally, this general layered arrangement resembles a multilayer perceptron (MLP) strongly — the resemblance with the so-called deep Boltzmann machine is easy to see. One historical point deserves emphasis, because it explains the shape of the entire field:

15.3.4 Restricted Boltzmann Machines

RBMs — restricted Boltzmann machines — are energy-based models with latent variables. Those latent variables are nothing but hidden units: the visible variables $x$ live in the range zero to one, like pixel values, while the latent variables $z$ are units you have no access to — a hidden layer.

The joint distribution of $x$ and $z$ takes an exponential-family form whose terms reveal the restriction:

$$P_{\theta }(x,z)=\frac{1}{Z_{\theta }}\exp \left(\sum _{i,j}{w}_{ij}{x}_i{z}_j\right)$$

Read the structure carefully. The sum runs over every visible index $i$ and every hidden index $j$, pairing original values with hidden values through weights $w_{ij}$. In the joint probability distribution there are no terms like $x_i{x}_j$. If terms like $x_i{x}_j$ appeared, that would mean connections exist between visible units — between visible dimensions. Likewise there are no $z_i{z}_j$ terms: no connections among the hidden units either. What you are seeing is a linear combination of the original values paired, term by term, with hidden values through weights $w_{ij}$.

When trained on small training images, an RBM can generate lookalike samples — with the caveat that these are deliberately small-size demonstrations, so you recognize the scale limits of the demo rather than of the method.

These families were historically important stepping stones, but they share one practical weakness: training them well needs samples from the model itself. The next two sections build exactly that machinery.

15.4 Training an Energy-Based Model

Energy-based models deserve special care at training time, because the technique differs from everything learned so far. You have a set of training data points $x$, and the goal is to maximize the model probability of those points with respect to the parameters $\theta$:

$$\underset{\theta }{\max }\frac{\exp (f_{\theta }(x))}{Z_{\theta }}$$

Every other generative model in this course gave you a tractable loss you could hand to gradient descent directly. This one does not — and understanding why it does not is the fastest way to understand everything that follows.

15.4.1 The Numerator-Denominator Tug of War

Theta appears both in the numerator and in the denominator, and that creates a genuine tug of war. The tempting strategy is: increase the numerator, decrease the denominator. The problem is that $Z_{\theta }$ depends on the numerator too — it sums the exponential of $f_{\theta }$ over all possible configurations:

$$Z_{\theta }=\int \exp (f_{\theta }(u))du$$

Changing the thetas does not guarantee that your data point becomes relatively more likely compared with the rest, because as you increase the numerator's value at your data point, the denominator can rise as well, swallowing the gain.

Concretely: suppose raising $\theta$'s value doubles the score at your training point but also raises scores elsewhere in space enough to double the total integral. The ratio $\exp (f_{\theta }(x))/Z_{\theta }$ has not improved at all. Plain gradient ascent on this objective therefore gives no useful direction until we untangle the two terms — which taking logarithms accomplishes.

So the working goal is framed differently. After training, $f_{\theta }$ should be arranged such that for the known training points the score is high, and for the remaining points the score is low — high probability for training data, relatively low probability for everything else.

15.4.2 Contrastive Divergence

To bypass the evaluation of $Z_{\theta }$, a different technique is used: contrastive divergence. The idea runs as follows. At any training iteration you hold a particular, partially trained parameter value $\theta$. Using that current model, you generate some samples out of the model being trained. Then you adjust the parameters so that the training data becomes more likely than typical samples from the model. "Sample from the model" means: draw points using whatever temporary value of theta you currently have. The update direction is set by the comparison — push $f_{\theta }(x)$ for real training data above $f_{\theta }(\tilde{x})$ for the model-generated $\tilde{x}$.

This is a procedure rather than a formula, so it is worth laying out as one:

The parameter trajectory makes this concrete: you start at ${\theta }_0$, generate samples using ${\theta }_0$ itself, compare those samples' probabilities against the training data under ${\theta }_0$, and produce ${\theta }_1$. Then at ${\theta }_1$ you once again create fresh samples, demand their probabilities sit below the probability of the real training samples, and step to ${\theta }_2$. Step by step, the parameters move while fresh samples keep arriving from the moving model. The same contrastive-divergence concept returns when diffusion applications are discussed.

15.4.3 The Log-Likelihood Gradient

Take logs of the individual probability. The complicated ratio turns into something manageable:

$$\log p_{\theta }(x)=f_{\theta }(x)-\log Z_{\theta }$$

Spoken: "originally it was e to the power of f theta x over z theta; if you take the log of that, you have this minus this, and you want to maximise this." The log turns the division into subtraction because $\log (a/b)=\log a-\log b$. This quantity is the log likelihood of the data point under the model. Now differentiate it with respect to theta. Here is the complete derivation, with every algebraic move written out:

$${\nabla }_{\theta }\log p_{\theta }(x)={\nabla }_{\theta }{f}_{\theta }(x)-{\nabla }_{\theta }\log Z_{\theta }$$

The only unresolved piece is ${\nabla }_{\theta }\log Z_{\theta }$. Expand it with the chain rule, writing the partition function as an integral over all configurations $u$:

$${\nabla }_{\theta }\log Z_{\theta }=\frac{1}{Z_{\theta }}{\nabla }_{\theta }{Z}_{\theta }=\frac{1}{Z_{\theta }}{\nabla }_{\theta }\int \exp (f_{\theta }(u))du$$

The derivative moves inside the integral (the network is smooth and its exponential decays fast enough for this interchange to be valid):

$$\frac{1}{Z_{\theta }}\int \exp (f_{\theta }(u)){\nabla }_{\theta }{f}_{\theta }(u)du$$

Now notice what the factor multiplying the gradient inside the integral is: by definition, $\exp (f_{\theta }(u))/Z_{\theta }=p_{\theta }(u)$. So the whole expression is an expectation under the model's own distribution:

$$\frac{1}{Z_{\theta }}\int \exp (f_{\theta }(u)){\nabla }_{\theta }{f}_{\theta }(u)du={\mathbb{E}}_{u\sim p_{\theta }}[{\nabla }_{\theta }{f}_{\theta }(u)]$$

Assembling the pieces gives the displayed result:

$${\nabla }_{\theta }\log p_{\theta }(x)={\nabla }_{\theta }{f}_{\theta }(x)-{\mathbb{E}}_{\tilde{x}\sim p_{\theta }}[{\nabla }_{\theta }{f}_{\theta }(\tilde{x})]$$

A quick sanity pass on this expression before trusting it: both terms are gradients with respect to $\theta$, so they live in the same parameter space and can subtract — shapes agree. As a limiting case, if the model were already a perfect spike concentrated on $x$ itself, the model's own expected gradient would equal ${\nabla }_{\theta }{f}_{\theta }(x)$, the two terms would cancel, and the update would correctly go to zero — nothing left to learn.

To evaluate that second term you must draw samples from $\exp (f_{\theta }(x))/Z_{\theta }$, and that sampling is not an easy thing when the distribution has no closed form. That requirement is what makes EBM training unlike any other method in this course — and it motivates the machinery of the next section.

15.5 Sampling from an Energy-Based Model

15.5.1 How Sampling Timing Separates EBMs from VAEs and GANs

Sampling plays a fundamentally different role in each generative family, and the contrast is worth pinning down precisely.

• VAE: during training there is no role for generating data samples. You first train the encoder and learn the latent distribution. Only after training is over do you look at the latent space — take the mean value, take the covariance matrix — and create a sample as the mean vector plus epsilon times the covariance matrix, where epsilon is a small random sample drawn from a standard distribution. That sampled latent value is passed through the decoder to generate new data. Sampling is restricted to generation time.
• GAN: the only sampling that plays a role in training is drawing a small-dimensional random vector — on the order of one hundred entries, say a 100-dimensional noise vector — and sending it through the generator to fabricate fake images. A discriminator then learns to discriminate true training data from generator-made data. The noise vector is tiny compared with the images it produces: a hundred numbers in, a full image out.
• EBM: you actually generate samples from the model itself, for the purpose of training the model itself. That is what makes EBM training more complex than VAE or GAN training.

15.5.2 Markov Chain Monte Carlo

Since the model changes from one iteration to the next (${\theta }_0$, then ${\theta }_1$, then ${\theta }_2$, and so on), and since $\exp (f_{\theta }(x))/Z_{\theta }$ admits no closed-form sampler, a statistical process is recruited to produce the samples: Markov Chain Monte Carlo, abbreviated MCMC. Each step of the chain depends only on the current point — that is the Markov property, the same memoryless structure seen in Markov decision processes — and randomness enters at every step, hence Monte Carlo, named after the famous casino.

The original procedure works like this:

1. Initialize $x_0$ randomly (at or near zero).
2. Add a noise perturbation to the current point to propose a candidate $x^{\prime }$.
3. Evaluate the network's probability output at the candidate and at the current point.
4. If the candidate's probability value is greater than the current one, the new value $x_{t+1}$ becomes the candidate.
5. Otherwise, keep the current point with a certain acceptance probability — concretely, the standard rule accepts a downhill move with probability $\min \left(1,\exp (f_{\theta }(x^{\prime })-f_{\theta }(x_t))\right)$. Note that this ratio needs no $Z_{\theta }$: it is exactly the comparison trick of Section 15.2.2 applied inside the sampler.
6. Repeat these iterations many times to create your samples; then update the parameters based on the contrastive-divergence comparison and proceed to the next round.

The trouble: original MCMC is slow to converge. The chain may wander for thousands of steps before its visits reflect the true distribution, especially if $x_0$ starts far from any high-probability region. Every training iteration pays that cost again because the model — and hence the landscape it must explore — keeps changing. That bottleneck motivated a faster strategy rooted in statistical physics.

15.5.3 Langevin Dynamics and the Score Function

The alternative is Langevin sampling — again a principle coming out of statistical physics, originally used to model the movement of molecules depending on the temperature of the matter: molecules drift along forces and get jostled randomly by heat at the same time.

The modern recipe starts from a remarkable object: for any continuous distribution $p_{\theta }(x)$, suppose you can compute the gradient of the log probability with respect to $x$:

$${\nabla }_x\log p_{\theta }(x)$$

This is called the score function. The important part to notice: you are taking the gradient with respect to the random variable itself — the input point — not with respect to the parameters. Everywhere else in deep learning the gradient flows toward weights; here it flows toward the data coordinates, pointing from wherever you stand in input space toward where probability mass increases. Following this gradient climbs the probability landscape. And there is a computational gift hiding here: substituting the EBM form,

$${\nabla }_x\log p_{\theta }(x)={\nabla }_x{f}_{\theta }(x)-{\nabla }_x\log Z_{\theta }={\nabla }_x{f}_{\theta }(x)$$

because $Z_{\theta }$ does not depend on $x$ at all. The score of an EBM is just the network's input-gradient — no partition function anywhere.

Starting from a prior distribution $\pi (x)$ that is easy to sample from — it could be a Bernoulli distribution or a Gaussian distribution, whatever is convenient — you draw $x_0$. Then each Langevin step updates the point as:

$$x_{t+1}=x_t+ϵ{\nabla }_x\log p_{\theta }(x_t)+\sqrt{2ϵ}{z}_t,z_t\sim \mathcal{N}(0,I)$$

Walk through the three ingredients: $x_t$ is the current point; epsilon is a small step-size number scaling the gradient; and $z_t$ is nothing but a random Gaussian noise draw (with mean zero and identity covariance $I$, meaning independent standard normals across dimensions), scaled by the square root of $2ϵ$, which keeps the chain exploring instead of freezing at the nearest peak. Without the noise term the walk would climb deterministically to the closest maximum and stop; with it, the walk keeps drifting around the whole high-probability region, spending time in proportion to local probability — exactly what sampling means.

The guarantee: as epsilon tends to zero and t tends to infinity, $x_t$ approximates samples drawn from $p_{\theta }(x)$. While that limiting condition holds strictly only in the limit, practitioners observed something liberating — modifying the running random samples with this relationship for just a few iterations, t equal to 3, 4, or 5, is enough in practice. Modern energy-based models using Langevin sampling produce very high-quality outputs: actual face samples, and ImageNet samples generated from modern EBMs, look convincingly real.

15.5.4 Implementation Choices, the Energy Landscape, and Generation

Typically people implement the scoring network with CNN-based or ResNet-based architectures. One point distinguishes this usage from ordinary vision networks: you are not using the CNN to classify the data. Instead, the output of the CNN models the energy landscape of the data — a surface over input space where training data points sit at high probabilities and implausible points fall into low basins.

Picture the landscape as terrain seen from above: bright plateaus over realistic images, deep valleys over garbage. During training, based on the sampled points, the network learns to sculpt this landscape: training data points are pushed to high probability, and the samples generated by the partially trained model are pushed lower.

If this sounds familiar, it should. During GAN discriminator training you also wanted real data to yield a high discriminator output while generator-produced data yielded a low outcome. A similar principle and concept is applied in training energy-based models: the network is trained to predict high probability for training data compared with the samples it generates during the training process. The difference is what happens afterward — a GAN discards its discriminator at generation time, while an EBM reuses its landscape directly as the sampler's map.

Generation after training flips the sign of the game. You use the opposite of the Langevin sampling direction: start from a particular point and move toward regions of higher probability, making the generated data approach the quality-probability regime of the training data.

15.6 Where Generative Modeling Stands Today

15.6.1 The Full Model Family Tour

Stepping back, the course has now covered the major families of data-generating models: autoregressive models from the pre-midterm sessions; VAE, beta-VAE, and VQ-VAE — including hands-on assignment experience with those three; extensive time on GANs, starting from GAN basics and continuing through Wasserstein GAN, variants such as SN-GAN, and CycleGAN together with related image-to-image translation models; then diffusion models; and finally energy-based models.

That tour is worth keeping as a mental checklist, because each family answers the same question — how do I model $p(x)$? — with a different trick:

• Autoregressive: factor the joint into sequential conditionals and predict left to right.
• VAEs: encode into a latent space and decode from it, matching a prior.
• GANs: two networks in an adversarial game, generator against discriminator.
• Diffusion: gradually corrupt data with noise, then learn to reverse the corruption step by step.
• EBMs: score every input's plausibility with one network and sample via MCMC/Langevin.

15.6.2 Desirable Properties Nobody Has Fully Achieved

Why so many models? Each comes with pros and cons, and depending on the scenario some models become more useful than others. The desirable property list for a perfect generative model reads like this: it should not take too much memory; inferencing should be fast; the training process should be stable; and it should generate data very fast, with enough diversity. No available model delivers all of these at once.

What has been reached today: we can create very high-quality data — provided we have access to lots of memory to store the models and lots of processing power to train them. And often the best-quality models generate data sequentially rather than in parallel.

Real-world: consider the top-performing broad-purpose systems offered by OpenAI using GPT technology. Generate an image with such a system and it takes a while — because it generates the data sequentially, and internally it must be using some variant of the diffusion model in many cases. A diffusion model has to remove the noise in steps, which makes it slower by construction. The wait you feel while an image materializes is the sound of dozens of denoising steps running one after another.

15.6.3 Hallucination Control Through Human Ratings

One more open problem: generally speaking, you want assurance that generated data is not hallucinated — meaning the output really is associated with high probability under the true data distribution, not merely plausible-looking. An interesting industry pattern has emerged: some models now return multiple candidate results and ask users to rate which one is good and which is not. Based on those judgments they retrain or fine-tune the model. Plausibly they are using something akin to contrastive divergence — computing relative probabilities of two candidates from user judgments and adjusting the parameters so that community opinion reshapes the model's probability landscape. Human thumbs-up and thumbs-down play the role that model-generated negative samples played in Section 15.4.

15.7 Why Unsupervised Learning Matters for Language Data

15.7.1 Labels Are Scarce and Expensive

This observation reaches beyond natural language processing and vision, and it opened this course back in the first session: look at the available digital data around us — there is lots of it, but its categorization is poor. Data accompanied by training labels is particularly powerful and valuable for supervised training, no doubt. But given the volume of data created by digital devices, only a very small fraction will ever carry category labels. Unless you employ unsupervised learning techniques, you simply cannot fully monetize the amount of data around us. That is the motivation behind so many model types that learn from raw data alone.

In NLP the same story repeats. Plenty of data exists, but it may not be machine-learning-grade data — meaning very curated data with training labels attached. Building such data is very expensive, whether for academic purposes or model development; annotating a single sentence for sentiment or entity tags costs human minutes, and useful corpora need millions of sentences. Curated resources do exist — certain datasets have been carefully assembled over years — but the vast bulk is raw digital exhaust: text generated by cameras and people through social media and similar channels. The question is how to learn from that not-so-ideal data.

15.7.2 Self-Supervised and Semi-Supervised Framings

Over the years, techniques have been developed to squeeze signal out of non-machine-learning-grade data. They typically operate in an unsupervised framework — or, more appropriately, a self-supervised one, where the raw data itself manufactures the training targets.

The self-supervised trick deserves a name-check because it powers everything from word2vec to BERT: hide part of the input, ask the model to reconstruct it, and the answer key comes free. No human ever labeled anything — "the next word" or "the masked word" is the label, generated by the corpus itself.

Alongside sits semi-supervised learning: primarily unsupervised, with most of the data unlabeled, plus a small subset that does carry training labels, used to fine-tune what unsupervised learning discovered. The valuable supervised data improves the model; its small volume is compensated by the cheap unlabeled mass.

15.7.3 The Road Map for Language Models

The topics lined up: word2vec, then GloVe, then context vectors (CoVe), then ELMo, and possibly BERT and GPT — starting from basics. Some of this repeats material from dedicated NLP courses, but the payoff is relating those ideas to the unsupervised-learning perspective developed here. Each stop on the road map adds one capability:

• word2vec: compact vectors learned by predicting words from their neighbors.
• GloVe: global co-occurrence counts folded into the same objective.
• CoVe: context-dependent vectors via supervised translation pretraining.
• ELMo: deep stacked contextual representations.
• GPT/BERT: transformer backbones pretrained self-supervised at scale.

Before the modern models, one almost zero-order concept anchors everything: word-to-word co-occurrence statistics — which is exactly where the next section starts.

15.8 Word-to-Word Co-occurrence Matrices

15.8.1 Building the Matrix

Language is not random. Words are not sprinkled arbitrarily through documents; people organize them according to the inherent patterns of the language. So a productive first statistic is: given a large volume of text, capture which words occur with which other words. A word-to-word co-occurrence matrix tabulates, for every pair of words, how often they appear together.

The construction is mechanical. Pick a context definition — say, "within the same sentence" or "within a few words of each other." Then walk the corpus once: every time word $i$ occurs near word $j$, add one to cell $(i,j)$. Rows are indexed by words, columns by words again, and entry $C_{ij}$ answers "how many times did word $j$ appear near word $i$?" The matrix is symmetric when the relation "near" has no direction.

The canonical toy example uses a handful of related words: water, steam, ice, and hot. Water, steam, and ice are three forms of the same substance — one at normal temperature, one at low temperature, one at high temperature — and hot is the adjective floating among them.

15.8.2 Reading Semantics Off the Counts

Now read the table the way the lecturer walked through it. The word water dominates: it appears a large number of times in the corpus, with the largest counts in its row — the displayed entry for its strongest partner dwarfed everything else nearby. Steam co-occurs with water heavily — several hundred joint occurrences across the relevant cells. Ice registers fair, more modest counts. Ice and steam barely co-occur with each other — they occupy opposite temperature ends, one speaking of cold and the other of heat, so a typical corpus puts them together rarely.

The hot row sharpens the semantic picture. Hot and ice co-occur only 17 times — nearly opposites. Hot and steam co-occur 1813 times — tightly related through the notion of heat. Hot and water also co-occur frequently, because people talk about "hot water" constantly.

The conclusion: co-occurrence patterns convey semantic information. Words that are semantically related show up together; words that clash do not. Meaning, at this stage of the road map, is nothing more than the company a word keeps — a slogan usually attributed to the linguist J. R. Firth that this whole subfield quietly runs on.

15.8.3 The Storage Problem

One problem with acting on word co-occurrence directly: the matrix is enormous. If the vocabulary holds 1,000,000 words, representing the full co-occurrence structure takes about 4 terabytes of information. The arithmetic behind that figure is worth seeing once: a million-by-million matrix has $10^6\times 10^6=10^{12}$ entries; storing each as a 4-byte integer gives $4\times 10^{12}$ bytes, which is exactly 4 terabytes — before counting overhead, and most cells would be zero anyway since most word pairs never meet.

We want a succinct representation of words that preserves the relationship between words and the information they represent — without such huge volume. Several techniques exist for finding faithful compressed representations, and they connect to a broader self-supervised principle already met with RNNs:

Every model on the coming road map — word2vec, GloVe, and beyond — is a different answer to the storage problem this section just quantified: compress the terabyte-scale co-occurrence table into small dense vectors while keeping what the counts reveal.

15.9 One-Hot Embeddings and Their Limits

15.9.1 Constructing a One-Hot Vector

The first representation anyone tries is the one-hot embedding. For every word in the vocabulary, attach a zero-one vector: depending on the word's position in the dictionary, put a 1 in that position and zeros everywhere else.

Walk through a miniature vocabulary of four words:

• the first word encodes as $1000$;
• the second as $0100$;
• the third as $0010$;
• the fourth as $0001$.

In a realistic dictionary, a word starting the alphabet lands a 1 at the front followed by zeros; a word late in alphabetical order — say one beginning with "z" — carries its 1 at the very end. The total dimension of these embeddings equals the number of distinct words in the dictionary. A real vocabulary with 20,000 entries makes each word vector 20,000 numbers long, almost all of them zero.

15.9.2 Two Failures: Memory Waste and Meaning Blindness

Failure one: memory waste. Every vector is extremely sparse — one entry is 1, the rest are zeros — so nearly all stored bits carry nothing. For a million-word dictionary you would ship a million floats per word to communicate exactly one fact about it: its index.

Failure two is the deeper wound, and it deserves the warning label the lecturer gave it:

We want representations that are memory-efficient and preserve semantics — words closely related in meaning should hold similar representations. That demand spawned everything that follows.

As supporting background from the reference texts: one-hot inputs force a model to treat all tokens as mutually independent — every pair of one-hot vectors is orthogonal — so a model given only one-hots cannot share statistical strength between related words like "movie" and "film." Dense learned embeddings remove both defects at once, packing meaning into a few hundred floating-point dimensions where similar words land close together and interpretable directions emerge (the famous example: the vector from "man" to "woman" closely matches the vector from "king" to "queen").

15.10 N-gram Language Models

15.10.1 Unigram and Bigram Assumptions

A language model represents the language itself — it assigns probabilities to word sequences. The earliest family is the n-gram model, named for how many consecutive words each probability factor is allowed to look at ("n-gram" = a chunk of n words). With the unigram as the simplest member, the model asserts that the joint probability of a string of words factors into a product of individual word probabilities:

$$P(w_1,w_2,\dots ,w_n)=P(w_1)P(w_2)\cdots P(w_n)=\prod _{k=1}^{n}P(w_k)$$

Each $w_k$ is the k-th word of the sentence and each $P(w_k)$ is estimated from the corpus by simple counting: the word's occurrences divided by the total number of words. Given a corpus, you count how many times individual words occur, assign each word its probability, and score any candidate sentence by multiplying the individual word probabilities — a cheap verdict on whether the sentence is likely.

The strong assumption here is that all words are independent, and that assumption is why unigrams cannot work that well. Word order carries no information under this model: "dog bites man" and "man bites dog" receive identical scores, because multiplication does not care about the order of its factors.

The bigram model relaxes one step: the next word depends on the previous one. The joint probability becomes a chain of conditional probabilities:

$$P(w_1,w_2,\dots ,w_n)=P(w_1)P(w_2\mid w_1)P(w_3\mid w_2)\cdots P(w_n\mid w_{n-1})$$

Each conditional like $P(w_k\mid w_{k-1})$ is again a count ratio from the corpus: how often $w_k$ followed $w_{k-1}$, divided by how often $w_{k-1}$ appeared at all. Bigrams already restore some order sensitivity — $P(\text{bites}\mid \text{dog})$ can be large while $P(\text{dog}\mid \text{bites})$ is small — but they still see only one word back.

15.10.2 Relation to Neural Sequence Models

Place the n-gram assumptions on the same axis as the neural models, using "how many past words can influence the next prediction?" as the axis:

The narration cited a span of about sixty-odd past words for the RNN; treat that figure as an indicative remark about short-term memory rather than an exact specification — the honest statement is that an RNN's effective memory is limited and fades with distance. These are the classical stepping stones; newer models begin with word2vec.

15.11 Word2vec: Learning Embeddings by Prediction

Word2vec answers the storage problem from Section 15.8. Here "word" means a natural-language unit of information, and word2vec builds embeddings — compact vectors that support prediction. Two popular designs exist: the continuous bag of words model (CBOW) and the skip-gram model. They are mirror images of each other: CBOW predicts the center word from neighbors, skip-gram predicts neighbors from the center word.

15.11.1 Continuous Bag of Words: Neighbors Predict the Center

The CBOW model defines a window around a given word. With a window size of 5 — specifically 2 words into the past and 2 words into the future — the network receives $w_{n-2},w_{n-1},w_{n+1},w_{n+2}$ as inputs and calculates the probability of the central word $w_n$:

$$P(w_n\mid w_{n-2},w_{n-1},w_{n+1},w_{n+2})$$

Training on the corpus tunes the system so every word can be predicted well enough from its neighbors. CBOW does word prediction based on local context. Notice what "bag of words" means here: the four context words enter as an unordered set — their positions relative to each other do not matter, only that they surround the center.

15.11.2 Skip-gram: Center Predicts the Neighbors

The skip-gram model inverts the arrow. Given the n-th word, predict what happens in the neighborhood — the words at positions $n-2,n-1,n+1,n+2$ for a 5-window:

$$P(w_{n-2},w_{n-1},w_{n+1},w_{n+2}\mid w_n)$$

Window size is a dial. With a 5-window: predict 2 words to the left and 2 to the right. Shrink to a 3-window: given the central word, predict the immediate left and right words. Grow to a 7-window: predict the 3 preceding and the 3 following words. Both models carry a parameter $n$ — the window size — and both train by maximizing the log likelihood of these predicted probabilities over the corpus.

A practical reading of the dial: small windows keep predictions local (syntactic, part-of-speech-like information), while larger windows pull in topical, semantic similarity. Both models carry a parameter $n$ — the window size — and both train by maximizing the log likelihood of these predicted probabilities over the corpus.

15.11.3 Input Representations: A Live Question and Answer

Right before the architecture came a pointed comprehension check aimed at listeners who had taken an NLP course.

Note the shape of the correction: a first answer guessed "one by N, depending on the dimensions of the transform," and the resolution pinned the dimension to the vocabulary size — a detail worth remembering because the architecture below depends on it.

15.11.4 Training Objective and Negative Sampling

The supervision is manufactured by the corpus itself — entirely self-supervised or unsupervised in nature. For every word occurrence you know what surrounds it, so training pairs cost nothing to construct: slide the window across the corpus and each position hands you one prediction problem with its answer key attached.

Under CBOW, the goal is a transform that maximizes the probability of the observed data given the surrounding words. Mechanically: maximize the product of all the probabilities; take an $\ln$ so the product becomes a sum; prepend a minus sign to create a negative log likelihood; then minimize that with respect to all the parameters of the transform. That is how the transform gets learned:

$$\mathcal{L}=-\sum _{\text{windows}}\log p_{\theta }(\cdot )$$

One computational snag deserves its own spotlight:

To train effectively, people use negative sampling, which sidesteps the full-denominator computation by contrasting the true neighbor against a handful of sampled non-neighbors — instead of asking "how does this word compare against the entire vocabulary?", it asks "can the model tell the real neighbor apart from a few random impostors?" A handful of binary comparisons replaces one enormous normalization, which is why negative sampling made word2vec trainable at industrial scale.

15.11.5 Skip-gram Architecture: The Embedding Matrix and the Context Matrix

Here is the basic skip-gram architecture, traced end to end. Let $V$ denote the vocabulary size and $d$ the chosen embedding dimension.

1. Input. The one-hot vector of the i-th word in the vocabulary — call it $x_i$, with a single 1 at position i and zeros elsewhere; $x_i\in {\mathbb{R}}^V$.
2. Embedding matrix. Multiply by a matrix $W\in {\mathbb{R}}^{V\times d}$, called the embedding matrix. Multiplying a one-hot by this matrix simply fetches the corresponding row: $h=x_i^{\top }W\in {\mathbb{R}}^d$, yielding the hidden representation with $d$ entries. This row-fetch behavior is why one-hot inputs make sense despite their waste — they act as lookup indices into the matrix.
3. Context matrix. A second matrix $W^{\prime }\in {\mathbb{R}}^{d\times V}$ — the context matrix — maps the hidden vector toward predictions: $s=h^{\top }{W}^{\prime }\in {\mathbb{R}}^V$, one score per vocabulary word.
4. Softmax. The scores convert into probabilities of candidate neighbor words: $P(w_j\mid w_n)=\exp (s_j)/{\sum }_{k=1}^V\exp (s_k)$.

Training adjusts both $W$ and $W^{\prime }$ via gradient descent on the negative log likelihood to maximize the probability of the true neighboring words.

Once training finishes, given any particular word, whatever emerges at the hidden layer — that $d$-dimensional vector — is the word2vec representation of the word according to skip-gram.

15.11.6 CBOW Architecture: Average Then Predict

Continuous bag of words runs the same machinery in the opposite direction. Take the neighboring words surrounding a target — say four context words around the center — and combine them by taking the average of their embedding vectors. Multiply that average by the embedding matrix to get the hidden vector; multiplying by the context matrix then yields the output distribution over the vocabulary, and training learns the parameters of $W$ and $W^{\prime }$ to reconstruct the central word. In one displayed textbook figure the hidden representation had 10 units — purely an illustration width, not a prescribed setting; working systems use hundreds of dimensions.

After training, the recipe for deployment is symmetrical: take the neighboring words, average their vectors, multiply by the trained $W$ matrix — and you get the embedding of the central word.

Both models share one more property worth stating directly: they calculate word embeddings purely from neighboring information, through a loss-minimization process with no nonlinearity anywhere — just linear transformations and the softmax.

15.12 Attention for Sequence-to-Sequence Translation

15.12.1 Encoder-Decoder with a Context Vector

Next, a brief revisit of attention in machine translation — material touched in NLP as well as DNN contexts. Inputs are the source words $x_1$ through $x_s$; the target is the translated sentence $y_1,y_2,\dots$. The basic engine is an encoder-decoder. A forward-directional RNN walks the source sentence producing hidden states $h_1,h_2,\dots ,h_T$, where T tracks the sentence length in the source language — it tries to predict the future from the past. A bidirectional RNN additionally runs over the sentence in reverse so that each position knows what happened before and after it.

The forward hidden state and the backward hidden state at each position are multiplied by learned weights and combined into a context vector. That context vector feeds the output RNN, which predicts the words of the translated language. The consequence: translation does not proceed word-to-word in isolation. Each translated word is produced based on the current word together with the context vector computed by the bidirectional RNN — the weights decide what type of attention to provide to words from the past and present.

15.12.2 Asynchronous Processing and the Landmark System

An important structural fact: this sequence-to-sequence modelling for language translation is asynchronous. You process the entire source sentence first, creating the context vector, and only then decode the destination-language sentence from it.

Writing the combination precisely: at decoding step $t$, the context vector is a weighted average over all encoder states,

$$c_t=\sum _{i=1}^{T_x}{\alpha }_{t,i}{h}_i$$

where $T_x$ is the number of source words, $h_i$ is the (bidirectional) hidden state for source position i, and ${\alpha }_{t,i}$ is the attention weight telling the decoder how much source word i matters when producing target word t. The weights are computed and normalized so each column sums to one:

$${\alpha }_{t,i}=\frac{\exp (v^{\top }\tanh (W_s{s}_{t-1}+W_h{h}_i))}{{\sum }_{k=1}^{T_x}\exp (v^{\top }\tanh (W_s{s}_{t-1}+W_h{h}_k))}$$

Here $s_{t-1}$ is the decoder's previous state; $W_s$, $W_h$, and $v$ are learned matrices and vector whose shapes make the bracketed expression a single scalar per source position. The inner $\tanh$ plus dot product is called additive attention because the two transformed states are added before scoring.

This encoder-decoder architecture with additive attention — associated with Bahdanau — was first trained using a parallel corpus of English to German. Hold that thought: this attention machinery returns shortly as the ingredient that builds richer word representations than word2vec can offer.

15.13 The Word Sense Disambiguation Problem

15.13.1 One Embedding per Word Is Not Enough

Whether you use CBOW or skip-gram, each word receives one unique embedding — the example given: "World Bank" associates the word bank with a single unit, a unique embedding. But some very common words change meaning with context. Compare:

• "I went to the bank for withdrawing some money."
• "I went to a river bank."

The word bank means completely different things in these two sentences — a financial institution in one, the sloping edge of a waterway in the other. Assigning the same word embedding to bank in both cases conveys something wrong, because the semantics of the word depend on the entire contextual information of the sentence where it occurs.

The scale of the problem is easy to underestimate. Linguists call such words polysemous, and they are not rare curiosities: high-frequency English words are almost all polysemous ("run", "light", "cell", "mean"). Since embeddings power downstream tasks — sentiment analysis, entity recognition, question answering — a single fixed vector for "bank" injects noise into every one of those systems whenever context picks the other sense.

So word2vec models cannot perform word disambiguation — handling a same word that means different things depending on context. Fixing this requires embeddings that incorporate contextual information about the entire sentence, even the entire document. That requirement drives the next two models: GloVe enriches the static embedding itself with global statistics, while CoVe and its successors make the embedding context-dependent.

15.14 GloVe: Global Vectors

15.14.1 Combining Local and Global Information

GloVe — global vectors — extends word2vec in a specific direction. Word2vec exploits only neighborhood information: whatever words occur immediately in the past and future, within the window. But often the meaning of a word depends on what happens globally across the whole corpus — "astronomer" may rarely sit within five words of "telescope," yet corpus-wide their association is unmistakable.

GloVe, introduced in 2014, aims to combine both the local information and the global information available from co-occurrence matrices. (A note on origins: the session credited it to Google; the standard attribution is Stanford's NLP group, whose researchers published it at EMNLP 2014 — word2vec, by contrast, is the Google contribution from 2013. Either way the model itself is unchanged.) Technically, GloVe merges the count-based factorization of the co-occurrence matrix $C$ with the context-prediction flavor of the skip-gram-style models: it trains embeddings that predict logarithms of global co-occurrence counts, so every cell of the giant Section 15.8 table gets squeezed into the vectors rather than ignored.

15.14.2 The Ratio Intuition: Ice, Steam, Solid, Fashion

The key observation uses a quartet of words: suppose there are two words, ice and steam, and probe words such as solid and fashion. The word solid is related to ice a lot more strongly than to steam — ice is solid. So the relationship strength should be captured by a ratio of co-occurrence probabilities. Define $p({\tilde{w}}_k\mid w_i)$ as the probability that probe word ${\tilde{w}}_k$ appears in the context of word $w_i$, estimated from the co-occurrence matrix. Then:

The intuition crystallizes: what carries meaning is the ratio of the co-occurrence probabilities, not the probabilities themselves. Ratios isolate the differential affinity between word pairs and wash out global frequency quirks — a hyper-frequent word like "the" inflates every raw probability, but leaves well-chosen ratios nearly untouched.

15.14.3 From Ratio Condition to Loss Function

GloVe designs the embedding structure so that a function $F$ of the linear difference between two word vectors reproduces that ratio:

$$F\left((w_i-w_j{)}^{\top }{\tilde{w}}_k\right)=\frac{p({\tilde{w}}_k\mid w_i)}{p({\tilde{w}}_k\mid w_j)}$$

Here $w_i$ and $w_j$ are the embedding vectors of the two compared words, ${\tilde{w}}_k$ is the separate context-vector embedding of the probe word, and $(w_i-w_j{)}^{\top }{\tilde{w}}_k$ denotes a dot product. Where the co-occurrence of $w_i$ with $w_k$ greatly exceeds that of $w_j$ with $w_k$, the right side departs far from one, and the left side must follow.

The remaining design choices fall out with short algebra, none hidden. First, write the conditional probability as a normalized count, $p({\tilde{w}}_k\mid w_i)=C_{ik}/C_i$, where $C_{ij}$ is the co-occurrence count of words i and j and $C_i={\sum }_k{C}_{ik}$. Substituting into the ratio, the normalization constants cancel:

$$F\left((w_i-w_j{)}^{\top }{\tilde{w}}_k\right)=\frac{C_{ik}/C_i}{C_{jk}/C_j}\stackrel{\text{choose }F=\exp }{\to }\exp \left(w_i^{\top }{\tilde{w}}_k-w_j^{\top }{\tilde{w}}_k\right)=\frac{C_{ik}}{C_{jk}}$$

Choosing $F=\exp$ turns the left difference into a quotient, which must now equal the count quotient on the right. Taking logarithms splits it back into a subtraction:

$$w_i^{\top }{\tilde{w}}_k-\log C_{ik}=w_j^{\top }{\tilde{w}}_k-\log C_{jk}$$

Both sides must equal some constant independent of the free index; symmetrizing by swapping the roles of word and context, adding learnable bias terms $b_i,{\tilde{b}}_j$ to absorb the constants, and regressing the dot products directly onto $\log C_{ij}$ yields the training loss — minimize the weighted squared residuals over the whole matrix:

$$J=\sum _{i=1}^V\sum _{j=1}^{V}g(C_{ij}){\left(w_i^{\top }{\tilde{w}}_j+b_i+{\tilde{b}}_j-\log C_{ij}\right)}^2$$

Here $g(\cdot )$ is a fixed weighting function that stops rare pairs (noisy counts near zero) and ultra-frequent pairs (like "the") from dominating the sum; a standard choice caps its growth and rises gently for small counts.

The net effect: you go from a vocabulary-sized co-occurrence structure to a much smaller representation — $n$ is the dimensionality of the word vectors, like 300 — obtained by minimizing this whole function. Everything else in the expression consists of the one-hot-style indexing vectors and the counts from the co-occurrence matrix. Summing up the technique: GloVe effectively combines local information — captured by the ratio-of-probabilities side — with global information captured by the counting method, the co-occurrence matrix.

15.15 CoVe: Context Vectors

15.15.1 Supervised Pretraining on Translation

Word2vec and GloVe are two earlier solutions to representation learning; neither solves disambiguation. The CoVe model — context vector model — attacks it directly. CoVe takes as input the GloVe-encoded vectors — not the one-hot representations of words, but GloVe embeddings — and passes them through a bidirectional LSTM, learning attention as it trains on an English-to-German parallel machine-translation dataset.

Pause on the supervision split, because the session flagged it twice:

Inside the translation decoder, the outputs are calculated through a combination of LSTM, softmax, and tanh components — the German-side words emerge starting from English words as input, whose GloVe vectors pass through the attention network of Section 15.12.

15.15.2 The CoVe Representation and Task-Specific Heads

The CoVe representation of a word concatenates three things: the forward-directional BiLSTM hidden state, the backward-directional BiLSTM hidden state, and the GloVe vector of the word. Concatenation is the operative word — the context-aware LSTM outputs are glued onto the static GloVe embedding, so the final vector carries both "what this word means everywhere" (GloVe) and "what it means here" (BiLSTM states). If each hidden state has 300 entries and GloVe gives 300, the CoVe vector for one word is 900 numbers long, arranged as [forward | backward | GloVe].

Now disambiguation falls out for free. The phrase World Bank will be associated with a different word embedding depending on the sentence: "I went to a river bank" produces one CoVe representation, while "I went to the bank for withdrawing money" produces another — because the input sentences drive different hidden states in the forward and backward directions. Same input token, different surrounding sentence, different output vector. That is the beauty of the CoVe model.

The pipeline organizes into stages:

1. A generic feature stage: GloPlus BiLSTM states pretrained on translation — reusable across tasks.
2. A task-specific model built completely around the end task — sentiment analysis, part-of-speech tagging, or question answering — sitting on top of those features.

The generic part transfers; the head customizes. Empirically, GloVe plus CoVe performs significantly better than GloVe alone across various tasks — question answering, IMDb sentiment analysis, and two further benchmarks.

The evident limitation: because the contextual stage needs supervised translation pretraining — parallel corpora are themselves expensive labeled data — the general applicability of the scheme is somewhat constrained. This bottleneck is exactly what the next section's self-supervised giants remove.

15.16 Deep Contextualized Representations and GPT

15.16.1 Stacking LSTMs for Multiple Granularities

RNN-based language models served successfully for many tasks — language translation, audio to speech and speech to audio conversions, text generation — but performance plateaued. The next leap came from going deeper: instead of the single bidirectional LSTM layer that CoVe used (past-to-present and present-to-pass directions combined), people stacked multiple layers of LSTM networks.

Depth bought granularity: lower layers capture something very concrete — word order, part-of-speech patterns, local phrases — while higher layers capture more abstract information like syntax-wide structure and semantic roles. One representation stack therefore holds a hierarchy of linguistic features, with each layer building on the abstraction of the one beneath it.

15.16.2 GPT: Transformers Replace LSTMs

The key innovation inside today's popular GPT models: replace LSTM calculations with transformer calculations. Two properties make transformers attractive here:

GPT uses the standard transformer structure not once but 12 blocks stacked one after the other. Each block performs masked multi-head self-attention, followed by layer normalization, then a feed-forward network, and that composition repeats 12 times.

The mask is the load-bearing piece. In self-attention every position would normally see every other position; masking sets the scores for future positions to negative infinity before the softmax, so they receive zero weight after normalization. Concretely, when predicting token $t$, positions $t+1,t+2,\dots$ are invisible. GPT strictly refuses to use any information from the future — everything is captured from the past alone, making the model autoregressive, generating left to right: predict the next token from all previous ones, append it, repeat.

15.16.3 One Architecture, Many Tasks

A second pillar of GPT: everything rests on unsupervised learning. The full model is pretrained on large volumes of raw corpus; only for fine-tuning is a small amount of supervised, labeled data used. Contrast this with the CoVe pipeline, where frozen contextual features feed separately built task-specific models — GPT does not do that. It fine-tunes one architecture on whatever downstream data arrives, updating the same weights rather than bolting on new heads over frozen features.

The representation proves remarkably versatile across problem formats:

• Classification of a single input — feed the document, read off a label.
• Contiguity checking: during pretraining, two sentences are placed together, and the model learns from raw text alone to classify whether the pair occurred one after the other in the source or sit far apart.
• Entailment and similarity: two inputs sit side by side — a premise and a hypothesis — using start and delimiter markers; pretrained GPT detects entailment and detects similarity between two sentences.
• Question answering: the same backbone learns QA given the training format, with only a small supervised set for fine-tuning.

So GPT learns representations that let one architecture solve a broad variety of problems — all built from the multi-head attention and masked attention blocks covered in the transformer material.

15.17 BERT Preview: Bidirectional Pretraining

15.17.1 Masked Language Modeling

BERT, contemporary with GPT, takes the opposite directional bet: it uses a bidirectional transformer for language understanding. Where GPT applies masked self-attention to bar future information and generates sequentially in autoregressive fashion, BERT ingests information flowing in both the forward and the backward direction — every word may look left and right during training.

Its signature training scheme is the masked language model: given the corpus, it probabilistically determines a few words to hide — about 15% of words — and tries to estimate the masked values given everything else present.

Predicting the hidden middles from both sides is exactly the "predict what is occluded from the visible" self-supervised task from the taxonomy in Section 15.8 — the same principle that drove word2vec's windows, now applied with full bidirectional attention instead of a fixed neighborhood.

The trade-off worth carrying into the next session: because every position sees both directions, BERT is a superb understanding engine (classification, similarity, QA over given text) but not a natural generator — nothing in its training asked it to produce the next word from the past alone. The discussion of BERT resumes next session, alongside vision applications of the generative toolkit.

Exam Guidance Summary

This session's exam briefing, gathered in one place:

• Weighting: most questions relate to material covered after the midterm; only a few marks — around 20% or so — may come from pre-midterm topics. Calibrate revision time accordingly.
• Past papers: past years' question papers and their solutions have been shared; reviewing them is strongly encouraged as primary practice — question style repeats more than content does.
• Drill partner: ChatGPT is endorsed as a practice tool for working through the topics discussed — use it to generate extra problems and check your derivations line by line.
• Online quiz: the quiz is open — attempting it provides useful practice and revision ahead of the exam, with the better of two attempts counted.
• Final tips: tips for the finals will be shared in an upcoming session; watch for them.
• Permitted aid: a watermarked set of course material will be shared for printing and carrying into the exam room; this is expected to be the only permitted aid, but whatever the exam team announces is final — verify there before relying on it.
• Self-study boundary: the extra energy-based-model material beyond the core coverage is for your own knowledge; no specific questions will be asked from the parts not covered in session.
• Assignments: review your evaluated assignments promptly and request reevaluation where needed before course closure deadlines.

Key Industry Applications

Where the models of this lecture earn their keep:

• Anomaly detection: trained EBMs flag outliers as points with negligible probability relative to the training distribution — a direct industrial use of relative-probability comparison. Fraud teams, network-security monitoring, and industrial quality control all run on this pattern.
• Image denoising: the Ising-style formulation recovers true pixel values from noisy observations by maximizing the posterior over neighborhoods — the same mathematics behind medical imaging cleanup and low-light photography enhancement.
• Face and ImageNet synthesis: modern EBMs coupled with Langevin sampling generate convincing face samples and ImageNet-scale imagery, proving the sampling machinery scales beyond toy corpora.
• Large language assistants: OpenAI's GPT-style systems generate sequentially; image generation in such systems proceeds slowly through internal diffusion-like stepwise denoising — the visible wait is the denoising chain running.
• Human-feedback tuning: platforms presenting multiple generations for user rating and then fine-tuning mirror contrastive-divergence-style adjustments driven by judged relative probabilities — community thumbs replace model-sampled negatives.
• Machine translation: Bahdanau-style encoder-decoder models with additive attention, trained on English-to-German parallel corpora; the same machinery powers CoVe pretraining and every attention-based translator since.
• Text classification stacks: sentiment analysis (including the IMDb benchmark), part-of-speech tagging, and question answering built atop GloVe+CoVe or fine-tuned GPT/BERT-style representations.
• Embedding infrastructure: word2vec (CBOW and skip-gram) and GloVe remain the baseline industrial word representations, with negative sampling enabling efficient training at vocabulary scale — recommendation engines, search ranking, and ad-targeting systems all consume these vectors.