Building Autoregressive Models

Autoregressive models learn the probability distribution of a data set, then sample from it to create new data. This document builds the idea from histograms up to MADE, WaveNet, and PixelCNN, with every formula, worked computation, question from class, and practical warning included.

5.1 From Histograms to Learned Density Models

5.1.1 Where Histograms Break Down

The previous session covered likelihood-based approaches to estimating a probability distribution function. You start with an a priori assumption about the form of the distribution, then find the parameters of that distribution which maximize the likelihood of the training data. The first model type was the histogram-based model.

A histogram models a one-dimensional distribution reasonably well, though it is not perfect, because it does not generalize well. Split the value range into bins (equal-width intervals along the axis), count how many training points fall into each bin, and divide by the total count. Each bar height is then an estimate of probability. If a new point lands in a tall bar, it is likely; in an empty region, unlikely. And once built, you can sample from it to generate new data: pick a bar at random with probability equal to its height, then pick a value inside that bar. That sampling ability is a nice gift.

The gift does not survive high dimension. Here is why, with numbers. One feature split into $B=20$ bins needs 20 counts. Ten features, each split into 20 bins, need $20^{10}=10,240,000,000,000$ cells — over ten trillion counts, and you have maybe a few thousand rows. Divide each axis into bins and most of the multi-dimensional space ends up empty: regions where no training point ever falls get mapped to nothing usable. Push this further and the counting explodes outright: a small binary image of $28\times 28$ pixels has $2^{784}$ possible configurations — about $10^{236}$, a number with more digits than there are atoms in the observable universe. No training set ever fills that space.

Think of a library that stores one shelf for every possible book of exactly 100 letters. With a 2-letter alphabet there are $2^{100}$ shelves. Your library owns a few thousand books, so almost every shelf sits empty, and a visitor asking for "the most typical book" learns nothing. A histogram in high dimension is that library: mostly empty shelves.

Picture the failure on a chart. Axes are two features, say pixel brightness at two spots. The surface is a spiky city skyline: a few tall towers where training points cluster, surrounded by flat plains of zero everywhere else. Now imagine that skyline needing one extra axis per feature. After ten axes, the "towers" are lonely points floating in an enormous dark void. The takeaway: counting cells cannot beat the void, so the model must instead learn to interpolate — to guess sensibly between observed points. That is what neural networks will do.

5.1.2 What a Practical Density Model Needs

Three requirements drive the design of every model in this topic:

1. Accuracy with a rich architecture. Many processing nodes, organized into many layers, can learn the probability distribution of the data reasonably accurately. Depth and width give the network room to represent complicated structure — remember from the autoencoder material how stacked encoders reconstructed images better than shallow ones.
2. Fast training. It should not take five days, or fifty days, to learn the model. The learning procedure must run at reasonable speed, because you will retrain often while tuning.
3. Efficient sampling. Once the model is learned, the final goal is generating data by sampling from the distribution, and that sampling process must be feasible and efficient. A perfect density you cannot draw from is a museum piece, not a generator.

No method available today does all of these things very well at once. Every family gives something up. That is why even older methods deserve study, and why one drawback matters: autoregressive models generate data sequentially, so sampling is slow — one variable after another, like a row of dominoes falling. They bring many other efficiencies in exchange: accurate densities, stable training, and compact models. Keep score of the trade-offs as each model appears in this lecture.

Real-world: when ChatGPT writes an answer or such a tool creates an image, the picture does not appear in one shot — it appears slowly, piece by piece, because sequential generation is built into the method. The same slow-reveal behavior shows up in AI image generators that paint top to bottom. Sequential sampling is not a bug you can patch away; it is the shape of the method itself.

5.2 The Autoregressive Idea

5.2.1 Factorizing a Joint Distribution with the Chain Rule

Let $\mathbf{x}$ be a training vector made up of $d$ features, so $\mathbf{x}=(x_1,x_2,\dots ,x_d)$, where $d$ is a scalar count of features. An autoregressive model assumes the features occur one after another in time: $x_1$ occurs, then $x_2$, then $x_3$, and onward.

Under that assumption the joint probability of all features splits into a product of conditionals. This split is not an approximation — it follows from the definition of conditional probability, which says $p(a,b)=p(a)p(b\mid a)$. Apply it twice and watch the pattern grow:

$$\begin{array}{rlr}p(x_1,x_2,x_3)& =p(x_1)p(x_2,x_3\mid x_1)& \text{(split off }{x}_1\text{)}\\ & =p(x_1)p(x_2\mid x_1)p(x_3\mid x_1,x_2)& \text{(split off }{x}_2\text{)}\end{array}$$

Repeating this for all $d$ features gives the chain-rule factorization:

$$p(\mathbf{x})=p(x_1)p(x_2\mid x_1)p(x_3\mid x_1,x_2)\cdots p(x_d\mid x_1,\dots ,x_{d-1})=\prod _{i=1}^{d}p(x_i\mid x_1,\dots ,x_{i-1})$$

Each factor $p(x_i\mid x_1,\dots ,x_{i-1})$ reads: "probability of feature $x_i$ given every earlier feature." Each conditional is a probability value in $[0,1]$. The key restriction is temporal dependence: $x_4$ may depend on $x_1,x_2,x_3$, but not on $x_5,x_7,x_8,x_9$ or anything in the future. That one-sided dependence is the core of autoregression — the name literally means "self-regressing," each part regressed on its own past. Two facts deserve emphasis. First, the factorization itself is exact: it is the general law of probability, with no approximation anywhere. Second, the ordering of who comes first is a choice the modeler makes — any fixed order gives a valid decomposition, a freedom MADE will exploit later.

5.2.2 Training as Likelihood Maximization

The factorization turns one impossible job (model a whole vector at once) into many small jobs (model each feature given its predecessors). Training then maximizes the probability the model assigns to the training data. Working with products directly is awkward — numbers multiply toward zero fast — so take the natural log first. Logs turn products into sums and never change where the maximum sits, because $\ln$ is a monotonically increasing function (it preserves order):

$$\begin{array}{rlr}\ln p(\mathbf{x})& =\ln [\prod _{i=1}^{d}p(x_i\mid x_1,\dots ,x_{i-1})]& \text{(log of the product)}\\ & =\sum _{i=1}^d\ln p(x_i\mid x_1,\dots ,x_{i-1})& \text{(log turns products into sums)}\end{array}$$

Training maximizes this log likelihood, or equivalently minimizes the negative log likelihood (NLL) — the same expression multiplied by minus one, so "highest score" becomes "lowest loss," which matches how optimizers work:

$$\text{NLL}=-\sum _{i=1}^d\ln p(x_i\mid x_1,\dots ,x_{i-1})$$

After training you want a small NLL. A quick sanity check of the scale: if the model thinks the observed data was very likely, each $\ln p$ term sits near zero (since probabilities are at most one) and the sum stays small; if the model assigns tiny probabilities like $10^{-4}$, each term is about $-9.2$ and the loss grows. Model selection uses the same yardstick: train two architectures on the same data, and the one reaching the smaller NLL wins, purely on prediction accuracy grounds. Every recurrent architecture reviewed below trains exactly this way: compute the loss over the entire sequence and the entire training set, adjust the internal parameters so the loss reduces, and repeat.

5.2.3 A Toy Two-Variable Model

Picture a training set with two features, $(x_1,x_2)$, holding about 10,000 data points. The goal is to estimate $p(x_1,x_2)$ via maximum likelihood. The chain rule splits the job exactly in two:

$$p(x_1,x_2)=p(x_1)p(x_2\mid x_1)$$

Assign each factor to the tool that handles it best:

• $p(x_1)$ is realized by a histogram. Its quality depends on bin choices, but in one dimension that is manageable — no empty-space explosion with a single axis.
• $p(x_2\mid x_1)$ is realized by a multilayer perceptron (MLP). Here $x_1$ is the input, and the output node encapsulates the probability of $x_2$.

Suppose $x_2$ can take 256 values. Use a softmax output layer with 256 nodes. Softmax takes the network's raw scores (called logits, $z_j$) and converts them into a proper probability table over all 256 outcomes. The output of softmax node $j$, with pre-activation scores $z_j$, is:

$$P(y=j)=\frac{e^{z_j}}{{\sum }_{k=1}^K{e}^{z_k}},K=256$$

Here $e^{z_j}$ makes every score positive and emphasizes bigger scores, and dividing by the sum of all exponentials makes the outputs add to one. That is the beauty of softmax: because each output divides its exponential by the total, the outputs always sum to one, which gives a naturally probabilistic interpretation.

This little two-variable construction is a genuine autoregressive model: a histogram plus one network, chained by the law of probability. Every architecture ahead — RNN, MADE, WaveNet, PixelCNN — is this same skeleton wearing richer clothes.

5.2.4 Why Generation Stays Sequential

Generation runs the arrows forward one variable at a time: draw $x_1$ from its learned distribution, feed it in, draw $x_2$ given $x_1$, and continue. Nothing about the training objective changes that — sampling is inherently a step-by-step process for autoregressive models, because step $i+1$ needs the value produced at step $i$. Sequential sampling feels slow, yet autoregressive models come with many other efficiencies in exchange: training can be highly optimized, the model stays compact, and the exact density lets you score any data point you like.

Real-world: the same machinery scales to the systems behind modern text and image tools. GPT-style assistants predict the next token given all previous tokens — literally $p(x_i\mid x_1,\dots ,x_{i-1})$ with words instead of pixels.

5.3 Recurrent Networks as Autoregressive Engines

Two building blocks from earlier coursework power everything ahead: the CNN, built for two-dimensional data such as images, and the RNN, built for one-dimensional sequential data. Recall from the autoencoder material how CNN encoders and decoders reconstructed data very accurately — more accurately than MLP-based autoencoders and definitely better than PCA. A third block, the plain MLP, returns when MADE arrives. Everything in this section is review; its purpose is to line up the tools so the new masking models make sense.

5.3.1 RNN, LSTM, and GRU Compared

An in-class pop quiz opened the comparison of the recurrent family members. The exchange went:

Here is why the vanilla RNN struggles, in one breath: during training, gradients are multiplied by the same recurrent weight matrix at every time step, so over many steps they shrink toward zero or blow up — the signal from step 1 barely reaches step 50. Gated cells fight this with an additive memory channel that lets information ride across steps without being repeatedly multiplied. That single mechanism difference explains most of the comparison table below.

When to pick which: default to GRU for speed, upgrade to LSTM only when sequences get really long.

5.3.2 Attention and Transformers: Parallelism Is the Big Deal

The quiz continued to attention and transformers.

Then the flip side:

A calibration on vanishing gradients followed. Saying "vanishing gradients are everywhere" is mathematically loose but practically fair. The problem exists in all these networks, yet its extent is worst in the vanilla RNN, milder with gating, and eliminated in the transformer design. Vanishing gradients also bridge to the next question — what basic mechanism trains any of these recurrent models? Compute the loss over the entire sequence and training set, adjust parameters to reduce it, and recognize this as likelihood maximization: maximize $\ln p(\text{predicted})$ against the expected targets, which is minimizing NLL. Section 5.2's objective never changed; only the engine computing the conditionals did.

5.3.3 Causal, Bidirectional, and Deep Variants

The standard RNN diagram shows an input at time $t$ feeding a hidden state, with three learned matrices usually labeled $U$, $V$, and $W$. In words: $U$ maps the current input into the hidden state, $W$ carries the hidden state from the previous step forward, and $V$ maps the hidden state to the output. These recurrent weights are learned as part of training by maximizing the log likelihood of predicted data — equivalently minimizing cross entropy over the entire sequence.

Variants differ in one key property:

• Unidirectional RNN. Every output depends only on whatever happened in the past. By definition this is autoregressive — the causal case.
• Bidirectional RNN. A forward pass and a backward (anti-causal) pass both feed the output, so information from the future propagates alongside the past. This is NOT an autoregressive model. It suits fill-in-the-gaps applications, where a missing datum is predicted from everything before and everything after it — think of a cloze test where the blank is surrounded by known words on both sides.
• Deep bidirectional RNN. Many stacked hidden layers improve the ability to predict missing data beyond a shallow bidirectional net — a general property of deep layers. Training such stacks is time consuming, and the gain only holds when enough training data exists; otherwise overfitting and other bad things arrive.

5.3.4 Inside the Cells: Gates in LSTM and GRU

Beyond the $U,V,W$ matrices of a vanilla RNN, LSTM trains extra gating machinery. A gate is a small network ending in a sigmoid $\sigma$, whose output lives between zero ("block completely") and one ("pass fully") — a dimmer switch rather than a plain wire. Tanh units generate the candidate signal that the gates modulate. The classic LSTM cell update shows both roles working together:

$$c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$$

where $c_t$ is the cell memory at step $t$, $f_t=\sigma (W_f[h_{t-1},x_t]+b_f)$ is the forget gate deciding how much old memory survives, $i_t$ is the input gate deciding how much new candidate ${\tilde{c}}_t=\tanh (W_c[h_{t-1},x_t]+b_c)$ enters, and $\odot$ means elementwise multiplication. Because the memory updates by addition of gated pieces instead of full reinvention each step, gradients can flow across many steps without vanishing — that is the "memory highway" from the quiz answer made concrete. All these trainable matrices are learned during the LSTM training process, which is more complex than RNN training; the reward is a much higher ability to carry context across long spans. GRU combines some gates into one, cutting parameter count so training gets easier.

5.3.5 Generating Handwritten Digits with an RNN

Real-world: this experiment is the classic demonstration that an autoregressive model trained on sequences of pixels learns a distribution good enough to synthesize new, plausible handwriting. Signature-verification systems and handwriting-based document pipelines build on exactly this ability to model stroke sequences.

5.4 MADE: Masked Autoencoder for Density Estimation

5.4.1 Motivation: Learn Every Conditional in One Shot

Everything so far was review; masking-based autoregressive models are the new content. First, the meaning of the mask: during training, a mask ensures you use only information that has happened in the past and none of the information that comes later.

MADE stands for Masked Autoencoder for Density Estimation. Recall what autoencoders did so far: learn features by driving reconstruction error down, learn to strip noise, learn sparse representations of dense data — with a variational autoencoder promised later for generation. MADE is an interesting variation: an autoencoder architecture bent toward estimating the density of the training data itself. Same skeleton — encoder, bottleneck, decoder — new job.

Why bother with a classical building block? Because understanding how ideas evolve matters. Great things do not appear overnight — as if an apple falls on the ground and Newton is born the same day. Ideas arrive step by step, and appreciating prior work is why basics come first. When you later meet fancier relatives of MADE, you will recognize the family traits instead of memorizing strangers.

The motivation in one contrast:

An RNN learns $p(x_1)$, then $p(x_2\mid x_1)$, then $p(x_3\mid x_1,x_2)$ sequentially, one batch after another. MADE learns all those conditional probabilities in parallel, in one shot, after a few iterations of standard autoencoder training. Once training completes, you can generate samples — and generation is still sequential, because the model is autoregressive — but the expensive part, learning every conditional, happened in parallel. Remember this asymmetry; it returns at the end of the section. Texts on normalizing flows describe the same trade: masked models evaluate or train everything at once, while sampling must walk the ordering step by step.

5.4.2 Architecture and Output Activations

The backbone is a plain MLP autoencoder. The running example uses three-dimensional input $(x_1,x_2,x_3)$, two hidden layers of four units each, and three output units. A standard autoencoder reconstructs data; by default it has no ability to calculate probabilities. MADE forces that ability through the output layer.

A review exchange pinned down the output activation choice:

The distinction deserves one more sentence. Softmax makes its outputs compete — they must sum to one across nodes, which suits "pick exactly one class." A binary vector has no such contest: $(1,0,1)$ is perfectly legal, so each output needs its own independent probability, which sigmoid provides node by node.

MADE adds one requirement: outputs must live between zero and one, because each output represents a probability — $p(x_2)$, or $p(x_3\mid x_1)$, and so on. A probability cannot be negative. Sigmoid outputs satisfy this, and the training objective treats each output as a probability estimate. Concretely: with binary data, output unit $j$ emits a number $q_j=\sigma (a_j)\in (0,1)$, read as "the model's current estimate of that factor's probability," and cross-entropy-style NLL scoring compares $q_j$ against the observed bit.

5.4.3 Masks: Cutting Connections with 0/1 Matrices

A mask is a matrix of zeros and ones multiplied elementwise against a weight matrix of the same shape. Where the mask holds 1, the connection survives; where it holds 0, the connection is eliminated — the weight might as well not exist, because multiplying any weight by zero gives zero. Mechanically, imposing masks on a fully connected autoencoder produces a partially connected network whose wiring enforces the autoregressive property.

Shapes in the running example:

• Input to first hidden layer: three input units, four hidden units — a 4-row by 3-column mask.
• Hidden to hidden: four units to four units — a 4 by 4 mask.
• Last hidden to output: four units down to three outputs — a 3 by 4 mask.

Rows correspond to destination units and columns to source units, so reading a row tells you exactly which incoming connections that unit keeps. In the example mask, one input fed only the first two hidden units; another fed all four; a third fed none at all — its column was zeros from top to bottom. That dead column looks alarming until the numbering scheme explains it.

5.4.4 The Feature Ordering Assumption

The numbering creates the autoregressive structure. Assume an ordering — in the running example, $x_2$ occurs first, at $t=1$; $x_3$ occurs next, at $t=2$; and $x_1$ occurs last, at $t=3$. Input nodes and output nodes carry labels 1, 2, 3 matching this assumed occurrence order: the input slot holding $x_2$ carries label 1, the slot holding $x_3$ carries label 2, and the slot holding $x_1$ carries label 3. The chain rule then reads, in occurrence order:

$$\begin{array}{rl}p(x_1,x_2,x_3)& =p(\text{first})p(\text{second}\mid \text{first})p(\text{third}\mid \text{first},\text{second})\\ & =p(x_2)p(x_3\mid x_2)p(x_1\mid x_2,x_3)\end{array}$$

Same law of probability as Section 5.2.1 — just a different walking order through the same house. With a training set of three-dimensional instances, $p(x_2)$ is easy: build a histogram of the middle coordinate. The hard parts are the conditionals, especially when the feature count grows large. MADE exists so that standard autoencoder training learns all of them in parallel, with the network wiring enforcing the ordering.

Three questions in class shaped this section:

5.4.5 Random Hidden Numbers and Connection Rules

Here is the numbering algorithm:

1. Number the input units 1 through $d$, where $d$ is the total number of features, according to the assumed occurrence order.
2. Assign every hidden unit in every hidden layer a random integer between 1 and $d-1$. With $d=3$, hidden units draw from $\{1,2\}$ — which is why a four-unit layer showed a pattern like 1, 2, 2, 2. Only the range is fixed; the exact draws differ run to run because they are random, and no specific draw sequence carries meaning. One pattern like 1, 2, 2, 2 is simply what the random draw produced on screen that day.
3. Number the output units 1 through $d$, matching the occurrence-order labels: the output labeled 1 computes $p(x_2)$, the output labeled 2 computes $p(x_3\mid x_2)$, and the output labeled 3 computes $p(x_1\mid x_2,x_3)$.

Why does the range stop at $d-1$? Two reasons. First, no output can ever use a hidden unit numbered $d$: an output labeled $m$ accepts sources numbered below $m\le d$, so a source numbered $d$ would sit permanently unemployed — capping at $d-1$ wastes nothing. Second, keeping every hidden number below $d$ guarantees no single hidden unit can collect information from all $d$ inputs, which would let future features leak into early predictions.

The connection rule turns these numbers into wiring:

Check the consequences for the first mask:

• Input 1 (holding $x_2$) connects to hidden units 1 and 2 — both satisfy $m\ge 1$.
• Input 2 (holding $x_3$) connects to hidden unit 2 only, since no hidden unit exceeds 2.
• Input 3 (holding $x_1$) connects nowhere, since every hidden number is below 3.

That last bullet is the dead column from the mask picture, fully explained. Following paths through the network confirms each output's dependence set. Trace the connections backward from each output: the output labeled 1 reaches no input, so its value conditions on nothing — it estimates $p(x_2)$ outright. The output labeled 2 reaches only the input carrying label 1, namely $x_2$. The output labeled 3 reaches the inputs labeled 1 and 2 — $x_2$ and $x_3$ — but never $x_1$. The colored connection traces shown in lecture — blue and green links highlighting exactly these paths — made each dependency visible. Do this trace yourself once on paper; it converts the whole scheme from mysterious to obvious-in-retrospect.

5.4.6 Type A and Type B Masks

Two mask flavors appeared:

• Type B. A node may connect to the node carrying its own number and all numbers before it. Every node sees itself and its past — the relaxed rule $m\ge n$ used between hidden layers above.
• Type A. A node connects only to strictly earlier numbers, excluding itself — the strict rule $n<m$.

The hidden layers use the relaxed, type-B-style rule. At the output layer, though, the walkthrough demands the strict flavor, and the dependency check shows why:

• Output 1 computes $p(x_2)$, an unconditional marginal. If it could touch even the input holding label 1, it would see the very value it is trying to predict — leakage. So output 1 touches no input at all.
• Output 2 computes $p(x_3\mid x_2)$. Under a self-inclusive rule it could reach the input holding label 2 — the value of $x_3$ itself — collapsing the conditional into a copy. It must stop at strictly earlier labels.
• Output 3 computes $p(x_1\mid x_2,x_3)$, so it should combine exactly the inputs labeled 1 and 2, never label 3.

Strictly-less-than masking at the output delivers precisely those three dependence sets; the relaxed rule would corrupt the first two. As a notation note: research papers package the same idea as two model variants called MADE-A and MADE-B, where A uses the strict comparison and B the inclusive one — identical meaning to the type A / type B language here.

5.4.7 The MADE Training Objective

Training maximizes the probability of the training data under the assumed ordering. The verbal statement: "I want to maximize ln of this plus ln of this plus ln of this — or put a minus in front and minimize the whole thing." Formally, start from the ordered factorization:

$$p(x_1,x_2,x_3)=p(x_2)p(x_3\mid x_2)p(x_1\mid x_2,x_3)$$

Take logs to turn the product into a sum, then negate to get a loss to minimize:

$$\text{NLL}=-[\ln p(x_2)+\ln p(x_3\mid x_2)+\ln p(x_1\mid x_2,x_3)]$$

Each log term is scored against one output unit: the first against output 1's estimate, the second against output 2's, the third against output 3's. Because the masks guarantee each output sees exactly the right conditioning variables, one forward pass scores all three factors at once — that is the parallel-learning promise made real. Gradient updates adjust the weights subject to the mask constraints until this quantity bottoms out.

What you own afterward: estimates of $p(x_2)$, of $p(x_3\mid x_2)$, and of $p(x_1\mid x_2,x_3)$ over the entire training data — and so, multiplied together, of the whole joint distribution. An autoencoder obeyed the feature ordering throughout, and density estimation fell out of ordinary reconstruction-style training.

5.4.8 Sampling New Data Step by Step

Learning the conditionals unlocks generation. The procedure walks the assumed ordering, filling one slot per pass:

Training learned the conditionals in parallel, but generation walks the ordering: first $x_2$, then $x_3$ given $x_2$, then $x_1$ given $x_2$ and $x_3$. The asymmetry — parallel learning, serial sampling — is the signature trade of masked autoregressive models.

5.4.9 Results on Binary MNIST

Trained on binary MNIST (zeros and ones, as before), MADE generates digit images through the sampling procedure above. The results look reasonable. Compare each generated digit with its nearest neighbors in the training set: the family resemblance holds.

Since MNIST pixels here are strictly zero or one, the competitor is a mixture of Bernoullis — weighted blends of "typical binary images." When intensity values varied between 0 and 255 in earlier machine learning coursework, the mixture components were Gaussians instead, because intensities are continuous. Match the component family to the data type: discrete bits take Bernoulli components, continuous intensities take Gaussians.

Real-world: mixture models remain strong baselines for tabular density estimation; knowing that a structured autoregressive network beats them by tens of NLL points calibrates when to reach for each.

5.4.10 Orderings, Ensembles, and the Capacity Limit

The factorization written as $p(x_2),p(x_3\mid x_2),p(x_1\mid x_2,x_3)$ is one choice among many. Any permutation of the features yields a valid chain-rule decomposition, and you can mix and match further. For tabular data with no natural ordering, random permutations work fine. Changing the ordering across training runs lets you learn an ensemble of MADE models — several networks, each with its own ordering — and averaging their density estimates sharpens the result, the way asking several experts and combining opinions beats asking one.

Capacity has a ceiling, though.

5.5 WaveNet: Dilated Causal Convolutions for Audio

5.5.1 Task and Motivation

WaveNet is a one-dimensional, temporal autoregressive model for generating audio. The input is a digitized speech signal — a sequence of numbers $x_i$, where this sample appears first, then that one, and so on, exactly like the waveform your voice produces. The goal is to learn the probability density of that input so new audio signals can be created. Make the setup arbitrarily complex and interesting, and the model can even synthesize someone singing.

Real-world: WaveNet is a very high-performing architecture, probably still used for speech-to-text at Google, where the speech signal is the input.

5.5.2 Tree Structure and the Linear Receptive Field

How can masking-style causality learn this distribution with convolutional machinery? WaveNet stacks layers in a tree-like structure. Each hidden node connects only to the input at its own time point and previous points. One unit sees $x_i$ and $x_{i-1}$, the next sees $x_{i-1}$ and $x_{i-2}$, and so on upward, until the top output predicts $p(x_{t+1})$ given everything in its past. Everything is causal — no future information leaks, exactly the autoregressive contract from Section 5.2.

Count the reach. Draw the connections upward and the picture is a binary tree lying on its side: each level doubles how far back parents sit, but because neighbors overlap heavily, the union grows slowly. With four levels the tree touches five time points (its own plus four past); in general a stack of $L$ layers reaches only $L+1$ time points. The receptive field size is limited, linear in the number of layers. A probability estimate built from such a narrow window of history will not be accurate — deciding how to pronounce a syllable may require context hundreds of samples back — so the achievable density estimation quality is capped.

5.5.3 Dilation Gives Exponential Reach

From the deep-learning coursework on convolutional filters — including transpose convolution — recall dilation. A dilation rate tells a convolution to skip inputs: with rate two, the kernel reads every second sample; with rate four, every fourth. Reference texts define it as inserting zeros between kernel weights — a kernel of size three with rate two touches positions $i-4,i-2,i$ — so the kernel spans a wide region while keeping few weights. Nothing about causality changes; the skips point backward only.

Apply dilation inside the causal stack and the picture changes. Give layer one rate 1, layer two rate 2, layer three rate 4, layer four rate 8 — the rate doubles layer by layer, the standard scheme WaveNet uses to grow its field fast and evenly. Now a dilated unit connects to its immediate neighbor and one further out; the next layer skips wider still; the next wider again.

Visual intuition: sketch the two stacks side by side. On the left, a skinny triangle climbing straight up, one new sample per layer. On the right, a wide fan spreading rapidly leftward, each layer stretching twice as far as the one below. Same height, wildly different footprint — the fan hears the whole phrase; the triangle hears a blink.

5.5.4 Training Deep Stacks: Residual Connections

Wide receptive fields demand many layers, and many layers invite the classic failure. Class worked through it Socratically:

Worth pausing on why the wrong answers were wrong. ReLU keeps activations alive but does nothing about gradients shrinking across forty multiplications. Dropout and regularization fight memorization of training data — a different disease. Vanishing gradients are a plumbing failure: the learning signal itself thins out on its way back through many layers. Residual blocks fix the plumbing by adding a shortcut wire around each block, so the identity path lets gradients bypass the transformations entirely and reach early layers intact. That is why the recipe absorbs one more ingredient: skip connections from ResNet let deep dilated stacks actually train.

5.5.5 Gated Activations Borrowed from LSTM

Recall the LSTM goal — use lots of past information, while controlling what passes and what gets forgotten — and its mechanism: gates. WaveNet adopts the same trick. Sigma-activation gates decide whether to stop some information or pass it; tanh branches carry the actual content signal that the gates moderate. Combining them produces a gated unit of the standard form:

$$z=\tanh (W_f\ast x)\odot \sigma (W_g\ast x)$$

where $\ast$ denotes the causal convolution, $W_f$ and $W_g$ are learned filter weights, $\odot$ elementwise multiplication, $\sigma$ the sigmoid gate, and $\tanh$ the content branch. Read it as two opinions fused: tanh proposes what the audio content could be; sigmoid votes, sample by sample, how much of that proposal gets through. The parallel is direct: LSTM used gates to manage memory across time; WaveNet uses gates to manage which parts of a long receptive field influence the current prediction. Long audio carries bursts, silence, and noise, and the gate learns when to listen hard and when to shrug.

5.5.6 The Full WaveNet Recipe and Sampling

Assemble everything into the procedural recipe:

1. Apply causal convolution so only past information enters.
2. Stack dilated convolutions with growing rates so a large receptive field forms from few layers.
3. Insert gated units — sigma gates plus tanh content — for information control.
4. Wire residual (skip) connections so the deep structure trains.
5. Finish with a softmax so outputs are probabilities between zero and one over the possible sample values.

Inputs and outputs: in goes a seed of audio samples; out comes, after training, a conditional distribution $p(x_{t+1}\mid x_1,\dots ,x_t)$ at every step. Once $p(x_2\mid x_1)$ and friends are learned, generation loops: seed a value, sample from the predicted distribution, feed the sample back, and continue — sequential audio synthesis from a learned density. Complexity-wise, generation costs one network pass per output sample, which is why long clips take noticeable time even on strong hardware; the parallel-training advantage from Section 5.3.2 belongs to transformers, not here.

5.5.7 Applying WaveNet Ideas to Images

For illustration, WaveNet ran on image data despite being one-dimensional by design: read the image row by row, flatten it into a sequence, and process line after line. Nothing stops you from replacing the CNN with an RNN structure on data processed this way.

The deeper challenge got airtime too:

One more reminder from the RNN recap transfers here: future information is used only for fill-in-the-gap tasks, never in causal autoregressive modeling.

5.6 PixelCNN and Gated PixelCNN

5.6.1 Raster Scan Ordering and Masked Convolutions

PixelCNN applies masked spatial convolution to image distributions. Pixels are the features, ordered $x_1$ through $x_{N^2}$ for an $N$ by $N$ image. One natural ordering scans left to right, row by row, top to bottom — exactly how old analog televisions illuminated screens along scan lines traced by cathode rays. History explains the layout: this is known as raster scan. Under that ordering:

$$p(\mathbf{x})=p(x_1)p(x_2\mid x_1)\cdots p(x_N\mid x_1,\dots ,x_{N-1})p(x_{N+1}\mid x_1,\dots ,x_N)\cdots =\prod _{i=1}^{N^2}p(x_i\mid x_1,\dots ,x_{i-1})$$

Every pixel conditions on all pixels before it — including the previous line's pixels when a new row begins, which is what lets vertical structure propagate at all.

The twist versus WaveNet is spatial. Instead of an ordinary convolution centered on a pixel, use a masked convolution that consults only pixels occurring earlier in the ordering. For a three by three kernel this means holding specific entries at zero: the entire bottom row of the kernel is zeroed (those pixels come later), the center entry is zeroed (a pixel cannot see itself), and within the middle row only the entries left of center survive (same row, earlier columns). The top row stays fully visible — everything above the current pixel already happened. Only preceding pixels flow through. Beyond that masked kernel, the training stack mirrors the WaveNet-style structure already discussed.

Visual intuition: imagine the kernel as a nine-cell window sliding over the image, with shutters painted over its bottom row and its own center cell. As it slides across each row, the window reads a growing strip of the past — everything above, plus the finished part of the current line — and nothing else.

The fundamental limit stands, as raised in the spatial-dependency discussion of Section 5.5.7: quality caps out, and generation proceeds by biased, step-by-step sampling of the learned distribution — one pixel predicted, sampled, appended, then the next.

5.6.2 The Blind Spot Problem

See the flaw before fixing it. Estimate the distribution at some pixel using a three by three masked kernel. The mask admits earlier pixels only, so pixels below-right of the target never contribute — they sit inside the kernel's footprint but come later in raster order. Stack another masked layer hoping to widen the view, and the situation fails to improve. The previous layer, focused on its own target, likewise ignored its own future neighbors; passing those blind corners upward just passes blindness along. Trace the reachable set carefully and some lower-corner regions end up invisible to the receptive field, permanently — like a security camera whose housing blocks part of its own view no matter how many identical cameras you stack behind it.

Consequence: the NLL quality achievable by standard PixelCNN stays low. The chain-rule factorization says every pixel may condition on all earlier ones; the wiring quietly delivers less.

5.6.3 How Gated PixelCNN Fixes It

Gated PixelCNN imports the gate idea from the LSTM literature — sigma gating decides what to forget or remember, tanh controls how much information flows — and attacks the blind spot structurally. Instead of one two-dimensional masked convolution, it runs two separate one-dimensional streams:

• Vertical convolution. Computed over the whole image with padding arranged so the activation at row $i$ depends on inputs up to row $i-1$ — everything strictly above, nothing beside or below. Follow any vertical-stack unit through its inputs: all information from previous rows, no blind spot.
• Horizontal convolution. Sees the current row's past pixels and also consumes the gathered vertical-stack results, merging the two views.

Why does splitting kill the hole? Because "everything above" and "earlier on this row" together cover exactly the full causal past — the two streams partition the raster-scan predecessors between them without gaps. A single two-dimensional kernel cannot express that coverage cleanly; two stacked one-dimensional passes can. Splitting the masked operation into vertical and horizontal steps lets every target pixel, in effect, use all the information that has happened — closing the hole plain PixelCNN left open. Gated ResNet blocks then supply depth with trainable gradients and finer information control, borrowing once more from the residual recipe of Section 5.5.4.

5.6.4 NLL Results

Next sessions continue the lineage — PixelCNN++ and further refinements — with recommended blog posts covering the details.

5.7 Beyond Images: Video, Diffusion Teasers, and Study Guidance

5.7.1 Video Generation and Diffusion Models

No single model wins every criterion — each family carries benefits and drawbacks, which is precisely why several get taught. Autoregressive models give exact probabilities but sample serially; diffusion models refine many candidates through many denoising rounds; each pays somewhere else. The area stays active, and today's trade-off table gets rewritten every few months.

5.7.2 Knowing the Probability of Your Own Samples

One underrated gift of autoregressive models: after generating a sample, the model can immediately score it — compute the probability of that particular generated data point under its own learned distribution. High probability signals the sample resembles your data set; low probability flags it as an outlier worth discarding or investigating. Few generative families offer that self-audit for free — a GAN, for example, never gives you a likelihood at all. Think of it as a built-in quality inspector standing next to the production line.

5.7.3 Business Value and the Case for Basics

A closing reflection aimed at future practitioners: you can create wonderful synthetic data — but what is the business value? Compute is not free. Learning the distribution, sampling it at scale, and serving the results all ride on infrastructure someone pays for. Whether that spend returns matching value to the company is the question that matters alongside the technology. From the technical standpoint the ideas remain fascinating; from the business standpoint, justify them with a concrete use before scaling them.

And a strategy note: the state of the art moves faster than any course can chase — by the time one system gets covered, something newer ships. Learning the basics deeply, then mapping them onto each new arrival, outlasts chasing headlines. MADE's mask trick, WaveNet's dilation, PixelCNN's ordering — these patterns reappear inside whatever ships next year wearing a new name.

5.7.4 Homework, Prompts, and Reading List

• Homework. Redesign the MADE architecture for 8-bit features ranging from 0 to 255 instead of binary inputs, and reason through what changes. Expect this question at the start of the next session. AI assistance is allowed; the exertion is the point.
• Prompt recipes. Ask an AI assistant to "explain the MADE architecture and math" and it produces step-by-step, easy-to-understand notes; the same works for "explain PixelCNN with architectural information and math". Pair the output with the recommended blog posts — and check the AI's math against this lecture, since generated notes can be confidently wrong.
• Reading. The original WaveNet paper is valuable but difficult to read; the accompanying blog posts cover the same ground accessibly. Blog posts on the PixelCNN family are likewise recommended before next class.

Exam Guidance Summary

• Expect the carried-over homework question: adapt MADE from one-bit binary features to 8-bit values between 0 and 255, and explain the architectural impact (output layer, conditioning, sampling). It opens the next session.
• Expect rapid-fire oral quizzes comparing RNN, LSTM, GRU, attention, and transformer trade-offs: vanishing gradients, memory horizons, parameter counts, and above all the sequential-versus-parallel training argument.
• Know the chain-rule factorization, the log-likelihood/NLL objective, and why "smaller NLL wins" serves as the model-selection rule across every architecture covered.
• Be ready to explain mask mechanics: random hidden numbering between 1 and $d-1$, the connect-if-$m\ge n$ rule, type A versus type B behavior, and why the output mask must be strict.
• Practice narrating the sampling procedures: MADE's Bernoulli draws in assumed order, and WaveNet/PixelCNN's sequential pixel loop.
• The take-home assignment practices learning features with the unsupervised techniques covered so far — treat it as applied rehearsal for this material.
• Carry-over discussion: whether backward-only masking limits image density quality — a considered comment earns credit next week. Anchor your answer in the three points from Section 5.5.7: exact factorization, weak long-range factors, and sampling as the end goal.

Key Industry Applications

• Large language models. GPT-1, GPT-2, GPT-3, and ChatGPT are autoregressive models; their token-by-token responses — and slowly appearing generated images — exhibit sequential sampling directly.
• Speech pipelines at Google. WaveNet-grade autoregressive audio modeling drives high-performing speech-to-text, taking raw digitized speech as input; the same lineage powers natural-sounding voice assistants.
• Video generation. Sora (OpenAI) and related Meta efforts scale autoregressive principles from 2D to 3D; diffusion models complement them with parallel-capable but multi-step generation.
• Handwriting and digit synthesis. Binary MNIST generation with RNNs and MADE shows learned-distribution sampling with nearest-neighbor sanity checks — the standard proving ground before production data.
• Density estimation for tabular data. MADE with randomized orderings and ensembles offers a strong alternative to mixture models, with the NLL comparison (about 86.64 versus roughly 140 on binary MNIST) quantifying the edge; fraud detection and risk scoring teams use exactly this kind of density model to flag unusual records.
• Sample auditing. Autoregressive scoring of freshly generated samples gives teams a free similarity check against the training distribution — useful wherever synthetic data feeds downstream systems such as augmentation pipelines or test-set simulation.