PixelCNN Variants and 1D Normalizing Flows

The session has two halves. In the first half, we study PixelCNN: how a convolutional network is turned into an autoregressive density model by masking its kernels, why stacked masks create a blind spot, and how each successor variant — Gated PixelCNN, PixelCNN++, PixelSnail — attacks one specific weakness. We also see how conditioning turns the same machinery into a class-conditional generator, a super-resolution engine, and a colorizer.

In the second half, we reset the scoreboard for what any generative model should deliver, rebuild exactly the probability tools we need (densities, integrals, maximum likelihood), and set up the one-dimensional normalizing flow: learn an invertible transformation that pushes the data's distribution onto a simple law such as a Gaussian, and run it backward to generate.

A map of the road ahead:

Keep one thread in mind throughout: every design decision below exists to serve the same two masters — a probability law that assigns sensible densities to images, and a generation process that respects the rule "only the past may inform the present."

6.1 Recap: Likelihood-Based Autoregressive Models

Autoregressive models form one branch of the likelihood-based family of density estimators. Their defining habit is to use only data that has already happened — the past — and never the current point or anything in the future. Think of a typist copying a sentence letter by letter: at every keystroke they know only what has been typed so far. An autoregressive model lives under exactly that restriction.

Earlier sessions built the masked-autoencoder version of this idea (the MADE-style estimator: a network whose connections are masked so each output can read only earlier inputs) and showed how to sample from it so new data can be created. This session carries the same idea to images with convolutional masks — that is PixelCNN.

6.1.1 Properties Shared by Likelihood-Based Models

Three traits recur across every method in this family, so keep them handy as a scorecard. We will reuse this scorecard at the end of the session when a new family — normalizing flows — walks on stage.

First, inference ability. Because training hands you access to the underlying probability distribution $p(x)$, the model can score any sample you show it. Given a generated data point, it answers the question "how likely is this?" Concretely: show the model an image and it returns the probability it would assign to seeing such an image. That is a built-in anomaly detector — feed it something strange and the probability comes out low.

Second, slow, sequential sampling. Producing one sample means producing values one after another — pixel after pixel, or audio sample after audio sample — because each value depends on the ones before it. Generation lags behind one-shot generators that produce everything in a single forward pass. Speed is the known tax of this family.

Third, stable training. The loss is a genuine likelihood — the negative log likelihood you will meet formally in Section 6.2.2 — so you can watch it fall in a reliable way. Stable does not mean perfect: a steady run can still end with mediocre samples. But the process behaves predictably, and that predictability matters when you debug. If your loss curve looks like noise, something is mechanically wrong; with adversarial-style training the same curve might mean nothing at all.

A compact comparison:

These methods also stretch across dimensionalities in a straightforward way: the chain-rule recipe does not care what the data looks like. They handle one-dimensional data such as raw audio waveforms, extend to natural language text as sequences of tokens, cover two-dimensional data such as images, and reach three-dimensional data too (for example, volumetric medical scans). One recipe, many shapes of data.

6.1.2 Prerequisites From Earlier Coursework

This course leans heavily on deep-learning basics from the earlier deep neural networks course. Before going further, make sure you are conversant with:

• Backpropagation — how the gradient of the loss reaches every weight by flowing backward through the network graph.
• Gradient descent updates — computing a weight update $\Delta w=-\eta \partial L/\partial w$ by hand, where $\eta$ is the learning rate.
• Small convolutions by hand — sliding a kernel over a small grid and computing each weighted sum with a calculator.
• Parameter counting — given a convolutional layer, writing down its number of trainable weights and biases from memory. For example, a layer with 16 filters of size $3\times 3$ operating on a single input channel has $16\times 3\times 3=144$ weights plus 16 biases.

PixelCNN will exercise all four skills, and the exam may too.

6.1.3 Student Questions and Answers

With the recap done, the session now builds its main character: PixelCNN, the convolutional take on "predict the future from the past."

6.2 PixelCNN: Masked Convolutions for Image Density Estimation

PixelCNN puts convolutional-network machinery to work on pixels. The CNN part of the name points at convolution layers; the pixel part says the input is an image grid. A sibling model, PixelRNN, instead processes pixels row by row with recurrent units such as LSTM cells. PixelCNN processes each pixel through its neighborhood, which makes 2D convolution the natural engine — convolutions are built to combine a pixel's neighbors efficiently, and that is exactly what per-pixel prediction needs.

Real-world: this model family is unusually well served by written walkthroughs. Multi-part blog series cover it from plain CNNs through every variant (one such series lives at thomasjubb.blog), and asking a chatbot "explain PixelCNN and its variant architectures step by step" walks the entire path in textual form with pictures. They make excellent second passes after these notes.

6.2.1 Masked 2D and 1D Convolutions

Picture a standard two-dimensional convolution with a three-by-three kernel. The kernel centers on a pixel and carries weights at all nine positions, so it reads the eight neighbors plus the center. Under an autoregressive ordering, though, some of those neighbors belong to the present or the future. If pixels arrive in raster-scan fashion — left to right along each row, top row first — then everything strictly to the right of a site, everything below it, and even the site's own value has not "happened yet" when we predict it.

Masked convolution fixes this by force: set the kernel entries that touch present-or-future positions to zero. For a raster-scan ordering, the surviving taps are the up-and-left neighborhood:

Kernel position      Mask M        Surviving weights
 w11  w12  w13        1  1  1       past rows: all columns
 w21  w22  w23        1  0  0   ←   current row: only strict left; center zeroed
 w31  w32  w33        0  0  0       future rows: nothing

In words: use information only from the past, and make every future entry zero so the layer physically cannot read it. No cleverness in the training can resurrect a weight that is hard-zeroed — the causality is enforced by the architecture itself, not learned.

The same trick works in one dimension. Center the mask on the current position, zero the center tap and everything after it, and keep only taps from positions that happened earlier. How far back the taps reach is your design choice — a five-tap mask looks further into the past than a three-tap one — but once you fix the data-occurrence convention, you must stick to that masking convention everywhere. Mixing conventions between layers or between training and generation silently corrupts what conditions on what.

6.2.2 Joint Distribution Factorization and Training Objective

Take an image with $n\times n$ pixels, so $n^2$ values in total, and write $x_i$ for the intensity value at the i-th pixel under the chosen ordering. The goal is the probability of the whole image — the joint distribution of $x_1,x_2,x_3$ through $x_{n^2}$. Estimating a joint over $n^2$ variables head-on is hopeless (for a $28\times 28$ grayscale image that is a distribution over $256^{784}$ combinations), but the chain rule of probability turns it into a product of conditionals, each of which is far more learnable:

$$p(x)=p(x_1)p(x_2\mid x_1)p(x_3\mid x_1,x_2)\cdots p\left(x_{n^2}\mid x_1,\dots ,x_{n^2-1}\right)$$

Compact form, with $x_{<i}$ shorthand for "all pixels before i":

$$p(x)=\prod _{i=1}^{n^2}p\left(x_i\mid x_{<i}\right)$$

Read it aloud: the probability of the image equals the probability of the first pixel, times the probability of the second given the first, times the probability of the third given the first two, and so on down the line. Nothing about the joint has changed — the chain rule is an identity — but now every factor is a small supervised prediction problem.

To learn this from data, maximize the product over all possible $x$, where $x$ ranges over the training images. A product of many probabilities is awkward to handle numerically (hundreds of factors below 1 multiply toward machine zero), so take the natural logarithm; products become sums:

$$\ln p(x)=\ln p(x_1)+\ln p(x_2\mid x_1)+\cdots +\ln p\left(x_{n^2}\mid x_{<n^2}\right)$$

This sum is the log likelihood of the data. Maximizing likelihood and minimizing its negative are the same act, so attach a minus sign and average over the N training images to get the negative log likelihood, usually written NLL:

$$\mathrm{NLL}=-\frac{1}{N}\sum _{k=1}^N\ln p_{\theta }\left(x^{(k)}\right)$$

Here $\theta$ is the collection of trainable parameters (all kernel weights, biases, and softmax coefficients) and $N$ is the number of training images. Minimizing the average NLL is exactly maximizing the likelihood — gradient descent on this loss pushes every conditional toward agreeing with the training images.

One factor deserves special mention. The first term, $p(x_1)$, involves a single one-dimensional value, and a large training set offers many samples of it — so a histogram or similar estimate handles it directly, no network needed. Every later conditional, such as $p(x_2\mid x_1)$, conditions on a growing list of past pixels, and there a learner is needed. That learner is the masked convolutional network, and this is exactly where the "CNN" in PixelCNN comes from: the network's job is to produce the conditional distributions inside the chain-rule product.

6.2.3 Architecture and the Per-Pixel Softmax Head

The pipeline runs as follows. The original data — say a simple RGB or grayscale image — enters a first masked convolution. After that come a bunch of further masked convolutions, standard CNN style, with ReLU activations along the way (occasionally one-by-one convolutions mix feature maps when several filters must interact). The stack produces feature maps, and the feature maps connect to a softmax layer holding the trainable coefficients' output.

Suppose the image is grayscale, so each pixel takes a value from 0 to 255. Rather than treating intensity as a bare number to regress, PixelCNN treats it as a 256-way classification problem per pixel. The softmax head then predicts a full categorical law per pixel:

$$P(X_i=0),P(X_i=1),\dots ,P(X_i=255)$$

That is 256 outputs for every single pixel — a $28\times 28$ MNIST-style image carries $784\times 256$ output probabilities. Training pushes probability mass onto whichever entry matches the true observed intensity and squeezes the other 255 entries down. Repeat this for every pixel in every training image. Once training ends, the network predicts any pixel's value from whatever happened earlier under the autoregressive ordering you attached — and sampling simply walks the image in order, drawing one value from each predicted law.

So the structure is CNN-like, but its output is the probability of a particular pixel having each intensity level given the training set.

6.2.4 Autoregressive Orderings

Nothing forces the raster scan. The famous default — left to right, top to bottom — is named for how the display elements of ancient TV screens used to get lit up, one row at a time: the electron beam swept each row before dropping to the next, and "raster" stuck as the name for that path. Row-wise orderings work too, and more exotic snaking paths work as well; an ordering of this flavor appeared earlier in the course in MADE — the masked autoencoder for distribution estimation — where input units were masked so each output could see only inputs earlier in a chosen order.

The designer owns this choice. You may specify any ordering, for whatever reason you have. The moment you fix it, though, the ordering implicitly fixes the masking structure: attaching to every pixel a notion of which one occurs before which tells you exactly which kernel taps must be zeroed. Raster scan is what people typically use, but the freedom is yours — the mathematics of Section 6.2.2 never mentioned directions, only "before."

6.2.5 Conditioning Across RGB Channels

Everything so far assumed grayscale — one input channel. With multiple color channels, attach an autoregressive ordering across channels as well: R occurs first, then G, then B. Write $x_i^R$, $x_i^G$, $x_i^B$ for the three channel values at pixel i. The green value depends not only on the green channel's past but also on what happened in the red data, and blue leans on both:

$$p\left(x_i^G\mid x_{<i}^G,x_i^R\right),p\left(x_i^B\mid x_{<i}^B,x_i^G,x_i^R\right)$$

A note on scope, since the narration leaves it loose: in the standard formulation, the green channel at position i sees all channels of strictly earlier pixels plus the red value at the current position, and blue sees all of that plus green at the current position. So "its own past plus the red data" really means: everything already generated anywhere in the image, with same-position cross-channel reads allowed because R was declared to happen before G. The takeaway stands either way: for color images, each conditional exploits the autoregressive structure inside its own channel and stacks the cross-channel structure on top — R first, then G given R, then B given both R and G. In practice this is implemented by splitting each color's prediction into separate one-dimensional masked operations, one per channel, with slightly different scopes.

6.2.6 First Extensions: Depth, Skip Connections, Gating

Several variants of PixelCNN exist, and their innovations are fairly natural upgrades.

Upgrade one: go deeper. With enough data, more layers should raise the ability to learn the probability density — after training, the NLL should land at a smaller value than a shallow stack achieves. But deep convolution stacks invite vanishing and exploding gradients: repeated multiplication by small derivatives shrinks the error signal until early layers stop learning. The cure is borrowed straight from ResNet: skip connections that let gradients bypass layers, so deep PixelCNN architectures train efficiently. A skip connection adds a layer's input back onto its output, giving the gradient a short highway around the new computation.

Upgrade two: control the flow of past information with a gating structure, in the spirit of LSTM. A gating function — often denoted by a sigmoid — decides which past inputs deserve weight and which do not. A filter function — a hyperbolic tangent — then decides how much of the chosen information passes into the density calculation. Multiplying the two, element by element, gives fine control over what the estimate uses. Both upgrades reappear concretely under Gated PixelCNN next — and they arrive together with the fix to PixelCNN's most famous flaw, the blind spot.

6.3 The Blind Spot Problem

Every later variant exists largely because of one structural flaw in plain PixelCNN, so this flaw deserves careful attention. It is known as the blind spot.

6.3.1 How Stacked Masks Lose Pixels

Set the scene: raster-scan ordering, and you want $p(x_i\mid x_{<i})$ — the law of one pixel given all pixels before it. Intention says every past pixel should inform the estimate. Now inspect what the machinery actually sees.

At the target pixel's own layer, a three-by-three masked kernel reads only the up-and-left neighbors — the left column and the top row of its window; the center and right taps are zeroed. So layer one sees a small L-shape hugging the target's top-left.

Deeper layers widen the view, but watch how the widening behaves. Layer two aggregates layer-one outputs, each of which read their own up-and-left L-shapes. Composing the two, layer two effectively sees one more ring up-left; layer three, another. The reachable region grows diagonally toward the top-left corner. Trace which cells have been touched after three such doublings:

Rows above target (each symbol = whether that pixel feeds the target)

row i-3 :  X X X . . .        X = reached by some stacked mask
row i-2 :  X X X . . .        . = blind spot (valid past, never read)
row i-1 :  X X X X X .
row i   :  X X ?              ? = target; left neighbors are fine
              ↑
     columns strictly left of the target, low rows: unreachable

The upper-right part of the past gets covered as depth grows, and the directly-left pixels do get used. But a stair-shaped wedge of legitimate past pixels — sitting below-left of the diagonal growth path, in rows between the higher rows and the target's own row — never enters anyone's view. Information from those pixels simply never contributes to the probability of the target.

A concrete way to feel it: take an image, change one pixel located down-and-left of a target, and recompute the model's probability for that same target. Nothing moves. The model is provably indifferent to information the ordering says it should be using.

6.3.2 Why the Blind Spot Matters

Name the quantity at fault: the receptive field, meaning the region of the original input image that can influence a unit's output through any path of stacked layers. In ordinary CNN terms, receptive fields grow with depth — with kernel width three, layer one sees three inputs, layer two five, layer three seven, and so on. Masked convolutions inherit that growth but clip its direction: only up-and-left expansion survives the masks.

The design intent is to condition on all pixels numbered 1 through $i-1$. The two-dimensional masking structure breaks that promise — pixels arranged in the staircase pattern go unused, and for a target sitting low in the image, roughly half of the valid past can be lost. Half of the evidence, silently discarded from every probability estimate.

So although we intend to use every past pixel, the masking lets us consume mainly what lies diagonally up-left; right-side history survives (through later rows' wide top-row coverage), bottom-left history does not.

Eliminating this defect became the first order of business for every successor design — and Gated PixelCNN, next, does it with a surprisingly simple geometric trick: split the single diagonal-growing 2D mask into two clean 1D passes.

Real-world: why care about a subtle conditioning flaw? Every downstream consumer of these probabilities — anomaly detection on manufacturing images, compression, conditional generation — inherits whatever the receptive field ignores. A model blind to half its past assigns the same likelihood to images differing in exactly the details it cannot see, which quietly caps how good any sample or score can get.

6.4 Gated PixelCNN

6.4.1 Vertical and Horizontal Convolution Split

Rather than one masked two-dimensional convolution, Gated PixelCNN runs two passes, one after the other.

First comes the vertical convolution: a stack-style pass going down the rows, using a mask that reads only rows strictly above the current position (never the current row itself). Process the whole image this way and you hold a vertically convolved map in which every site has aggregated all past rows above it — every column, every width, nothing skipped. A column-only mask has no diagonal growth to hide information in, so no vertical blind spot can exist by construction.

Then apply a horizontal mask to the result of the vertical pass, reading only the current row's prefix — positions strictly to the left on the same row. Any past pixel is now reachable. Pixels above the current row travel through the vertical stream; pixels earlier in the same row travel through the horizontal stream. The union covers everything that happened in the past — the blind spot problem is gone.

Vertical stream sees:        Horizontal stream sees:
 . . . . .                    . . . . .
 . . . . .   rows above       ← ? . . .   current-row prefix only
 . . . . .   (all columns)      (then combined with vertical output)
 ─────────
 ← ? . . .   nothing below    ? = target pixel

In short: plain PixelCNN uses one 2D masked convolution; Gated PixelCNN breaks that 2D mask into a vertical piece and a horizontal piece, does the vertical piece first, and feeds the horizontal piece on top of the vertical output.

6.4.2 Gating and Residual Blocks

On top of the split sits the gating structure, just as in LSTM-kind architectures. Written out, the controlled signal is a gate times a filter, combined element by element:

$$g=\sigma \left(a_{\mathrm{gate}}\right)\odot \tanh \left(a_{\mathrm{filter}}\right)$$

where $\sigma$ is the sigmoid function $1/(1+e^{-u})$, producing values between 0 and 1 and acting as the gate selecting which past inputs deserve weight; $\tanh$ is the hyperbolic tangent, producing values between −1 and 1 and acting as the filter setting how much of each selected input passes; and $\odot$ denotes element-wise multiplication of equally shaped feature maps. A sigmoid near zero slams a feature channel shut; near one it flings it open. The filter then shapes how much, signed positive or negative, of each opened channel flows into the current pixel's density estimate.

Every one of these weights — the vertical-mask kernel, the horizontal-mask kernel, the gate weights, the filter weights — is learnable, and they are learned as part of the overall loss-minimization process. Nothing about the masks or gates is hand-designed beyond their zero patterns; minimizing the NLL trains the filters, the masks, and both gating components together, end to end.

Depth safety comes from a ResNet block: the skip connection reduces the vanishing-gradient problem so the architecture can grow, and a deeper stack approximates the density better.

6.4.3 NLL Results on CIFAR-10

Here is the payoff measured as NLL on the CIFAR-10 image dataset — the same dataset family as the assignment, reported in bits per pixel dimension (smaller is better). A baseline near 3.14 falls to 3.03 for Gated PixelCNN on the validation/test side, with the training-side figure shown alongside. Read it this way: smaller NLL means a better probability estimate, and the drop from 3.14 to 3.03 is the improvement. To pin down the comparison rows precisely: the 3.14 figure belongs to plain PixelCNN (its published test score, with about 3.08 on the training side), while Gated PixelCNN reaches 3.03 test (about 2.90 train) — and for context, the recurrent sibling PixelRNN sits near 3.00 at a much higher sampling and training cost. So the gated redesign closes most of the gap to the LSTM-style model at less than half the training expense.

Two causes earn the gain. Better receptive field, delivered by the vertical-plus-horizontal decomposition that removes the blind spot — the model finally conditions on all the evidence the ordering promises it. More expressive architecture, delivered by many ResNet-like layers plus the gating mechanism deciding which and how much past information enters each estimate.

Real-world: this decomposition-plus-gating pattern — factor a global operation into clean directional streams, then let learned multiplicative gates route information — recurs across generative modeling, notably in WaveNet-style audio models where the same gated unit generates raw speech waveforms sample by sample.

6.5 PixelCNN++

The next variant, PixelCNN++, attacks a different weakness: the sheer wastefulness of predicting 256 independent categories per pixel.

6.5.1 Wasteful 256-Way Softmax Outputs

Both PixelCNN and Gated PixelCNN stack masked convolutions, sprinkle ReLUs, sometimes add one-by-one convolutions when multiple filters mix at a layer, and finish with a softmax. For a grayscale pixel that means 256 softmax nodes per pixel — one output node for each possible intensity — and the winning node names the pixel's value. Memory requirements balloon, and much of the machinery is redundant.

Why redundant? In real images, neighboring pixel values sit close together most of the time. If a pixel holds intensity $i$, its predecessors likely held $i$, or $i-1$, or $i-2$ — not $i-55$. The exceptions are edges and color changes, but count the edge pixels in a typical natural image and they form a small fraction of the total — a few percent. An unrestricted 256-way law spends its capacity on empty ground: wasted memory, far more parameters to estimate, more training data consumed, longer training time, heavier compute. It is an inefficient design.

6.5.2 Mixture of Logistics Output Law

PixelCNN++ keeps the fact that nearby intensities co-occur and rebuilds the output around it: express the pixel-value law as a mixture model — specifically a mixture of logistic distributions:

$$p(x\mid \pi ,\mu ,s)=\sum _{k=1}^K{\pi }_k\mathrm{logistic}\left(x;{\mu }_k,s_k\right)$$

Here ${\pi }_k$ are the mixture coefficients (non-negative weights that sum to one, saying how much each component contributes), ${\mu }_k$ the mean of each logistic component (where its bell peaks), and $s_k$ its scale — the spread, playing the role that standard deviation plays for a Gaussian. Recall the mixture-of-Gaussians work from the machine-learning course's unsupervised clustering part: a batch of mixing coefficients learned alongside component parameters by iterative optimization. The same skeleton applies here, learned during PixelCNN++ training by gradient descent on the NLL — typically with small $K$, such as ten components per pixel channel, instead of 256 free category probabilities.

What does a logistic distribution look like? Close to a Gaussian, but slightly flatter — a bit broader in the shoulders around the peak. Its density is

$$\mathrm{logistic}(x;\mu ,s)=\frac{e^{-(x-\mu )/s}}{s{\left(1+e^{-(x-\mu )/s}\right)}^2},$$

with $s>0$ setting the width. Now the fact the speaker hedged on ("I'm not sure, but I think…") can be settled outright: yes, this density integrates to exactly the familiar sigmoid curve. Differentiate $\sigma \left(\frac{x-\mu }{s}\right)$ using the chain rule and the sigmoid derivative identity ${\sigma }^{\prime }(u)=\sigma (u)(1-\sigma (u))$:

$$\begin{array}{rl}\frac{d}{dx}\sigma \left(\frac{x-\mu }{s}\right)& =\sigma \left(\frac{x-\mu }{s}\right)\left(1-\sigma \left(\frac{x-\mu }{s}\right)\right)\cdot \frac{1}{s}\\ & =\frac{\sigma \left(\frac{x-\mu }{s}\right)\left(1-\sigma \left(\frac{x-\mu }{s}\right)\right)}{s}=\frac{e^{-(x-\mu )/s}}{s{\left(1+e^{-(x-\mu )/s}\right)}^2},\end{array}$$

where writing $u=\frac{x-\mu }{s}$ turns the last equality into algebra: $\sigma (u)(1-\sigma (u))=\frac{e^{-u}}{(1+e^{-u}{)}^2}$. So the area under the logistic density from $-\infty$ up to $x$ is precisely

$$F(x)=\sigma \left(\frac{x-\mu }{s}\right)=\frac{1}{1+e^{-(x-\mu )/s}},$$

the sigmoid curve — rising smoothly from 0 to 1, steepest at the mean $\mu$. A Gaussian mixture could serve too, but logistics reportedly fit image intensities better and give more realistic behavior in intensity space.

6.5.3 Discretizing the Continuous Density

One wrinkle: the mixture is a continuous law over real-valued $x$, while pixel intensities are discrete integers — 0, 1, 2, and so on up to 255. The bridge is a differencing trick applied to the mixture's cumulative distribution function (CDF):

$$P(X=v)=F(v+0.5)-F(v-0.5)$$

In words: the probability that the integer-valued pixel equals $v$ is the probability mass the continuous law places inside the half-unit-wide window straddling $v$. Nothing deeper is happening — a continuous density becomes a discrete, binned representation, one bin per integer, each bin half a unit on either side of the integer.

6.5.4 Reading the Logistic PDF and CDF Plots

The plots reward a careful look — they are the same object seen twice. Each logistic PDF is a bell peaking at its mean $\mu$ (horizontal axis: intensity $x$; vertical axis: density $p(x)$), and the scale $s$ sets how rapidly the values fall off around that peak. The CDF of any density integrates the PDF from minus infinity up to $x$; it rises from 0 toward 1 and its maximum value is exactly 1.0, because total probability must sum to one.

For the standard member with $\mu =0$ and $s=1$, the bell is narrow, so the CDF climbs sharply in a small window — nearly vertical around the origin. Take instead the wide member — the red curve with standard deviation 4 in the demonstration plot — and its CDF rises smoothly and gradually over a span several times wider.

The reading rule that makes these curves useful:

• Steep CDF ⇒ packed mass. Most pixels in a narrow range, few outside. A model confident about a pixel's neighborhood looks like this.
• Gentle CDF ⇒ spread mass. Plenty of pixels between minus 5 and plus 5 relative to the mean, few clustered tightly at the peak — the model hedges.

Learning to read CDF steepness is learning where a density stores its data: the steep segment of the CDF sits directly above the dense part of the histogram, and flat segments above the empty parts.

6.5.5 Downsampling and Long-Range Structure

PixelCNN++ also adds downsampling to capture long-term dependencies — because even with the blind spot fixed, stacking local convolutions means distant pixels influence each other only through many layers. Six residual blocks form the backbone — residual connections again, letting many layers cooperate on approximating the density. Along the way the maps shrink: the original $32\times 32$ drops to $16\times 16$, then to $8\times 8$. Then the maps grow back — $8\times 8$ holds, then $16\times 16$, then $32\times 32$. Skip connections join the same-size maps across the squeeze.

Why does shrinking help reach? Each convolution sees a fixed neighborhood; halving the map size doubles how much image one position effectively stands for. After two downsamplings, a single $8\times 8$ position aggregates evidence spanning a large share of the whole $32\times 32$ image — long-range structure becomes cheap. The arithmetic is stark: a convolution operating on an $8\times 8$ map touches 64 positions instead of the 1,024 positions of a $32\times 32$ map — sixteen times less work per filter — which is exactly what lets the model afford enough depth to use that reach.

The measured NLL lands at 2.92 — a real drop against the 3.14-and-3.03 figures above, earned by the mixture output head, the downsampling backbone, and the residual blocks together. Worth noting: PixelRNN also does reasonable work in this league, processing rows in one-dimensional fashion left to right with recurrent units.

Real-world: the discretized-mixture-of-logistics trick now appears far beyond PixelCNN++ — any model that must produce integer-valued data (pixel values, audio sample codes, token IDs) borrows the same CDF-differencing bridge between continuous densities and discrete data.

6.6 PixelSnail: Masked Self-Attention

The last variant swaps convolution's locality for attention's reach. PixelSnail stands for improved likelihood thinking fused with a non-local, transformer-style mechanism.

6.6.1 Attention versus Convolution

Every variant so far gathers past information through convolutions. Even with the 2D-mask broken into 1D pieces and many layers stacked, the reachable past stays limited unless the network grows very deep — each layer extends reach by only a few pixels. Self-attention removes that ceiling.

Convolution couples each site to a fixed window — a three-by-three or five-by-five mask, a bounded extent of information no matter what the content is. Self-attention, especially in transformer form, lets every position consult every other position through matrix-vector multiplications: each site emits a query ("what am I looking for?"), every other site offers a key ("what do I contain?"), and query-key matches — scaled and passed through a softmax — decide how much of each site's value (its content) flows back. Distance plays no role: pixel 5 can attend to pixel 500 as easily as to its neighbor.

Applied to an autoregressive model, it wears a mask: masked attention zeroes access to the future, enforcing causality — before the softmax over keys, any key sitting at or after the current position is set so it receives zero weight. Masking supplies the order in which information operates — and that order remains a designer's choice, exactly as before. Whatever autoregressive structure you design dictates the masking strategy that guarantees causality.

There is a practical bonus. Encoding a fancy custom ordering as a zero-one kernel pattern for masked convolution is cumbersome — you must redesign the 0/1 layout by hand for each new ordering, and verify no tap leaks. Doing it with attention is much easier, because the choice point is explicit everywhere: for each query site you simply declare which keys it may read — flip one boolean per pair, done.

6.6.2 Receptive Field Wins and Compute Costs

Set the variants side by side on receptive field:

Standard PixelCNN wastes much of its usable past — significant information loss. Gated PixelCNN loses less, helped by its decomposition and extras. PixelCNN++ sees part of the past directly and the rest through its downsampled bottleneck. PixelSnail, thanks to masked attention, uses everything available at the current point immediately.

The reward shows in the numbers: PixelSnail pushes the NLL down to the smallest values in this lineup. The price: training is slow, and the transformer backbone is computationally expensive — attention compares every position against every other, a cost that grows quadratically with the number of positions, whereas a convolution touches each position with only its small window.

Real-world: this exact trade — attention's global reach bought with quadratic cost — is the defining engineering tension of modern sequence models; managing it (through sparse, linear, or windowed attention) is a whole research area, and image autoregression was one of the first places the tension showed up.

6.7 Conditional Generation with PixelCNN

Plain PixelCNN learns the unconditional law of images — $p(x)$. Feed it side information during training and generation, and the same machinery serves directed tasks: the law becomes $p(x\mid \text{condition})$.

6.7.1 Class-Conditioned Generation

Train a PixelCNN on MNIST — $28\times 28$ handwritten digits from 0 to 9, the same kind of data as the assignment. At generation time you may want only fives. The mechanism: encode the label as a one-hot vector — one followed by zeros, a different position hot for each digit 0 through 9 (so digit 5 is $[0,0,0,0,0,1,0,0,0,0]$) — and inject this encoding, call it $h$, into the network during both training and generation. The model then learns $p(x\mid h)$ instead of plain $p(x)$.

A few extra weights learn how $h$ should bend the density — typically by adding $h$'s contribution into the feature maps at every layer, including inside the gating functions, so the condition can open or close information pathways rather than merely nudge one output. After training, fixing $h$ to the digit-5 slot steers every sample onto fives; sliding through $h=0,1,2,\dots$ walks the model across the whole alphabet of digits. During training the true label rides along with every image, so each conditional in the chain-rule product quietly becomes label-aware.

6.7.2 Super-Resolution and Colorization

Super-resolution works the same way with a spatial condition instead of a categorical one.

Color creation follows identically: condition on the grayscale version and generate colors, learning $p(x_{\text{color}}\mid x_{\text{gray}})$ and going from grayscale to color images. Nothing about the network changes across these three tasks — labels, thumbnails, and gray channels are all just extra inputs.

Real-world: class-conditioned sampling, super-resolution, and colorization all reduce to one recipe — send conditional information in as another input during probability-density learning. The same pattern powers practical tools today: text-conditioned image generators are the scaled-up descendants of exactly this conditioning idea, and photo-restoration pipelines use learned conditional laws for deblurring and colorization of archival material.

6.8 What We Want From a Generative Model

Before flows, reset the scoreboard: what should any generative model deliver?

6.8.1 The Bucket List

Five wishes, in order.

1. Estimate the probability distribution from training data, obtaining $p_{\theta }(x)$ with learned parameters $\theta$.
2. Sample from that distribution — and remember how big sampling lives: even a small $28\times 28$ image is 784-dimensional data, and a bag-of-words text over a vocabulary of 1,000 lives in 1,000 dimensions. Sampling well in high dimensions is genuinely hard.
3. Train stably, watching a well-behaved loss fall.
4. Evaluate fresh samples — after generating new data, say how likely it is. This is the self-audit wish: a model that can score its own outputs can be compared against others on equal terms.
5. Carry a meaningful latent representation or embedding space: a smaller-dimensional code for the data — of the kind an autoencoder's coding layer provides — from which you can even sample to create new data.

All five sit in the bucket list; almost no model fills every slot. Each family you meet is best understood by which wishes it grants and which it quietly drops.

6.8.2 Scoring Autoregressive Models and Previewing Flows

Score the autoregressive family honestly against the list:

Good fit to training data: yes, especially with many layers. Ability to evaluate how likely a new $x$ is: yes. Ability to sample: yes. Drawbacks: sampling is sequential and slow; there is no latent representation to sample from; and the family shines mainly on discrete data — image intensities 0 through 255, for instance — at least where it has proved most effective.

Their historic weak spot is qualitative: the raw samples trail the best later models. Newer work narrows that gap — TARFlow, a Transformer-based autoregressive flow out of Apple around 2024, performs very well indeed. One more upfront note: flows are born in the continuous domain, and a minor but important adaptation called dequantization bridges the continuous formulation to discrete images and text; that arrives next session.

Real-world: the bucket list is not academic bookkeeping — it is the checklist practitioners actually run when choosing a generative approach. Need certified likelihoods (compression, anomaly detection, scientific density estimation)? Wishes 1 and 4 dominate. Need fast high-quality generation for media? Wish 2 dominates, and families that sacrifice likelihoods often win.

6.9 Foundations of Normalizing Flows

Flow models lean significantly on probability and statistics — specifically derivatives of probabilities and how they behave under transformations. Every tool used below is rebuilt here from first principles, so the construction needs no outside machinery.

6.9.1 Probability Refresher: Densities and Integrals

For a continuous random variable $X$, the function $p(x)$ is the probability density function (PDF): a non-negative curve whose total area is exactly 1. A density itself may exceed 1 at a point — only areas carry probability. The chance that $X$ falls between two points $a$ and $b$ comes from integrating the density between them:

$$P(a<X<b)={\int }_a^{b}p(x)dx$$

Concrete instance: suppose the interesting window runs from 0.35 to 0.45. Given the density curve $p(x)$, compute $P(0.35<X<0.45)$ as the integral from 0.35 to 0.45 of $p$ of $x$, $dx$. Geometrically: calculate the area under the curve over that window — the area under the curve is the probability. On axes labeled "$x$" (horizontal) and "density" (vertical), the region between the curve, the horizontal axis, and the two vertical cuts at 0.35 and 0.45 is exactly the answer.

Worked version of that answer, with real numbers. Split the span into ten slices of width $\Delta x=\frac{0.45-0.35}{10}=0.01$ each (the narration's loose "multiply all of them by 0.1" should read this way: the whole interval is 0.10 wide, so ten equal slices are 0.01 wide apiece). Take a concrete rising density, $p(x)=2x$ on $[0,1]$, whose values at the left edges $0.35,0.36,\dots ,0.44$ run from 0.70 up to 0.88:

$$P(0.35<X<0.45)\approx \sum _{i}p(x_i)\Delta x=[p(0.35)+p(0.36)+\cdots +p(0.44)]\cdot 0.01$$

The ten left-edge heights sum to $0.70+0.72+\cdots +0.88=7.90$, so the estimate is $7.90\times 0.01=0.079$. Sense check: integrating exactly, ${\int }_{0.35}^{0.45}2xdx=0.45^2-0.35^2=0.080$ — the slice sum lands just under it, because every left-edge height slightly undershoots a rising curve. Smaller slices close the gap; in the limit of infinitely many, sum becomes integral.

Hold onto this pairing — integral equals area, summation approximates it. It is the kind of thing worth remembering years from now.

6.9.2 Fitting Densities by Maximum Likelihood

How do you fit a density model to data? Maximize the likelihood. Assume samples $x_1,x_2,x_3,\dots ,x_N$ are independent of one another; then their joint probability is the product of individual probabilities:

$$\mathcal{L}(\theta )=\prod _{k=1}^N{p}_{\theta }(x_k)$$

Maximize this product over the parameters $\theta$. Products are hard to evaluate and differentiate (hundreds of factors below one multiply toward zero), so take the natural logarithm — products become sums:

$$\ln \mathcal{L}(\theta )=\sum _{k=1}^N\ln p_{\theta }(x_k)$$

Flip the sign and minimize the negative log likelihood instead — in practice the average over all training samples:

$$\mathrm{NLL}=-\frac{1}{N}\sum _{k=1}^N\ln p_{\theta }(x_k)$$

Different density families plug into the same recipe. Choose a plain Gaussian and minimization hands you the mean vector and the covariance matrix in closed form. Find a Gaussian too restrictive — real histograms are lumpy, skewed, multimodal? Use a mixture of Gaussians:

$$p_{\theta }(x)=\sum _{k=1}^K{\pi }_k\mathcal{N}\left(x;{\mu }_k,{\sigma }_k\right)$$

Here ${\pi }_k$ are mixture weights (${\pi }_k\ge 0$, summing to 1), ${\mu }_k$ the constituent means, and ${\sigma }_k$ the constituent standard deviations. On the one-dimensional slide shown, sigma runs ${\sigma }_1$ through ${\sigma }_K$ as plain standard deviations; multidimensional Gaussians demand covariance matrices instead. More components, more bumps — mixtures can hug arbitrarily complicated histograms, which is precisely why they will reappear as flow building blocks later in this session.

6.9.3 The Flow Idea: Learn a Push to a Simple Law

Now the main construction. Start from a training sample $x$ — treat it as one-dimensional, a scalar, not a vector. Plot all training samples as a histogram: that histogram is the empirical probability distribution of the data. The ambition is an analytical function for that law — a closed-form expression mixing exponentials, powers, and logarithms — because once an analytical law exists, a CDF follows, and sampling follows the CDF.

Here is the sampling link worth internalizing. Build the CDF from the analytical density by integrating from minus infinity to $x$; the CDF climbs from 0 to a maximum of exactly 1.0, since probabilities sum to one. Draw uniformly along the CDF's vertical axis — pick a random height between 0 and 1 — then read across to the $x$ value beneath that height. Each such readout is one sample obeying the law: tall dense regions occupy large vertical stretches of CDF, so uniform draws land there often; thin regions barely register. Generating new data, then, means generating an analytical probability distribution and sampling it.

The flow model builds that analytical distribution by pushing data through transformations. Apply a transformation $f_1$ to $x$, feed the result to $f_2$, continue through $f_n$:

$$z=f_n\left(f_{n-1}\left(\cdots f_2\left(f_1(x)\right)\cdots \right)\right)$$

These transformations are not handed down — they are designed as part of training, realized as layers of a deep network. The training goal: choose them so that the pushed data $z$ obeys a simple law. $p_Z(z)$ should be a uniform distribution, a Gaussian, a mixture of Gaussians — something with a closed form you can sample easily. Generation then runs the movie backward: sample $z$ from the simple latent law, pass it through the inverse transformations, and a new data point $x$ falls out. That is the whole idea at altitude: learn transforms that push data to a simple latent law, then invert them to create.

An everyday picture for the push: think of kneading dough. Each layer folds and stretches the dough a little more; after enough folds, an originally lumpy blob becomes a smooth, uniform ball. Training learns the folding recipe; generation un-folds a smooth ball back into a lumpy — but realistic — shape. The analogy breaks where it must: dough folds lose information, while flow layers are forbidden from losing any (next subsection).

6.9.4 Invertibility: Which Transformations Qualify

Generation travels the chain backward, so the forward map must be one-to-one — a bijective transformation: start from one data point, go forward, invert, and exactly one point comes back, never several. If the inverse of $x$ were not a unique partner, sampled latents would spawn multiple candidate outputs and generation breaks down — which half of the ambiguity do you render?

This exchange corrects a tempting shortcut ("powers are fine, aren't they?") and hides the general rule: admissible transformations are monotonic — rising everywhere or falling everywhere, never changing direction. A parabola changes direction at its vertex, so it fails; a cubic never does, so it passes. Equivalently, check the derivative: if $f^{\prime }$ flips sign anywhere, some pair of inputs collides.

One honest caveat, flagged upfront: many people suspect this invertibility-only restriction is why classical flows occasionally trail newer generators in sample quality — constraining every layer to be one-to-one may throttle the method's data-generation ability. Newer designs chip away at exactly this suspicion, adding expressive power without giving up the reversible guarantee.

6.9.5 Change of Variables

Everything now rests on one probability identity. Write $z=f_{\theta }(x)$, where $f_{\theta }$ is the learned transformation, $x$ the training data, and $\theta$ the transformation's parameters; $z$ must land in a simple, analytical, samplable law. The variable $x$ carries its own density $p_X$, and $z$ carries another, $p_Z$. Both are probability laws, so both integrate to one:

$$\int p_X(x)dx=\int p_Z(z)dz=1$$

That shared-total-probability fact forces the two densities to move together wherever the transformation stretches or squeezes space. Here is the derivation, every step annotated. Assume first that $f_{\theta }$ is monotonically increasing:

$$\begin{array}{rlrl}P(a<X<b)& =P(f_{\theta }(a)<Z<f_{\theta }(b))& & \text{(same event, since }{f}_{\theta }\text{ preserves order)}\\ & ={\int }_{f_{\theta }(a)}^{f_{\theta }(b)}{p}_Z(z)dz& & \text{(probability is area under the latent density)}\\ & ={\int }_a^b{p}_Z(f_{\theta }(x))f_{\theta }^{\prime }(x)dx& & \text{(substitute }z=f_{\theta }(x),dz=f_{\theta }^{\prime }(x)dx\text{)}\\ & \Rightarrow p_X(x)=p_Z(f_{\theta }(x))f_{\theta }^{\prime }(x)& & \text{(equal integrands give equal densities)}\end{array}$$

If $f_{\theta }$ is decreasing instead, the integration bounds swap and a minus sign appears; a density can never be negative, so the general statement keeps the magnitude of the slope:

$$p_{\theta }(x)=p_Z(f_{\theta }(x))\left|\frac{\partial f_{\theta }(x)}{\partial x}\right|$$

Read it aloud: the density at $x$ equals the latent density evaluated at the transformed location, times the magnitude of the transformation's slope. The intuition matches the stretching picture: stretch a region of the axis and its probability mass spreads thinner (slope above 1 lowers the density); compress it and mass piles up (slope below 1 raises the density). The absolute value exists because a probability density function must stay positive — slopes may be negative, densities may not. A notation care-point: the $\theta$ here belongs to the transformation, not to a fitted density; if $f(x)=x^3$, then bundling the coefficient and the cube into $\theta$ describes the whole map at once. Textbooks sometimes write the equivalent form in the other direction — with $x=f[z]$ generatively, $\mathrm{Pr}(x)=|\partial f/\partial z{|}^{-1}\mathrm{Pr}(z)$ — which agrees with ours because inverting a map reciprocates its slope.

6.9.6 The Flow Training Objective

Training a flow means maximizing likelihood, now powered by the change-of-variables identity. Take natural logs of both sides — the product inside turns into a sum of two terms:

$$\ln p_{\theta }(x_i)=\ln p_Z(f_{\theta }(x_i))+\ln \left|\frac{\partial f_{\theta }(x_i)}{\partial x_i}\right|$$

Sum over the training set and optimize:

$${\theta }^{\star }=\arg \underset{\theta }{\max }\sum _{i=1}^N\left[\ln p_Z(f_{\theta }(x_i))+\ln \left|\frac{\partial f_{\theta }(x_i)}{\partial x_i}\right|\right]$$

Two terms, two messages. The first rewards placing transformed data where the latent law is generous — land your pushed points near the peak of $p_Z$ and that term grows. The second rewards transformations whose slope spreads the mass appropriately — squash everything into a tiny interval and the slope term punishes you, because a small $|f^{\prime }|$ makes its log deeply negative. Neither term alone suffices: cramming data onto the latent peak with a collapsing map scores well on term one and terribly on term two.

Equivalently, minimize the negative of this sum; practitioners speak of driving the NLL small. Either way, stochastic gradient descent adjusts the parameters of the transformation layers across the whole training set.

Once training ends, $f_{\theta }$ is frozen, and generation is trivial: draw $z$ from the latent law — under a standard normal, sampling is as easy as it gets — and apply $f_{\theta }^{-1}$ to harvest a new $x$. Notice the latent-representation wish from the bucket list quietly granted: the $z$-space is exactly a meaningful latent space you can sample from to get the original data back. And the name finally earns itself: "normalizing flow" means $z$ lands on a normal Gaussian with mean zero and standard deviation one.

6.9.7 Choosing Invertible Functions

Which function families qualify as layers? The monotonic test from earlier is the gatekeeper, and it sorts candidates quickly.

The quadratic $x^2$ fails: rising on one side of zero, falling on the other — its derivative $2x$ switches sign, so it folds inputs together and cannot be inverted. The cubic $x^3$ passes: monotonic through the origin, one output per input.

A rich, easy-to-adopt class follows immediately: polynomials carrying only positive coefficients and only odd powers, such as $a_{1}x+a_3{x}^3+a_5{x}^5$ with all $a_i>0$. Every member is monotonic — each term rises everywhere and positive sums of rising terms rise — so each can be inverted. More candidates: the exponential $e^{\theta x}$ works (strictly increasing for any real $\theta$); the shifted scaled sigmoid $\mathrm{sigmoid}(ax+b)$ with $a>0$ works (sigmoid is monotonic, and positive scaling plus shifting preserves that); and so do cumulative distribution functions — the CDF of a mixture of Gaussians, or weighted sums of logistics — because every CDF rises from 0 to 1 by construction.

Composition closes the system neatly: apply one transformation, then another, then another — and the entire chain can be replaced by a single equivalent transformation (the composition of the parts, whose slope is the product of the individual slopes). Flows obey composition, which is why stacking invertible layers stays principled: depth adds expressiveness without ever endangering reversibility.

6.9.8 Watching a Flow Learn: Worked Walkthrough

Make the abstraction concrete with the picture sequence. Begin with the raw training data and its histogram — a lumpy empirical distribution, perhaps with scattered sample dots around it. Pick an arbitrary starting transformation $f_{\theta }$. Push the histogram through it: the transformed histogram comes out misshapen — not Gaussian yet, just differently lumpy.

Now run the training procedure from Section 6.9.6 with $p_Z$ fixed as a standard Gaussian. Iteration by iteration, gradient steps reshape $f_{\theta }$ wherever the objective complains: points pushed into low-latent-density zones drag on the first term; over-compressed stretches drag on the slope term. Afterward, pushing the original data through the learned transformation produces a histogram closely matching a Gaussian bell — the learned transformation has bent the data's law onto the target law.

Swap the target to a uniform distribution and repeat: the starting transformation tunes itself into a different final shape, and the empirical distribution of $z$ flattens to uniformity. Generation completes the loop: sample $z$ from the tidy latent law — this sample is your latent-space draw — pass it through the inverse flow, and a brand-new $x_{\theta }$ appears, distributed like the training data. In one sentence: a trained flow reshapes the empirical histogram toward the chosen latent law, then runs backward to create.

6.9.9 CDF Parameterization and Layer Choices

One observation ties the pieces together elegantly. Parameterize the flow directly as a cumulative distribution function — let $f_{\theta }$ itself be a parameterized CDF, such as the CDF of a mixture of Gaussians or of logistics. The derivative of a CDF is the PDF itself:

$$\frac{d}{dx}{F}_{\theta }(x)=p_{\theta }(x)$$

so fitting the parameterized CDF as a flow recovers the original objective of fitting the corresponding parameterized PDF — the two views coincide, one object seen from the derivative's angle or the integral's. Practical instantiations use exactly those families: Gaussian-mixture CDFs and mixtures of logistics, both monotonic (every CDF is) and both differentiable, so they are admissible flow layers out of the box.

Layer selection obeys two hard requirements. Every transformation must be invertible — otherwise one latent sample spawns several outputs and generation loses meaning. And every transformation must be differentiable — without derivatives there is no slope term, and without the slope term there is no gradient training at all. Happily, the everyday activations qualify: sigmoid, hyperbolic tangent, and ReLU are all monotonic, so they are safe to invert — a sigmoid maps each input to exactly one output, tanh likewise, and ReLU (with a strictly positive slope on its active side) survives with the usual care at the hinge. What fails is anything quadratic-shaped: $x^2$, $x^4$, $x^8$ curves fold inputs and are barred.

Where next: the following session extends flows to two dimensions and n dimensions, covers dequantization — the trick that ports this continuous formulation onto discrete images and text — and a revision session follows. Later, in the application classes on vision and natural language processing, autoregressive models return: pre-trained autoregressive models are the basis behind GPT and the whole family of ChatGPT-style systems.

Real-world: one-dimensional flows are not merely a teaching device — density modeling of scalar quantities is a real task in quantitative finance (modeling daily return distributions, whose fat tails defeat single Gaussians) and in anomaly detection on sensor streams, where the learned density flags readings the law finds improbable. And the composition-of-invertible-layers principle scales straight up to the image and audio flow models used for high-fidelity speech synthesis.

Exam Guidance Summary

• Assignment-anchored questions. Mid-sem questions may be framed around the assignment — expect about two. No code writing; you must read a snippet and explain what it does, drawing only on the lab sheets and the assignment. Nothing from untaught territory — the questions sit around the assignment, not on unseen material.
• Deep-learning prerequisites to retain by memory (from the earlier coursework): backpropagation, weight updates via gradient descent, small convolutions computed by hand or with a calculator, and parameter counting for convolution layers. For the last one, rehearse the pattern: filters × kernel height × kernel width × input channels weights + one bias per filter.
• Assignment observations must tally with known results. Alignment of your empirical findings with what the technical community already knows is part of the grading expectation — check your numbers against published behavior before submitting.
• Practice material. Past question papers will be shared shortly; supplement them with ChatGPT-generated practice questions, which work well for extra drilling.
• Know the NLL leaderboard cold, including why smaller is better: baseline about 3.14, Gated PixelCNN 3.03 on CIFAR-10, PixelCNN++ 2.92, PixelSnail lowest of all. Expect conceptual questions on why each variant improves: blind-spot removal (vertical/horizontal split), mixture-of-logistics outputs (smooth low-parameter pixel laws instead of 256-way softmax), and masked attention (full receptive field).
• Change of variables and the flow objective. Understand $p_{\theta }(x)=p_Z(f_{\theta }(x))|\partial f_{\theta }/\partial x|$ and the two-term objective well enough to explain each symbol — the slope term and the latent-density term carry distinct roles: one rewards landing data where the latent law is generous, the other punishes compressing mass away.

A suggested revision order: re-derive the chain-rule factorization and NLL from scratch; sketch the blind-spot staircase and its vertical/horizontal fix; compute one discretized-mixture probability end to end; then derive change of variables once without notes.

Key Industry Applications

• Cross-domain generation. Audio synthesis, language modeling, and image and 3D generation all run on the same autoregressive recipe — factor the joint into per-position conditionals and sample sequentially. The recipe transfers across dimensionalities, from one-dimensional waveforms to volumetric data. The same family underlies modern text generators: pre-trained autoregressive models power GPT-class systems and ChatGPT-style applications.
• Class-conditional image generation. One-hot label conditioning on MNIST-style digits produces on-demand samples of any chosen class — the mechanism behind controllable synthesis tools that must honor a user's category request.
• Super-resolution. Condition a PixelCNN on a seven-by-seven averaged embedding of a $28\times 28$ image (each entry the mean of one four-by-four block), and it synthesizes the full-resolution version. Any domain with paired low-res/high-res data supports the same trick — satellite imagery, medical scans, legacy photo restoration.
• Colorization. Conditioning on grayscale inputs lets the trained density paint plausible colors, the same conditional law applied to archival footage restoration and old-photo colorization services.
• Study tooling. Multi-part written walkthroughs such as the thomasjubb.blog series, and step-by-step chatbot prompts ("explain PixelCNN and its variant architectures step by step"), both serve as supplementary references alongside the formal material.
• Frontier flows. TARFlow — a Transformer-based autoregressive flow from Apple, circa 2024 — shows the flow family competing on generation quality, narrowing the historic sample-quality gap that kept flows behind later model families.

The common thread across every row: one probability-law learner, steered or scaled by what you feed it — labels, low-res pairs, grayscale twins, or attention instead of convolutions.