Normalizing Flows: Masked Autoregressive Models

These notes cover the flow-based branch of unsupervised deep learning: how masking works in autoregressive models, why flow models exist, how change of variables drives their training, and how specific architectures — MAF, IAF, affine flows, NICE, RealNVP, checkerboard-masked couplings, and Glow — trade sampling speed against likelihood evaluation. Two full exam-style worked problems on autoencoders close the notes.

The chapter builds in one straight line. First comes the motivation: autoregressive models generate data one coordinate at a time, and that serial bottleneck is what flows attack. Then comes the framework: a simple base distribution, an invertible transformation, and a change-of-variables rule that ties the two densities together. On top of that framework sit the concrete designs — masked autoregressive flows (fast at scoring data, slow at drawing samples), inverse autoregressive flows (exactly flipped), and the coupling-layer family from NICE through RealNVP to Glow. Two practical matters complete the theory: digital data must be dequantized before flows can touch it, and the trained models power super-resolution, text, audio, and 3D point-cloud generation. The final two sections are hands-on exam practice grounded in the coding assignments: tracking output sizes and parameter counts through a convolutional autoencoder, and running a full backward pass on a tiny binary autoencoder.

8.1 Masking and Autoregressive Modelling Recap

The class picks up where the previous session ended: masked auto-regression, masked autoregressive flow, and inverse autoregressive flow. One specific implementation of masked autoregressive flow is called MAF, and a companion implementation is the inverse autoregressive flow, IAF. The Eric Jang tutorial is the recommended reference for a detailed walkthrough of both.

Before any new machinery appears, two ideas from the previous session need to be firm, because everything in this lecture stands on them: the causal mask that lets a model predict one coordinate at a time, and the price that idea pays at generation time. This section refreshes both and then asks the question that motivates the whole chapter — why build yet another model at all?

8.1.1 Raster Scan Ordering and the Causal Mask

The basic property of masked auto-regression is simple: every predicted value must depend only on values that already happened in the past, never on data from the future. Masking is one way to enforce this. In PixelCNN, for example, a binary mask hides parts of the input so that when we estimate the probability density of one particular pixel — one feature dimension — we can only use the data that came before it.

What counts as "past" and "future" is a convention. Image data is read in raster scan format: left to right across the top row, then down a row, left to right again, and so on until the bottom-right corner. Under that ordering, a convolution estimating the density at some pixel may look at every position in previous rows and at positions to the left in the same row, and nothing else. Any position that comes later in the scan is future, and the mask blocks it.

A concrete pass makes the pattern visible. Take a 4 by 4 image and number its sixteen pixels in scan order, 1 through 16:

Suppose the model wants the density of pixel 7 (row 2, column 3). The mask leaves six positions visible: the whole first row (pixels 1–4) and the two same-row neighbours to the left (pixels 5 and 6). The remaining ten positions — including every pixel below and to the right — are future, so their values are hidden. Move the target one step down to pixel 11 and the visible set grows to ten positions; move it anywhere and the rule never changes: all fully scanned rows, plus the part of the current row left of the target.

The same causal idea carries straight over into flow models. An autoregressive component conditions each output dimension on the dimensions before it, whether the component is predicting pixels directly or producing the parameters of a flow transformation. That single sentence is the seed of MAF later in this lecture: keep the mask, change what the masked network outputs.

8.1.2 Why Build Another Model?

The autoregressive setup views training data as an n-dimensional vector $x=(x_1,x_2,\dots ,x_n)$, and the goal is to estimate the density $p(x)$. An autoregressive module can do this through the chain rule of probability: factor the joint density into conditional densities along the scan order,

$$p(x)=p(x_1)p(x_2\mid x_1)p(x_3\mid x_1,x_2)\cdots p(x_n\mid x_1,\dots ,x_{n-1}),$$

and let a masked network produce each conditional. As a density estimator this works well. So a natural question arises: if we already have an autoregressive model, what motivates yet another model?

Several students guessed gradients before the real reason surfaced, and the rejection is worth remembering as a habit: when someone proposes a new architecture, hunt for the deficiency it removes, not for a fashionable training pathology. Here the deficiency has a name — sequential data generation — and it is structural, not numerical. No amount of ReLU units or normalization layers removes it, because it lives in the factorization itself: $x_i$ needs $x_{i-1}$, so sampling must walk.

So the primary purpose of flow models is to improve data generation speed. There is a secondary purpose as well: flow models are designed with the hope of also producing a latent representation of the data. The data you observe is $x$, but something hidden may be generating it — those hidden values are the latents. In an autoencoder, the encoded dimensions are latent data: not observable directly, created by processing, and then used for decoding. Latent variables of this kind are absent from plain autoregressive models, so the flow model chases both goals at once: faster generation and a latent space.

Scope check on the promise: the speed gain is real but conditional on design choices, and the rest of the lecture shows the trade-off precisely. Masked autoregressive flows keep the slow sequential walk in the generative direction; inverse autoregressive flows push the slowness into evaluation instead; coupling architectures such as RealNVP and Glow make both directions parallel by giving up full autoregression. No free lunch appears anywhere — each design moves the bottleneck somewhere it hurts less for the intended use.

Real-world anchor: the sequential-decoding latency is not hypothetical. WaveNet produced striking audio quality but generated one audio sample after another, which pushed the follow-up system Parallel WaveNet toward exactly the flow ideas introduced here. Whenever a product needs many samples fast — image editing assistants, speech synthesizers, content filters — serial generation is the wall, and flows are one of the standard ladders against it.

8.1.3 Exam Framing for This Topic

8.2 The Normalizing Flow Framework

8.2.1 Base Distribution, Transform, and Sampling

A flow model is defined by three pieces. First, the original data $x$, an n-dimensional training sample. Second, a transformation $f$ applied to the data, written $z=f(x)$, which produces the latent space $z$. Third, an assumption about the latent distribution: $p_Z(z)$ is simple — uniform, or more naturally a normal Gaussian.

Generation runs the transformation backwards, and two quick questions from class pin down the setup before any formulas appear.

Training is done so that applying $f^{-1}$ to a latent sample creates a sample in the original space which obeys the data distribution. The means of achieving this is maximum likelihood: maximize the likelihood of the generated data. Both autoregressive models and flow-based models are likelihood-based models — the shared principle is that maximizing the likelihood of the data teaches the model the right thing. Section 8.3 turns that sentence into an explicit loss function.

There is one hard constraint on the transformation. When you sample from the $z$ space and invert it, you must get one result, not many. The forward function must be bijective — one-to-one.

An everyday analogy keeps the geometry honest. Think of kneading dough with a rolling pin. Pressing and stretching moves the dough around and reshapes each region — thick spots flatten out, narrow spots spread — yet the total amount of dough never changes, and you can always imagine rolling time backwards to undo every motion. A flow transformation does to probability mass what the rolling pin does to dough: it relocates and reshapes a fixed total mass of probability. The analogy breaks exactly where the mathematics is strict: real dough tears and folds over itself, while an invertible transformation may never fold — folding is precisely the failure that squaring exhibits.

Visual intuition: picture the standard-normal bump as a smooth hill over the latent axis on the left. The transformation bends that axis like a flexible ruler — compressing some stretches, extending others — and on the right the hill reappears as a lumpy, multi-peaked landscape over data space. Arrows run both ways along the bent ruler: forward arrows generate samples (the generative direction), backward arrows take a real data point back to its latent coordinate. That backward trip earns the framework its name — pushing complex data densities through the inverse layers gradually normalizes them back to the Gaussian, giving us normalizing flows.

This is also why latents echo the autoencoder idea: hidden codes created by processing rather than observed directly, except here the transform is built to be invertible. The practical payoff of an exact, invertible latent code is a model that can score any data point with its true probability — the property behind anomaly detection systems that flag inputs receiving unusually low likelihood under a flow trained on normal data.

8.2.2 The Jacobian Constraint in Many Dimensions

In one dimension, finding $f^{-1}$ is no big deal as long as the mapping is bijective. Many dimensions change the picture. For, say, 100-dimensional data, the inverse operation becomes a question about the Jacobian matrix of the transformation.

Why does this matrix appear the moment dimensions grow? Because in one dimension the derivative $\mathrm{d}z/\mathrm{d}x$ is a single number saying how much the map stretches near a point. In many dimensions the stretch depends on direction: a map can double lengths horizontally while halving them vertically. The Jacobian gathers those per-direction stretch factors, and its determinant condenses them into one number — the factor by which the map scales area (two dimensions) or volume (in general) at that point. Section 8.3 shows that this volume factor is exactly the correction term the change-of-variables rule needs.

Two facts make triangular matrices the hero. First, the determinant of a triangular matrix is just the product of its diagonal entries. For example,

$$\det \left[\begin{array}{cc}2& 5\\ 0& 3\end{array}\right]=2\cdot 3-5\cdot 0=6,$$

with no work beyond multiplying the diagonal — and the same shortcut holds at any size, turning a 100-by-100 determinant from an expensive computation into one hundred multiplications. Second, a triangular matrix is invertible by back-substitution in about $O(n^2)$ steps whenever no diagonal entry is zero, and its determinant is nonzero exactly when all diagonal entries are nonzero. One structure buys tractability and the guarantee of a valid inverse at the same time.

So the two key requirements for building masked autoregressive flow models are: the determinant of the Jacobian of the transformation from $x$ to $z$ must be easy to compute, and the triangular shape guarantees both tractability and a nonzero determinant. Keep these two requirements in view — every architecture in Section 8.6 is a different trick for satisfying them.

8.3 Change of Variables and Maximum Likelihood Training

8.3.1 Scaling, Shifting, and Skewing a Uniform Distribution

Before the general formula, consider what simple transformations do to a distribution — every intuition needed later lives in these small examples.

Take uniform training data on the interval $[0,1]$ — flat density at height 1, so the shaded rectangle has area $1\times 1=1$ — and apply

$$f(x)=2x+1.$$

Every point moves to twice its position plus one, so the support stretches from $[0,1]$ to $[1,3]$: the square-shaped region of support becomes rectangular, stretched to twice the length. The distribution remains uniform, but the density value changes: it was flat at 1 across the original interval, and now it is flat at one half across $[1,3]$, because the same probability mass is spread over twice the range. The area check confirms nothing was created or destroyed:

$$\text{height}\times \text{width}=\frac{1}{2}\times 2=1.$$

Scaling stretches support and compensates by lowering height — double the range, half the height; triple the range, a third of the height. A pure shift, say $f(x)=x+b$, moves the interval without changing widths at all, so heights stay exactly where they were — sliding a rectangle sideways changes no areas.

A more general map fixes neither width nor straightness: it alters the shape of the distribution. A skewed transformation bends the flat uniform into a lopsided hump by compressing some regions (piling mass up into taller spikes) while stretching others (spreading mass thin). Rotations act on many dimensions at once: they stretch one axis and squeeze another while translating along both axes, and the result stays a valid probability density — the area under the curve remains the same after integration, because both sides of the transformation are probability distributions. Probability mass may be pushed around, but never manufactured or discarded.

Visual intuition: draw the before-picture as a unit square of density over the axis, and the after-picture as a wider, shorter rectangle of equal area sitting further along the axis. Mark the two landmarks: the leading edge moved from 1 to 3 (the stretch), and the plateau dropped from 1 to ½ (the compensation). Every change-of-variables computation later is bookkeeping for exactly this trade.

8.3.2 The Change-of-Variables Formula

The verbal rule from class: the density of $x$ equals the density of $z$ times the absolute value of the determinant of the derivative of $z$ with respect to $x$. In higher dimensions:

Where does such a formula come from? In one dimension it follows from conserving probability mass, and writing the argument once removes all mystery. Let $z=f(x)$ be invertible with inverse $x=f^{-1}(z)$... more usefully, track a tiny interval: if $x$ sits in a window of width $\Delta x$, then $z=f(x)$ sits in the image window of width $\Delta z\approx \left|\frac{\mathrm{d}z}{\mathrm{d}x}\right|\Delta x$. The same probability mass must occupy both windows:

$$\begin{array}{rl}{p}_X(x)\Delta x& \approx p_Z(z)\Delta z\\ & =p_Z(z)\left|\frac{\mathrm{d}z}{\mathrm{d}x}\right|\Delta x\end{array}$$

Divide both sides by $\Delta x$ and let the window shrink to zero; the approximation becomes equality and the one-dimensional version of the rule appears:

$$p_X(x)=p_Z(f(x))\left|\frac{\mathrm{d}z}{\mathrm{d}x}\right|.$$

The multivariate formula above is the same statement with the stretch factor in each direction replaced by the single number that summarizes stretch in all directions — the absolute determinant. This is why the determinant matters: it captures how area (in two dimensions) or volume (in general) changes under the transformation. To create $x$ from $z$, you must be able to calculate this determinant. The absolute value appears because probability values are always greater than zero, so the density ratio must stay positive even when the transformation flips orientation (a reflection has negative determinant but still only reshuffles mass).

A quick sanity pass on the formula using the warm-up example: there $z=2x+1$, so $\frac{\mathrm{d}z}{\mathrm{d}x}=2$ and the rule reads $p_X(x)=p_Z(z)\cdot \frac{1}{2}$... careful — direction matters, and this is the classic trip-up. If instead we transform from the uniform to data via $x=g(z)$ with $g(z)=2z+1$, then $\frac{\mathrm{d}x}{\mathrm{d}z}=2$ and $p_X(x)=p_Z(z)\left|\det \frac{\partial z}{\partial x}\right|=1\cdot \frac{1}{2}=\frac{1}{2}$, matching the halved height found by hand. Stretching output space by 2 divides the density by 2 — spread twice, height halves.

With trainable parameters, the transformation $f_{\theta }$ is implemented by a neural network, and the model density becomes:

$$p_{\theta }(x)=p_Z(f_{\theta }(x))\left|\det \frac{\partial f_{\theta }(x)}{\partial x}\right|.$$

This is the basic change-of-variables formula for probability distributions: a random variable is transformed into another random variable, and the distribution attached to the original variable changes for the transformed one.

Notation note: reference texts often write the same rule in the sampling direction as $\mathrm{Pr}(x)={\left|\frac{\partial f[z]}{\partial z}\right|}^{-1}\mathrm{Pr}(z)$ with $x=f[z]$. The two forms agree, because inverting a map reciprocates its Jacobian determinant: $\det \left(\frac{\partial z}{\partial x}\right)=1/\det \left(\frac{\partial x}{\partial z}\right)$. The lecture keeps the normalizing direction ($z=f(x)$), which matches what training actually computes.

The autoregressive constraint now earns its central place. Because $z_i$ may depend only on $x_1,\dots ,x_{i-1}$, changing $x_j$ with $j\ge i$ cannot affect $z_i$ at all, so the partial derivative $\partial z_i/\partial x_j$ is zero whenever $j\ge i$. Filling row after row with zeros on and above the diagonal leaves the Jacobian lower triangular,

$$\frac{\partial z}{\partial x}=\left[\begin{array}{cccc}\ast & 0& 0& \cdots \\ \ast & \ast & 0& \cdots \\ \ast & \ast & \ast & \ddots \\ ⋮& ⋮& ⋮& \ddots \end{array}\right],$$

(where $\ast$ marks possibly-nonzero entries), its determinant is trivial to evaluate — multiply the diagonal — and the determinant is nonzero by construction as long as each diagonal entry $\partial z_i/\partial x_i$ is nonzero, which the flow's design ensures. Without this trick, evaluating the determinant of a large Jacobian carefully would not even be feasible — that is the whole point of putting the autoregressive structure into the flow model.

8.3.3 Maximum Likelihood Training and Composition

Training uses the principle of maximum likelihood. Assuming data points are independent and identically distributed, the data likelihood is a product of probabilities:

$$\prod _{i=1}^I{p}_{\theta }(x^{(i)}),$$

where $x^{(1)},\dots ,x^{(I)}$ are the $I$ training points. Products are difficult to deal with — gradients of long products misbehave numerically — so we take the logarithm, natural log throughout, and the product becomes a sum. Maximizing $\log p_{\theta }(x)$ is equivalent to minimizing its negative, averaged over the data:

Applying the same logarithm to the change-of-variables expression splits the per-point loss into two pieces, and writing the split line by line shows exactly what the network must supply:

$$\begin{array}{rlrl}-\log p_{\theta }(x)& =-\log \left[p_Z\right(f_{\theta }(x)\left)\right|\det \frac{\partial f_{\theta }(x)}{\partial x}\left|\right]& & \text{(substitute change of variables)}\\ & =-\log p_Z(f_{\theta }(x))-\log \left|\det \frac{\partial f_{\theta }(x)}{\partial x}\right|& & \text{(log of product = sum of logs)}\end{array}$$

So each training point costs two terms: the log-density of the base distribution evaluated at $f_{\theta }(x)$, plus the log of the absolute Jacobian determinant. (Some texts display the second term with a minus sign; that happens when the map is written in the sampling direction $x=g(z)$, where the determinant reciprocates — same quantity either way.) The first term pulls latent coordinates toward regions the base likes; the second term rewards transformations whose local volume change explains how the data spread out. The expectation over generated data is computed on batches: the loss for one step is the sum of $-\log p_{\theta }(x)$ over all training points in the mini-batch, and minimizing that sum drives learning. Batch processing exists for computational efficiency — GPUs chew through many points at once, and an average over a batch estimates the full-data expectation.

One practical warning before building anything:

Flows also compose. Starting from $z_0$, apply one invertible map to reach $z_1$, another to reach $z_{k-1}$, and a final one so that $z_k=x$. Why do determinants multiply down the chain? Each stage has its own Jacobian $J_i=\partial z_i/\partial z_{i-1}$, and the chain rule stacks them:

$$\frac{\partial z_k}{\partial z_0}=J_k{J}_{k-1}\cdots J_1,$$

so taking determinants and using the fact that the determinant of a matrix product is the product of the determinants:

$$\det J=\det J_1\cdot \det J_2\cdots \det J_k.$$

Each stage must itself be invertible. Practically this means you can use a multilayer network: compute the determinant locally at each layer, multiply the results, and you have the likelihood of the whole stack. Numerical spot-check with one-dimensional stretches: a stage that scales lengths by 2 contributes Jacobian determinant $2$, a stage that scales by 3 contributes $3$, and the composite stretches lengths by 6 — matching $\det J=2\times 3=6$. X flows to $F_1$, another flow applies to $F_2$, another to $F_3$ — composition is free as long as every piece is invertible.

That freedom is what makes deep flow networks possible: stack simple invertible layers whose determinants are known in closed form, multiply their contributions inside the log-likelihood, and train the whole thing end to end with backpropagation.

Real-world anchor: this exact objective — minimize mean negative log-likelihood over batches — is the same training loop used by language models and diffusion models' precursors alike. Learning it here transfers directly: wherever a system reports "loss: 3.2 nats", it is optimizing a formula shaped like the one above.

8.4 Masked Autoregressive Flow (MAF)

8.4.1 Generative Equations and Sequential Sampling

The masked autoregressive flow generates each dimension from a latent sample using a shift scaled by an exponential. In class this was stated as: the generated value equals $z_i$ times $e$ to the power ${\alpha }_i$, plus ${\mu }_i$, for $i$ running from 1 to n:

Concretely: $x_1=\exp ({\alpha }_1)z_1+{\mu }_1$. To calculate $x_2$, you need $\exp ({\alpha }_2)z_2+{\mu }_2$, and the network predicts ${\mu }_2$ and ${\alpha }_2$ using the information about all the data that has happened in the past — namely $x_1$. Generally speaking, the network predicts ${\mu }_i$ and ${\alpha }_i$ as functions of $x_1$ through $x_{i-1}$:

$${\mu }_i={\mu }_i(x_1,\dots ,x_{i-1}),{\alpha }_i={\alpha }_i(x_1,\dots ,x_{i-1}).$$

Walk through the consequence: generation of $x_2$ depends on $x_1$; generation of $x_3$ depends on $x_1$ and $x_2$, because ${\mu }_3$ and ${\alpha }_3$ are functions of them. So data generation time stays sequential — the very thing flows wanted to escape, kept alive inside MAF. Each coordinate waits for all earlier ones; a million-dimensional image means a million dependent steps at sampling time.

8.4.2 Likelihood Factorization and Conditionally Gaussian Dimensions

MAF rests on the chain-rule assumption that the joint density factors into conditionals along the scan order — stated loosely in class as "probability of x given by the probabilities of each $x_i$ given the previous ones":

$$p(x)=\prod _{i=1}^{n}p(x_i\mid x_1,\dots ,x_{i-1}).$$

This product form is the standard autoregressive chain rule from Section 8.1.2, and it must be a product rather than a sum because joint probabilities multiply: the chance that all coordinates take their observed values together is the chance that $x_1$ takes its value, times the chance $x_2$ adds its value given $x_1$, and so on. Sums appear one step later, when logarithms convert this product into the training loss — that is the only place addition enters.

Each conditional factor is a normal Gaussian — but not a fixed one. Its mean is a function of all the dimensions that came before, and its spread depends on those previous data points too:

Why is this the right conditional shape? Check it against the generative equation: if $z_i\sim \mathcal{N}(0,1)$ independently of everything else, then

$$x_i=e^{{\alpha }_i}{z}_i+{\mu }_i$$

is just a rescaled and shifted standard normal, so $x_i$ given $x_{<i}$ is Gaussian centred at ${\mu }_i(x_{<i})$ with standard deviation $e^{{\alpha }_i(x_{<i})}$ — exactly what the density above states. The two equations are two views of one design: the generative view samples, the density view scores.

So every individual dimension is Gaussian distributed, but the mean and variance of that Gaussian change from one pixel to the next. Letting both parameters drift with context is what gives MAF its pretty good modeling of the original data: flat regions get tiny variance (tight, confident conditionals), textured regions get large variance (wide, uncertain conditionals), and the mean chases local structure such as edges.

8.4.3 What MAF Is Fast At

Because the conditioning variables $x_1,\dots ,x_{i-1}$ are observed when you want to score a data point, MAF gives fast evaluation of $p(x)$ for arbitrary $x$ — the likelihood direction is cheap. Every ${\mu }_i$ and ${\alpha }_i$ depends only on inputs that are already sitting in memory, so a single pass through the masked network produces all parameters at once, then all $z_i=(x_i-{\mu }_i)e^{-{\alpha }_i}$ follow in parallel, and the change-of-variables machinery from Section 8.3 assembles the exact likelihood.

Sampling is the slow direction, since it must walk through the dimensions one at a time — the worked example above showed exactly why. The companion model flips exactly this trade-off, and it is next.

Real-world anchor: because exact, cheap density evaluation is MAF's strength, flows of this family fit naturally wherever "how normal is this input?" matters — industrial defect screening flags sensor readings whose learned likelihood drops below a threshold, and fraud pipelines score transactions the same way.

8.5 Inverse Autoregressive Flow (IAF)

8.5.1 Parallel Generation, Sequential Evaluation

The inverse autoregressive flow makes generation parallel, while training — fitting those transformation functions — stays slow and sequential. The trick is almost embarrassingly simple: IAF uses the same affine form as MAF, but lets the network read the latent sample instead of the data.

Start from the n-dimensional base distribution, sample $z_1$ through $z_n$ from a normal Gaussian in one shot, and compute every output at once:

The forward mapping from $z$ to $x$ is parallel: since every $z_j$ was drawn together at the start, all of them are available immediately, so the network calculates all the $\mu$ and $\alpha$ values in one shot from the sampled $z$'s — a masked computation tree exactly like PixelCNN's, but fed with latents — and then $x_1,x_2,\dots$ all appear together. One pass, one sample: generation no longer walks.

During training time, the reverse direction — producing $z$ from a given $x$ for likelihood evaluation — is sequential:

Computing $z_2$ requires ${\mu }_2$ and ${\alpha }_2$, and those depend on latents up to index $i-1$ — on $z_1$, not on $x_1$. But $z_1$ is unknown until you invert the first equation, so the inversions must line up in order: get $z_1$, feed it forward to get the parameters for level 2, get $z_2$, and so on. So IAF is faster to sample from, but slow to evaluate the likelihood for a given data point, and direct maximum-likelihood training inherits that slowness — every gradient step needs a likelihood, and every likelihood needs the sequential inversion walk.

8.5.2 Choosing Between MAF and IAF

The two models are inverses of each other, and their strengths mirror that fact.

Neither dominates; the application decides. If your product must draw samples under tight latency — speech responding in real time, images streaming into an editor — IAF's parallel sampling wins even though scoring is sluggish. If your system mostly scores incoming data and rarely samples — anomaly screening, calibration, compression — MAF's cheap exact likelihoods win. Pick by asking which direction your workload runs hot.

Real-world: models like Parallel WaveNet build on the IAF idea with a teacher-student arrangement — a slow but accurate teacher transfers its behavior to a fast parallel student — which was covered in the previous session. The teacher (a sequential autoregressive model) supplies training signal so the student never needs the expensive sequential inversions during learning; the deployed student inherits fast sampling. Real-world: this is the same tension behind ChatGPT producing images slowly: sequential decoding trades speed for autoregressive exactness, and IAF-style parallelism is one way out.

8.6 Flow Building Blocks from Affine Coupling to Glow

8.6.1 Affine Flows

The simplest concrete flow is the affine flow. In class: $f$ of $x$ is given by $A$ inverse of $x$ minus $b$ — all vectors and matrices:

Entries of the map look like $x_1=a_1{z}_1$ and $x_2=a_2{z}_1+a_3{z}_2$: each output mixes latent coordinates linearly. Pushing Gaussian data through this affine transformation yields a much more complex density for $x$ — described in class as being like a mixture of Gaussians. One clarification keeps the picture precise: a single Gaussian pushed through one affine map remains exactly a single Gaussian, but its shape changes in the ways that matter — the covariance becomes $A{A}^{\top }$, so previously independent coordinates become correlated and the cloud tilts, stretches, and rotates instead of staying round. The genuinely mixture-like densities appear once such maps stack, or once the base is not a lone Gaussian; the class description points at that richer end result. The multiplication by $A$ supplies scaling and rotation, while the addition of $b$ supplies translation — three geometric motions, one formula.

The Jacobian of the transformation $z=A^{-1}(x-b)$ is the constant matrix $A^{-1}$ — every partial derivative $\partial z_i/\partial x_j$ is just the corresponding entry of $A^{-1}$, independent of where you evaluate it — and the log-likelihood calculation involves the determinant of $A$. You had better be able to evaluate $\det (A)$ easily, or the whole system cannot be trained. A numerical taste: for $A=\left[\begin{array}{cc}2& 0\\ 0& 3\end{array}\right]$, $\det (A)=6$, so this map inflates areas six-fold and every density value it produces carries the correction factor $1/6$ in the normalizing direction.

Affine flows alone share the fate of linear maps everywhere: no matter how you stretch, rotate, and translate a Gaussian cloud, it stays a Gaussian cloud. They are indispensable as components — normalizing layers inside bigger stacks — but they cannot carve a Gaussian into a face by themselves.

8.6.2 Element-wise Flows and Why Mixing Matters

Element-wise flows are much simpler: coordinate $x_1$ maps only to $z_1$, $x_2$ only to $z_2$, and so on. Each dimension passes through its own invertible nonlinearity, with no cross-talk anywhere. The Jacobian $J=\partial z/\partial x$ is simply diagonal — all off-diagonal partials vanish because coordinate $i$ never feels coordinate $j$ — and you may apply any valid invertible mapping independently per coordinate. Its determinant is just the product of the diagonal entries, so even a deep stack of element-wise layers costs only per-coordinate multiplications.

Conceptually this is a perfectly legal flow — but people have observed that the quality of newly generated data is not that good. Element-wise transforms have limited value precisely because coordinates never interact: each output axis is a bent copy of exactly one input axis, so the model can reshape marginal distributions one at a time yet can never express relationships between dimensions — and structure such as "this pixel sits next to a dark edge" lives entirely in those relationships.

It is very important in flow modeling to mix different dimensions of the base distribution in order to model complicated training data distributions accurately. Everything that follows — coupling layers, masks, one-by-one convolutions — is machinery for mixing.

When to pick which: element-wise layers serve as cheap reshapers between mixing layers; affine layers normalize and correlate; coupling layers carry the real modelling load.

8.6.3 NICE: Additive Coupling Layers

NICE uses additive coupling layers. Suppose the total data dimension is n. Split the coordinates into two groups: dimensions 1 through $d$, and dimensions $d+1$ through $n$. The first group passes through untouched, and the second group gets shifted by a learned function of the first:

The inverse mapping is immediate — this is the beauty of addition: whatever was added can be subtracted again, because both sides know $x_{1:d}=z_{1:d}$:

$$z_{1:d}=x_{1:d},z_{d+1:n}=x_{d+1:n}-m_{\theta }(x_{1:d}).$$

No matrix inversion, no iteration — one forward pass of $m_{\theta }$ rebuilds the shift, and subtraction undoes the layer.

The Jacobian of the forward map is block triangular:

$$\frac{\partial x}{\partial z}=\left[\begin{array}{cc}{I}_d& 0\\ \frac{\partial m_{\theta }}{\partial z_{1:d}}& I_{n-d}\end{array}\right],$$

with an identity block in the upper-left, zeros in the upper-right, the derivatives of $m_{\theta }$ in the lower-left, and an identity block in the lower-right. Why do the derivatives of $m_{\theta }$ sit below the diagonal? Row block $d+1:n$ differentiates outputs $x_{d+1},\dots ,x_n$ against inputs; each such output depends on all inputs ($z_{1:d}$ through $m_{\theta }$, plus $z_{d+1:n}$ with derivative one), while rows of the top block never feel $z_{d+1:n}$. Taking the determinant along the diagonal blocks then gives

$$\det \frac{\partial x}{\partial z}=\det (I_d)\cdot \det (I_{n-d})=1,$$

and interestingly, in the Jacobian calculation you do not need to worry about the entries coming out of $m_{\theta }$ at all, because they sit below the diagonal — arbitrary networks may live there without touching the determinant. Nothing contracts or expands: this is volume preservation. The picture offered in class: take a square and press it sideways into a rhombus, like pushing a malleable block — the area stays the same, only the shape shears. Every NICE layer slides mass around at fixed volume, which is why the name reads nonlinear independent components estimation — it reshapes without inflating or deflating.

Still, NICE has good capability for creating high-quality new data, and face samples generated by NICE were shown as evidence. Depth compensates for each layer's modesty: shear after shear after shear accumulates into complicated global deformation, at zero determinant cost per layer.

A rescaling layer extends NICE beyond pure volume preservation. For the first block of dimensions, multiply by a learned scale:

8.6.4 RealNVP: Affine Coupling Layers

RealNVP — the real-valued, non-volume-preserving sibling of NICE — is the more practical version. The coupling becomes multiplicative rather than purely additive:

The first $d$ features remain an identity transformation from the original distribution side, and the coupling only lets dimensions $d+1$ through $n$ consume dimensions 1 through $d$ — so the construction stays autoregressive in spirit (later coordinates read earlier ones, never the reverse), and the Jacobian is again lower triangular. Writing it out makes the determinant fall out on its own:

$$\det \frac{\partial x}{\partial z}=\prod _{j=d+1}^n\exp \left(s_j\right)=\exp \left(\sum _{j=d+1}^n{s}_j\right),$$

because each coupled row's diagonal entry is the exponential scale acting on its own coordinate, and a triangular determinant multiplies diagonal entries. Because the scale enters through an exponential, the determinant can be greater than or less than one depending on the value of that exponential — positive sums of $s_j$ expand volume, negative sums contract it — and the map shrinks or expands volume freely, which is where the "non-volume preserving" name comes from. Training adjusts these scales per region, letting the model concentrate probability where data actually lives.

A common configuration splits the variables into halves, copying dimensions 1 through $n/2$ and coupling the remaining half, which reads as translation plus scaling and orientation of the base distribution.

8.6.5 Checkerboard Masks for Mixing Channels

Coupling layers expose a basic tension noted earlier: half the dimensions are merely copied, so within one layer nothing mixes across the whole set — the copied half just relays information unchanged. The fix used in CNN-based generators is to change the autoregressive pattern from channel to channel using binary masks. You are still copying — but because the pattern shifts across channels, information mixes in a different way each round.

Work through the four-by-four example from class. Number the sixteen positions of a 4 by 4 image row by row, 1 through 16:

Positions 1, 5, 9, 13 fall in the first processing group; 2, 6, 10, 14 in the next; 3, 7, 11, 15 in the third; and the remaining column-four positions complete the fourth group — four interleaved sets, each taking every fourth stop along the raster scan:

• Group 1 = {1, 5, 9, 13}, Group 2 = {2, 6, 10, 14}, Group 3 = {3, 7, 11, 15}, Group 4 = {4, 8, 12, 16}.

Treating each pixel as having one channel, the original 4 by 4 tensor becomes a 2 by 2 by 4 tensor — four channel slices, each holding a quarter of the positions. The channel-wise mask sets $B=1$ for the first half of the channels and $B=0$ for the second half, so one coupling pass transforms Groups 1–2 conditioned on Groups 3–4, and flipping the mask next layer swaps their roles. Stacking layers with alternating masks lets information travel between positions that any single layer kept apart — a pixel that was copied in layer one is transformed in layer two, and everything communicates over depth.

On full-resolution images this idea takes its familiar visual form: a binary mask $B$ holding only ones and zeros in a checkerboard pattern — alternating 1,0,1,0 and 0,1,0,1 rows, like a chessboard — so that adjacent pixels sit on opposite sides of the mask and swap roles between consecutive layers.

Specific checkerboard patterns have been published for 32 by 32 images of the CIFAR-10 kind, and $C$ denotes the number of channels — RGB gives three, and multiple CNN stages may carry many more channels, with mixing happening across those as well. Illustrations contrast good versus bad partitionings: a badly partitioned dataset entangles information that later layers struggle to untangle, while a well-partitioned one is far better, even if still not perfect. Masking of this checkerboard kind is quite important for flow-based image generation.

8.6.6 Glow and One-by-One Convolutions

Flow++ and especially Glow matter enough that you should go over them yourself. Glow comes from OpenAI, performs strongly, and introduces a different modification technique than the couplings above: one-by-one convolutions, which pull information from multiple feature maps and channels at once, restoring the mixing that plain couplings lack. Mechanically, a 1×1 convolution applies the same small invertible linear map at every pixel position — think of an affine layer (Section 8.6.1) running independently at each location, re-weighting the channel axis while leaving the spatial grid untouched. Alternating coupling layers with these channel-shuffling convolutions means every half of the features eventually conditions on every other half.

Glow's signature demo is interpolation. Start from one image and travel toward another in latent space; intermediate points are coherent blends. Because you can move intelligently among base-distribution samples, you can also edit attributes: create smiling faces from a neutral portrait, make hair blonder, morph one face smoothly into another — animations shown in class even included celebrity likenesses, including the technique's inventor and a well-known film star. Very realistic new data and meaningful modifications come out of this technique.

Real-world anchor: the coupling-layer recipe powers production-grade image systems — GLOW-family models generate high-fidelity faces, audio variants (WaveGlOW-style speech synthesis) reuse the same invertible blocks on spectrograms, and the interpolation trick behind celebrity-morph demos is now standard footage in media-editing pipelines.

8.7 Discrete Data and Dequantization

8.7.1 The Continuity Assumption Breaks on Digital Data

Everything in flow modelling rests on change of variables for random variables, and every partial derivative inside that machinery is a rate of infinitesimal change. The quiet assumption is that random variables are continuous — that between any two possible values lie infinitely many others, so slopes and volume factors are well defined everywhere. Practical data usually is not: unless you process analog signals from analog sensors, inputs such as images are digital, made of discrete sample values. An 8-bit pixel admits exactly 256 levels; a value of 137 is an atom, not a point on a continuum, and "just below 137" describes no pixel at all.

When this inherent continuity assumption is violated, training shows it immediately: the loss curve against training progress becomes erratic and unstable. The mechanism is worth one paragraph. A density over discrete values is a row of spikes with empty space between them; its logarithm lurches from very negative to extreme whenever the model shifts probability onto or off an atom, so gradients swing wildly from step to step. Continuous densities, by contrast, spread mass smoothly, and the same optimization settles into a steady descent.

Quantized data produces rough, unusable optimization behavior; the same pipeline on smoothed data settles down. Class demonstrations showed exactly this contrast on side-by-side training runs.

8.7.2 Dequantization: Adding Uniform Noise

The standard repair is dequantization. Draw noise uniformly over the D-dimensional unit cube, where D is the dimension of your data, and add it to every training value:

Why uniform noise does the job: each discrete level $k$ was an atom carrying all the mass at one point; adding $u$ smears that mass evenly across the interval $[k,k+1)$, turning the spike into a smooth plateau. No information about which level a sample came from is destroyed — the integer part of $\tilde{x}$ recovers it — yet the input now lives on a continuum where derivatives exist. Adding this uniform noise to the training values makes the data continuous again, and the whole flow apparatus applies cleanly.

Class demonstrations showed discrete data trained poorly while dequantized data trained stably — the fix costs one line of code and is standard practice in every serious image-flow implementation, including Glow's own training recipe.

8.8 Applications of Flow Models

8.8.1 Vision: Super-Resolution

SRFlow applies flow models to super-resolution: starting from a small image, it creates a big image. A plain regression decoder maps the low-resolution input to one fixed high-resolution guess, averaging away everything uncertain. SRFlow instead learns the conditional distribution of the large image given the small one: sampling walks through the invertible latent space, and different draws yield different plausible enlargements — sharp texture here, a slightly different strand of hair there. The invertible latent space gives control that plain regression decoders lack: by choosing where in the latent space to sample, the user steers which plausible version gets rendered, instead of accepting the single blurred average.

8.8.2 Text, Audio, and 3D Point Clouds

Flows also do well in text synthesis when you manipulate the latent with suitable assumptions on the latent dimensions — moving along chosen directions changes attributes such as sentence style while keeping content anchored. In audio synthesis, flow-based vocoders generate raw speech waveforms in parallel passes rather than sample-by-sample, attacking exactly the sequential-latency wall that motivated flows at the start of this lecture. Point-cloud generation matters particularly for 3D computer vision: autonomous-driving pipelines need thousands of plausible LiDAR-style point sets for simulation and testing, and invertible models can both synthesize new clouds and score real ones for anomalies.

Plenty of further detail exists; if these techniques touch your workplace, you will pick up the specifics from the principles covered here. The pattern to carry forward is constant across all four domains: an exact likelihood makes scoring possible, and an invertible latent makes generation controllable.

8.9 Exam Worked Example: Convolutional Autoencoder Code Analysis

A new exam angle deserves attention: questions grounded in the coding assignments. Nobody expects you to write pipelines of code or memorize syntax. What is tested is whether you understand the libraries you used, the arguments of the functions you called, what each argument does and does not do, and what changes if you edit one line. Such questions will be based on the assignment you already completed — not on, say, the autoregressive implementations covered elsewhere, which stay numerical.

The running example is the encoder of a convolutional autoencoder, roughly:

Conv2D(32, (3, 3), padding="same", activation="relu", input_shape=(28, 28, 3))
MaxPooling2D((2, 2))
Conv2D(64, (3, 3), padding="same", activation="relu")
MaxPooling2D((2, 2))
Flatten()
Dense(30, name="latent")

Reading the arguments before computing anything: 32 is the number of filters; (3, 3) is the kernel size; padding="same" keeps spatial dimensions unchanged by surrounding the input with zeros; activation="relu" applies the rectifier to every output feature map; and input_shape=(28, 28, 3) declares 28-by-28 images with three channels — matching the three-channel input count that the parameter arithmetic below uses.

8.9.1 Layer-by-Layer Output Sizes

The complete arithmetic deserves its own worked pass, step by step:

Two habits earn marks here: track all three numbers (height, width, channels) at every row, and know which argument changed what — pooling touches only the spatial sides, convolution depth comes only from the filter count.

8.9.2 Role of the Latent Layer: Thirty Units Versus More

Push the thought to its limit: if the latent held as many units as the flattened layer, you would get perfect reconstruction, because no information is lost — but the latent space would be junk, giving you nothing usable for representation learning. The bottleneck is the point: forcing 3136 values through 30 numbers compels the network to spend its capacity on the regularities that matter — strokes, shapes, parts — which is precisely what makes the code reusable for downstream tasks such as classification or search. Compression here is not storage-saving; it is curriculum for the network.

8.9.3 Counting Trainable Parameters

Each filter slides across the whole image, but it always reads all three input channels at once — that is why the channel count enters per filter, not once overall. The general counting rule for a convolution layer is

$$(k_h\cdot k_w\cdot C_{\text{in}}+1)\cdot C_{\text{out}},$$

where $k_h,k_w$ are the kernel height and width, $C_{\text{in}}$ the input channel count, and $C_{\text{out}}$ the filter count; the "+1" per filter is the bias, present when the layer's bias option (use_bias) is switched on. One clarification settles a remark from class: the bias belongs to the layer's bias setting, not to the activation choice — turning activation from "relu" to "sigmoid" changes no parameter count, while disabling biases removes exactly $C_{\text{out}}$ of them.

Applying the same rule to the second layer:

$$(3\cdot 3\cdot 32+1)\cdot 64=289\cdot 64=18496.$$

For the Dense layer: $3136\cdot 30+30=94110$ — weights plus one bias per output unit. Summing the encoder: $896+18496+94110=113502$ trainable parameters, with pooling and flatten contributing none. Notice where the mass sits: over ninety percent of the encoder's parameters live in the final Dense layer, because it connects every value of a large flattened vector to every latent unit — a useful intuition when deciding whether shrinking feature maps early saves real compute.

8.9.4 Removing Both Max-Pooling Layers

The sixteen-fold figure deserves its own emphasis because it is the classic trap: seeing "two pools removed" tempts a factor of four, but each pool halves both dimensions, so each removal contributes $2\times 2$, and factors multiply down the stack. The Dense parameter count moves in lockstep — roughly $50176\times 30+30=1505310$ against $94110$ before, about sixteen times larger — so memory and training time balloon together with the harder compression task.

8.9.5 Wider Filters: Parameters, Training Time, and Receptive Field

Consider the variant encoder with a 32-filter convolution followed by a 64-filter convolution, and grow every kernel from 3 by 3 to 7 by 7.

The ring-counting rule behind those numbers: each new convolution widens the view by one ring of $(k-1)$ pixels per side, so receptive field grows as $1+(k_1-1)+(k_2-1)+\cdots$. Two 3-by-3 layers: $1+2+2=5$. Three: $1+2+2+2=7$ — matching one direct 7-by-7 layer, which is why small kernels stacked deep are usually preferred to single huge kernels: same view, far fewer parameters, plus an extra nonlinearity in between.

8.9.6 Filter Size Versus Image Content: MNIST Against CIFAR

How would you check such a statement seriously? Not as a classroom exercise but as a working engineer with a client image: run a controlled experiment, compare reconstructions, and decide which filter suits which material. That is the real context of the assignment pairing: one dataset of MNIST-style digits and one of CIFAR-10 images. CIFAR images carry far more intricacy than digits. The educated guess offered: on CIFAR, 7 by 7 filters will make things worse, while on MNIST they may improve reconstruction — because MNIST has little high-frequency content, and large smooth strokes suit large filters. Test it yourself.

Another self-test: shrink the 28 by 28 images to 7 by 7 and see whether you can still make sense of them. On digits you can probably still tell a 2 from an 8; on CIFAR you likely cannot identify much at all. That asymmetry mirrors what big receptive fields do to fine detail — averaging windows wipe out exactly the small-scale structure that CIFAR lives on. Expect higher reconstruction error for large filters on CIFAR, and the opposite possibility on digits.

8.9.7 Overfitting in Autoencoders and Split Discipline

The correction matters because autoencoders make overfitting unusually sneaky: unlike a classifier, there is no accuracy number staring at you, and a beautiful reconstruction montage can be pure memorization. The honest signals live in the split comparison — train loss falling while validation loss turns upward is the classic crossing pattern to catch early.

Exam grades aside, this is the habit that matters in the real world: getting exam numbers does not guarantee you will handle a live model well, and handling a live model well starts with disciplined experiments and honest validation.

8.10 Exam Worked Example: Binary Autoencoder Backpropagation

8.10.1 Setup and Architecture Choices

Consider a fully connected autoencoder at iteration t. Inputs are two-dimensional and binary, drawn from 00, 01, 10, 11. There is one hidden layer with sigmoid activations, the weights at iteration t are given, the input vector is $x=(1,0)$, and all biases are zero — so biases drop out of every calculation. The learning rate is $\eta =0.25$, a momentum constant is supplied, and the weight values at iteration t−1 are supplied for the momentum terms.

The method discipline stated in class applies to every line that follows: go step by step with plain numbers rather than jumping to symbols.

8.10.2 Forward Pass at Iteration t

Go step by step with plain numbers rather than jumping to symbols. The weighted input to the hidden neuron collects the input products; with the given weights it comes to

$$u_h=1\cdot 0+0\cdot 1=0,$$

so with zero bias the hidden activation is $\sigma (0)=0.5$, where the logistic sigmoid is

$$\sigma (u)=\frac{1}{1+e^{-u}}.$$

One property of this curve does half the work today: a sigmoid always outputs one half at exactly zero input, and it squashes every real number into $(0,1)$, which is what lets its outputs stand in for probabilities of binary targets. With $h$ known, both output nodes can be evaluated, and the loss assembled — the full trace runs below.

8.10.3 Binary Cross-Entropy Loss Calculation

The loss for both nodes together is:

Target output is $t=(1,0)$, so exactly one term per node survives: for node 1, the $t_1=1$ term keeps $-\ln y_1$; for node 2, $t_2=0$ kills the first term and leaves $-\ln (1-y_2)$. Substituting the forward-pass predictions gives the two contributions and their total, as traced in the worked example: $-\ln (0.6225)\approx 0.474$, $-\ln (0.5)\approx 0.693$, summing to $L\approx 1.17$.

When this arithmetic was floated aloud in class some intermediate values wandered near 0.69 and 0.9 before settling — the reliable route is exactly the one traced above: evaluate each surviving log separately, then add.

8.10.4 Weight Updates by Chain Rule

Every weight moves by gradient descent, and the momentum term pulls from the previous iteration's stored values:

$$w\leftarrow w-\eta \frac{\partial L}{\partial w}+\text{momentum term from }t-1.$$

Written in the standard incremental form, the momentum contribution is a constant times the previous update, $\Delta w_t=-\eta \frac{\partial L}{\partial w}{|}_t+\alpha \Delta w_{t-1}$, where the supplied values at iteration t−1 fill in $\Delta w_{t-1}$.

The weight $w_3$ sitting directly on an output path is simple: one derivative of the loss with respect to it suffices, because no layer stands between it and the loss. The weight $w_1$ feeding the hidden layer needs the chain rule of differentiation — take the derivative of the loss with respect to the output, then the derivative along each link down the path to the weight, and multiply the pieces:

$$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial y_1}\cdot \frac{\partial y_1}{\partial u_1}\cdot \frac{\partial u_1}{\partial h}\cdot \frac{\partial h}{\partial u_h}\cdot \frac{\partial u_h}{\partial w_1},$$

where $h$ is the hidden activation, $u_1$ the first output's weighted input, and $u_h$ the hidden node's weighted input. Every factor in that chain has a closed form worth memorizing, and filling them in shows how short the algebra really is:

$$\begin{array}{rlrl}\frac{\partial L}{\partial y_1}& =-\frac{t_1}{y_1}+\frac{1-t_1}{1-y_1}& & \text{(differentiate cross entropy)}\\ \frac{\partial y_1}{\partial u_1}& =y_1(1-y_1)& & \text{(sigmoid derivative)}\\ \frac{\partial L}{\partial u_1}& =\left(-\frac{t_1}{y_1}+\frac{1-t_1}{1-y_1}\right)y_1(1-y_1)=y_1-t_1& & \text{(multiply and cancel)}\end{array}$$

That last cancellation is the celebrated shortcut: with sigmoid outputs trained by cross entropy, the error signal reduces to prediction minus target. Numerically here,

$$\frac{\partial L}{\partial u_1}=0.6225-1=-0.3775,\frac{\partial L}{\partial u_2}=0.5-0=\mathrm{0.5.}$$

Continuing down the chain toward $w_1$ with the given weights (the hidden-to-output path into node 1 carries weight 1, and $x_1=1$):

$$\frac{\partial L}{\partial w_1}=(-0.3775)\cdot 1\cdot h(1-h)\cdot x_1=(-0.3775)(1)(0.25)(1)=-0.0944,$$

using $h(1-h)=0.5\times 0.5=0.25$. The gradient step then moves the weight uphill against this negative gradient:

$$w_1^{\text{new}}=w_1-0.25\times (-0.0944)+\alpha \Delta w_{1,t-1}=w_1+0.0236+\alpha \Delta w_{1,t-1},$$

with the final numeric value completed by inserting the supplied momentum constant and previous-update values from the problem sheet. The finished numerical answer sits in the posted problem set, and ChatGPT reproduces it on request — but the repetition odds for this exact question are ten to fifteen percent, so treat it as practice for the method, not a prediction. Practice several such problems; doing all of them can only help.

Exam Guidance Summary

• Closed book exam. Take the regular exam rather than a makeup whenever possible.
• Carry a scientific calculator. Most problems are quantitative, and minor calculation slips cost only small deductions given the working-professional audience.
• The course continues the deep neural networks course: refresh backpropagation-based weight updates, CNN architectures, convolution operations of every kind — including transposed convolutions — and parameter counting. Those are memory-refresh items, not new study.
• Syllabus scope is what was covered in class sessions so far: PCA and its variants; the autoencoder family — undercomplete, overcomplete, denoising, sparse, regularized, and the linearized version that implements PCA; autoregressive models — WaveNet, Parallel WaveNet, MADE, PixelCNN, PixelCNN++, and Gated PixelCNN; flow models and variants — MAF, IAF, RealNVP, affine flows. Material sitting in decks but never discussed in class will not appear.
• Previous question papers are posted with roughly twelve to fourteen questions; practicing them puts you in good shape. Generate extra problems with ChatGPT — for example on WaveNet or PixelCNN, sized small enough for hand calculation — and verify your results there. When answers disagree, the cause is usually a technique outside class coverage, and you will spot it.
• New this time: questions about your assignment code. Expect output-dimension tracking, the meaning of library arguments, and what one changed line does to the output. Not syntax memorization, and not implementation questions on the autoregressive models handled elsewhere — those stay numerical.
• Four main questions with sub-questions, mirroring the posted papers' patterns.
• Grading is relative across the whole class cohort: scoring 90 means trouble if everyone scores above 90; scoring 35 is comfortable when others sit at 30 or below. Do not obsess over individual question marks.
• Question difficulty aims low on purpose — typical averages near thirty percent, with strong students reaching eighty. Worked problems shown in class have a ten to fifteen percent chance of repeating in some form.
• Study the posted solutions; where understanding feels shaky, prompt ChatGPT for elaborate explanations with a reasonable prompt rather than a lazy one.

A practical reading order for revision falls straight out of this list: first re-derive the two worked autoencoder problems until each numeric step is automatic; second, re-run the parameter-counting and shape-tracking tables from the assignment code with one deliberate change per pass (remove a pool, grow a kernel, widen the latent); third, generate fresh small numericals on WaveNet-style and PixelCNN-style factorizations and solve them by hand against the calculator. Relative grading rewards steady, boring correctness over brilliance on any single question.

Key Industry Applications

• Real-world: sequential autoregressive decoding is why ChatGPT-style systems create images slowly; flow models attack exactly this latency problem with parallel sampling.
• Real-world: Glow (OpenAI) supports attribute editing and interpolation on faces — smiles, hair color, smooth morphs between identities — from intelligent movement among base-distribution samples.
• Real-world: SRFlow brings flow models to super-resolution, synthesizing large images from small ones with user-steerable latent choices instead of a single averaged guess.
• Real-world: flow-based text synthesis manipulates latent dimensions under explicit assumptions; audio synthesis and 3D point-cloud generation reuse the same machinery — point clouds matter especially for 3D computer vision and autonomous-driving simulation.
• Real-world: dequantization by adding uniform noise is mandatory whenever flow models meet digital sensor data — images, in practice — because change of variables needs continuous inputs to stay trainable.
• Real-world: exact likelihood evaluation turns flow models into anomaly screens: industrial inspection and fraud pipelines flag inputs whose learned probability drops below threshold.
• Real-world: the filter-size versus image-content analysis is a live engineering decision — with a client image, run controlled experiments on filter choices and let validation numbers decide.

Across every bullet runs one thread: the same two capabilities — draw a sample when you need one, price any input exactly — are what let these models move from lecture slides into production systems for media editing, sensing, and monitoring.