Unsupervised Deep Learning

Normalizing Flow Models

Published: 2026-08-25

Level: postgraduate

Audience: Postgraduate students in machine learning and deep learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Normalizing flows at first pass — transforming a simple base distribution and the bijectivity constraint — covered in Lecture 1
Autoregressive generative models and chain-rule factorization of a joint density — covered in Lectures 1 and 4
The autoregressive idea — likelihood training and why generation stays sequential — covered in Lecture 5
MADE — masked autoencoder for density estimation — covered in Lecture 5
WaveNet — dilated causal convolutions for audio generation — covered in Lecture 5

These notes cover likelihood-based generative models known as normalizing flows. A generative model is a system that creates new data which never appeared in the training set yet still obeys the same probability law as the training data. The core move of a flow is simple to state: learn an invertible transformation that reshapes messy data into a simple, known distribution. Once trained, the model can score how likely any data point is, and it can generate brand-new samples that obey that same law.

Here is the road we travel:

One dimension first. We recap the full one-dimensional construction — base distribution, change of variables, and the likelihood objective — because every later idea is this picture plus bookkeeping.
The dimension wall. Moving to many dimensions replaces a single slope with a Jacobian matrix, and computing its determinant becomes expensive. We count exactly how expensive.
Architectures around the wall. Autoregressive flows make the determinant cheap by design; inverse autoregressive flows trade training time for sampling speed; Parallel WaveNet combines both with a teacher–student recipe; and the masked autoregressive flow gives the idea a concrete probabilistic body.
Using a trained flow. We close with the two distinct operations a trained flow supports — scoring data and drawing new data — and why flows never compress data.

By the end you should be able to derive the change-of-variables identity from scratch, explain why an autoregressive structure forces a lower-triangular Jacobian, quantify the generation bottleneck for real audio, and trace the teacher–student distillation procedure step by step.

7.1 One-Dimensional Normalizing Flows: Recap and Foundations

7.1.1 The Core Idea and Why We Call It Normalizing

Hook: What would it take for a machine to invent brand-new data — a sentence, an image patch, an audio clip — that no human ever produced, yet that still looks like it came from the same world as the training set? A normalizing flow answers this with one learned function and one calculus identity.

Start from the job description of a generative model: a system that creates new data — data that never appeared in the training set, yet still obeys the same probability law as the training data. A flow model earns this ability through one learned function.

Let $x_{1}$ denote a single scalar drawn from some unknown data distribution $p_{X}$ . We train a transformation $f$ , a function with adjustable parameters, and apply it to get $z_{1} = f (x_{1})$ . The output $z_{1}$ has the same dimension as $x_{1}$ , but no such value ever appears in the dataset. It is a latent variable — an unobserved quantity living behind the data.

Two requirements shape everything that follows:

$f$ must be invertible, also called a bijective or one-to-one mapping: each input produces exactly one latent value, and each latent value leads back to exactly one input. Without invertibility we could not run the machine backwards.
$f$ must be trained so that the distribution of $z$ becomes a known, simple distribution. Two standard choices for this base distribution are the uniform law and the standard normal law:

The two canonical base distributions for the latent $z_{1}$ :

$p_{Z} (z_{1}) = 1 for z_{1} \in [0, 1]$

is the uniform law on the unit interval — every value between 0 and 1 equally likely, so the density has constant height 1 and area 1. More commonly the base is the Gaussian $p_{Z} = N (0, 1)$ , the bell curve centered at zero with spread one:

$p_{Z} (z_{1}) = \frac{1}{2 π} e^{- z_{1}^{2} /2}$

Both are easy to draw samples from and easy to evaluate densities at, which is exactly what training will demand.

That second requirement explains the name. The transformation normalizes an arbitrary input distribution into a standard form, so the whole family is called a normalizing flow. Keep the picture in mind: data enters looking complicated, and after passing through $f$ it comes out shaped like the plain bell curve. Some texts run the arrow the other way and write $x = f [z]$ mapping latent to data; both descriptions say the same thing, and we keep the lecture's convention $z = f (x)$ throughout.

An everyday analogy helps before the math. Picture a fixed amount of honey spread on a tray. You cannot create or destroy honey — you can only reshape the tray. Squeeze the tray narrow and the honey layer gets deep; stretch the tray wide and the layer turns shallow. Total probability mass plays the role of the honey: it stays 1 no matter what. The map $f$ reshapes the tray, and the density is just how deep the honey sits at each point. Where the map stretches space out, the density drops; where it squeezes space together, the density rises. The analogy breaks in one place worth flagging: in one dimension "depth times width" is a single slope, but in many dimensions the stretching happens differently along different directions, and we will need a matrix — the Jacobian — to track it.

Why bother? Because sampling from the base is easy. Draw a sample $z$ from the uniform or Gaussian base. In practice you call a random-number routine — for example Python's random generator — and tell it which distribution to draw from. Then push the sample through the inverse map $f^{- 1}$ . What comes out is a fresh $x$ that was never in the training set but still matches the data distribution. That is the core of every generative model: the ability to create new data obeying the input probability law.

One consequence worth flagging early: because $z$ and $x$ share dimensions, flows do not compress data the way autoencoders do. There is no narrow bottleneck anywhere in the architecture; more on that in Section 7.8.

7.1.2 Change of Variables and the Likelihood Objective

Two facts from probability start the derivation. Any valid density integrates to one, so both of these hold:

$\int_{- \infty}^{\infty} p_{Z} (z) d z = 1 and \int_{- \infty}^{\infty} p_{X} (x) d x = 1$

We know the first density; the second is the unknown we want to learn. The change-of-variables identity links them, and we can derive it from scratch rather than quote it.

Suppose first that $f$ is monotonically increasing (bigger $x$ always gives bigger $z$ ). The cumulative distribution function (CDF) of the data collects the probability up to a point. Because $f$ preserves order, the event " $X$ is at most $x$ " is exactly the event " $Z = f (X)$ is at most $f (x)$ ", so

$F_{X} (x) = P (X \leq x) = P (Z \leq f (x)) = F_{Z} (f (x))$

Now differentiate both sides with respect to $x$ , using the chain rule on the right side:

$p_{X} (x) = p_{Z} (f (x)) \frac{\partial f}{\partial x}$

If instead $f$ is monotonically decreasing, the inequality flips inside the CDF argument, an extra minus sign appears when differentiating, and the derivative $\partial f / \partial x$ is itself negative. Either way the density must stay positive, which is exactly what the absolute value delivers. With $z = f (x)$ :

Univariate change of variables. For an invertible, differentiable transformation $z = f (x)$ ,

$p_{X} (x) = p_{Z} (z) \frac{\partial z}{\partial x}$

Here $\partial z / \partial x$ is the ordinary derivative of the transformation evaluated at $x$ , a scalar slope. The absolute value matters because probabilities are always positive while slopes can be negative. Verbal form: "p of x is nothing but p of z times the partial derivative of z with respect to x."

Notation note: some references write the same statement in the generative direction as $p_{X} (x) = p_{Z} (z) \partial f [z] / \partial z^{- 1}$ . Since $\partial z / \partial x = \partial x / \partial z^{- 1}$ , the two forms agree; we use the lecture's version.

Training now becomes a maximization problem. For data points $x_{1}, \dots, x_{n}$ , maximize the log-likelihood:

$θ max i = 1 \sum n lo g p_{X} (x_{i}) = θ max i = 1 \sum n [lo g p_{Z} (z_{i}) + lo g \frac{\partial z}{\partial x}_{z_{i} = f_{θ} (x_{i})}]$

where $θ$ collects the trainable parameters of $f$ , and each term evaluates the base density and the log absolute slope at the latent point $z_{i} = f_{θ} (x_{i})$ . Equivalently, minimize the negated sum — the negative log-likelihood (NLL). Every piece is computable: the base density is known by assumption, and the slope of a parametric function comes from calculus. Verbal form: "sum of log of p of z sub i plus the partial of z with respect to x, evaluated where z equals z sub i."

Worked example — checking change of variables with real numbers. Take data generated by a Gaussian with mean $μ = 2$ and spread $σ = 3$ , and let the transformation be the standardizing map

$z = f (x) = \frac{x - 2}{3}, \frac{\partial z}{\partial x} = \frac{1}{3}$

Evaluate the model density at $x = 5$ . First push the point through the map: $z = (5 - 2) /3 = 1$ . The standard normal density at $z = 1$ is

$p_{Z} (1) = \frac{1}{2 π} e^{- 1/2} \approx 0.2420$

Change of variables then gives

$p_{X} (5) = p_{Z} (1) \cdot \frac{1}{3} \approx \frac{0.2420}{3} \approx 0.0807$

Sense check: the direct Gaussian formula $\frac{1}{3 2 π} e^{- (5 - 2)^{2} / (2 \cdot 9)} = \frac{0.3989}{3} \times 0.6065 \approx 0.0807$ matches, so the identity behaves. Notice also the honey analogy at work: the map spreads a width-3 slice of input over a width-1 stretch of latent space, so the density must drop by the factor 3.

Scope and assumptions. The identity above needs three things. First, $f$ must be invertible — if two inputs landed on the same latent value, the backwards trip would be ambiguous and the density formula undefined. Second, $f$ must be differentiable, because the whole correction term is a slope. Third, the data density must be continuous; a flow cannot put positive probability at a single exact point. If the slope hits zero somewhere ( $\partial z / \partial x = 0$ ), the transformed density would blow up there, which signals the map has crushed distinct inputs together and stopped being a valid flow.

A visual anchor: sketch the bell curve $p_{Z}$ on axes labeled latent value (horizontal) and density (vertical), and the unknown data density $p_{X}$ on matching axes labeled data value. The transformation is the curved ramp connecting the horizontal axes. Wherever the ramp is steep (slope above 1), the bell curve's height shrinks in the data picture; wherever the ramp is gentle (slope below 1), the height grows. One-line takeaway: the shape of $p_{X}$ is entirely determined by where $f$ speeds up and slows down.

7.1.3 Learning the Transformation with a Neural Network

Nothing restricts $f$ to a fixed formula. Implementing it as a neural network turns the objective above into ordinary network training, and the procedure runs like this:

Pass a training batch $x_{1}, \dots, x_{n}$ through the network to get latents $z_{i} = f_{θ} (x_{i})$ .
Evaluate $lo g p_{Z} (z_{i})$ under the chosen base and the log-slope $lo g ∣ \partial z / \partial x ∣$ using automatic differentiation.
Average to get the NLL and let gradient descent update $θ$ .
Repeat until the latents really do look like draws from the base.

As we minimize the NLL, gradient descent adjusts the network weights, and those weights indirectly become the mapping $f$ . Given any input $x$ , the trained network outputs the latent $z$ ; given any latent, the inverted network returns data. This is the full one-dimensional story from the previous session, and it is worth internalizing before scaling up, since every later idea is this picture plus bookkeeping.

Exam note: expect questions connected to your assignments rather than raw code writing. You should be able to read your own code, explain why a library routine appears, predict what breaks without it, count trainable parameters of a declared network, and state input and output sizes.

7.1.4 Study Tools Worth Using

Well-written tutorials on flow foundations exist online, and reading one or two helps exam preparation — this is the same advice you would give a colleague starting a new topic: see the ideas explained twice, from two angles. ChatGPT makes a solid drill partner too. A prompt that works well is "explain normalizing flow models and its variants with mathematics with math formulas" — the answer arrives concise yet complete, and follow-up practice problems sharpen the algebra. Budget real reading time; a summary only pays off if you actually study it.

Recap: a flow trains an invertible network so that data pushed forward becomes a simple base distribution; likelihoods come from the change-of-variables identity, and samples come from running the base backwards through the inverse. Next, we ask what happens to that single slope $\partial z / \partial x$ when the data stops being one number — and meet the determinant that dominates everything else in this module.

7.2 From One Dimension to Many: Jacobians and Determinants

7.2.1 The Multi-Dimensional Change of Variables

Practical data rarely lives in one dimension. A structured record of any real consequence fills perhaps 50 columns of a spreadsheet. Speech and text signals reach 100-dimensional bag-of-words counts or beyond. Text embedding vectors used in language problems span several hundred dimensions. Even a small image presents about 4096 input values — a 64 by 64 grayscale picture, for instance, has exactly 64 × 64 = 4096 pixels. So the one-dimensional recipe must extend to vectors. We will build the two-dimensional case first and then jump to the general n-dimensional case.

Write the data as a vector $x = (x_{1}, \dots, x_{n})$ and the latent as $z = f (x)$ . The change-of-variables identity gains a determinant:

Multi-dimensional change of variables. For an invertible map $z = f (x)$ with $f : R^{n} \to R^{n}$ ,

$p_{X} (x) = p_{Z} (z) ∣ det J ∣, J_{ij} = \frac{\partial f _{i}}{\partial x _{j}}, J \in R^{n \times n}$

The matrix $J$ is the Jacobian matrix: row $i$ collects the slopes of latent coordinate $f_{i}$ against every input coordinate $x_{j}$ . It is square, and its size equals the data dimension. Verbal form: "px equals pz times the determinant of the Jacobian, whose entries go del z del x across each row." During training, every evaluation of the likelihood needs this determinant, just as the one-dimensional case needed the absolute slope.

Why a determinant rather than a single slope? The one-dimensional slope measured how the map stretches lengths. In $n$ dimensions the map can stretch each direction differently: one axis may be squeezed while another is stretched. The determinant measures the total effect — how much a tiny cube of probability volume around $x$ inflates or deflates under the map. It generalizes the honey analogy: in many dimensions the "depth correction" must account for stretching along all axes at once, and the determinant is precisely that volume-scaling factor. As a limiting case, when $n = 1$ the Jacobian is the single entry $\partial z_{1} / \partial x_{1}$ , its determinant is that number, and the formula collapses to the one-dimensional identity from Section 7.1.

The cost bites immediately. Five-dimensional data means a determinant of a 5 by 5 matrix. A 128-dimensional dataset means evaluating the determinant of a 128 by 128 matrix — computationally cumbersome, and it repeats on every training step for every batch.

7.2.2 What Does Computing a Determinant Cost?

A class discussion pinned down the price, and it is worth replaying because exam questions love this counting argument.

Q: How much computation does evaluating the determinant of an n by n matrix need? A: Start small. A 2 by 2 determinant is ad minus bc, so two multiplications and one subtraction. A 3 by 3 determinant through Laplace expansion needs three factorial, which is 6, paired products. A 4 by 4 needs four times three times two, which is 24. The cofactor pattern grows factorially. Fast determinant algorithms based on row reduction bring the runtime down to order n cubed — cubic time — which is still punishing once the dimension reaches the hundreds.

Let us make every one of those counts concrete. The 2 by 2 case in symbols is

$det (a c b d) = a d - b c$

which uses exactly two multiplications and one subtraction.

Worked example — counting determinant operations at each size. Size 2, real numbers:

$det (3214) = (3) (4) - (1) (2) = 12 - 2 = 10$

Cost so far: 2 multiplications, 1 subtraction.

Size 3 by Laplace expansion along the first row:

$det a d g b e h c f i = a (e i - f h) - b (d i - f g) + c (d h - e g)$

Count the work: there are $3! = 6$ terms, each a product of three entries — the six "paired products" from the discussion. In operation terms, each of the three bracketed pieces is a 2 by 2 determinant (2 multiplications each), and each result is multiplied by an outer entry (3 more multiplications), giving about 9 multiplications plus additions. For instance with $a, b, c = 1, 2, 0$ the third bracket vanishes and only two brackets need computing.

Size 4: the same recursion gives $4! = 24$ quadruple-products; expanding one row costs 4 outer multiplications plus four 3-by-3 expansions at roughly 9 each — about 40 multiplications already. The pattern $C (n) \approx n \cdot C (n - 1)$ grows factorially.

The alternative — row-reduction algorithms (Gaussian elimination, equivalently LU decomposition) — reduces the matrix to triangular form with elementary row operations and reads the determinant off the diagonal in order $n^{3}$ operations — cubic time. To feel the gap at realistic scale: a 128 by 128 determinant by elimination costs about $12 8^{3} \approx 2.1$ million multiply-adds per evaluation, heavy but feasible to repeat batch after batch; a factorial-time expansion would need on the order of $128!$ products, a number so large no computer could ever finish it.

Pitfall — brute force does not scale. Either way you count it, a brute-force determinant is not practical for a flow on realistic data unless we force structure onto the Jacobian. And remember the stakes: this determinant is not computed once. Training evaluates it for every data point in every batch on every step, so even cubic time hurts at hundreds of dimensions. The fix is architectural — design $f$ so its Jacobian has a special shape before any algorithm runs.

7.2.3 Which Matrix Structures Make Determinants Cheap?

The natural follow-up question asks which matrix shapes keep determinants cheap.

Q: Which property of a matrix keeps its determinant cheap to compute? A: A diagonal matrix works perfectly: the determinant is the product of the diagonal elements. Sparse matrices help too. A zero determinant signals linear dependence among columns and a degenerate, non-invertible case, so it is a dead end here. The least restrictive structure that guarantees an easy determinant is triangular: multiply only the diagonal elements.

A quick sanity check on why zero determinants are disqualified: take the singular matrix $(1224)$ ; its determinant is $(1) (4) - (2) (2) = 0$ , because the second column is just twice the first. Geometrically the map crushes the plane onto a line — probability volume collapses to zero, distances between distinct inputs vanish, and the inverse map cannot exist. A flow built on such a Jacobian would boil the probability model down to a degenerate solution.

In symbols, for a triangular matrix $T$ :

Triangular determinant collapse. If $T$ is lower or upper triangular (all entries strictly above, or strictly below, the main diagonal are zero),

$det (T) = i = 1 \prod n T_{ii}$

the determinant collapses to the product of diagonal entries, whether the triangle points up or down. Numerically: $det (2504) = (2) (4) - (0) (5) = 8 = 2 \times 4$ — the off-diagonal 5 never enters. Diagonal matrices are the special case where even the triangle is empty, costing only $n$ multiplications instead of the $n^{3} /3$ or so that elimination needs.

Triangular wins over diagonal because it is the least restrictive structure that keeps the determinant cheap: diagonal maps treat every coordinate independently, while triangular maps let coordinate $i$ respond to coordinates before it — exactly the room needed for expressive transformations. Two conditions now guide all flow design: keep the Jacobian triangular, and keep it invertible (every diagonal entry nonzero, so the product stays positive). Much of what follows on n-dimensional flows is rooted in manufacturing a lower-triangular Jacobian — and the next section shows the assumption that produces one automatically.

Recap: in $n$ dimensions the change-of-variables factor is $∣ det J ∣$ , a volume-stretching measure; computing it naively costs factorial time and even fast algorithms pay cubic time, so practical flows force the Jacobian into a triangular shape whose determinant is a cheap product of diagonals. Next: the autoregressive assumption delivers exactly that shape for free.

7.3 Autoregressive Flows and the Three Design Requirements

7.3.1 The Autoregressive Assumption Builds a Lower-Triangular Jacobian

Here is the trick that makes the determinant tractable: assume an autoregressive structure on the data. Coordinate 2 depends on coordinate 1; coordinate 3 depends on coordinates 1 and 2; and so on. Each output may look at its own input and all earlier inputs, never at later ones.

This assumption is quite realistic. In images, nearby content relates to what came before; in text, the next word or character depends strongly on the previous words and characters — exactly the dependence exploited by recurrent networks and other sequence models. So we are not contorting the data into an unnatural shape; we are writing down the dependency pattern real signals already show.

Under the assumption, the map takes the form

Autoregressive transformation. Each latent coordinate is a function of its own input plus all previous inputs:

$z_{1} = f_{θ_{1}} (x_{1}), z_{2} = f_{θ_{2}} (x_{1}, x_{2}), z_{3} = f_{θ_{3}} (x_{1}, x_{2}, x_{3}), \dots$

Each latent coordinate sees only present and past inputs, never future ones. The Jacobian then automatically comes out lower triangular:

$J = \frac{\partial z _{1}}{\partial x _{1}} \frac{\partial z _{2}}{\partial x _{1}} \frac{\partial z _{3}}{\partial x _{1}} 0 \frac{\partial z _{2}}{\partial x _{2}} \frac{\partial z _{3}}{\partial x _{2}} 00 \frac{\partial z _{3}}{\partial x _{3}}$

because $z_{i}$ does not depend on $x_{i + 1}, x_{i + 2}, \dots$ , so those derivatives vanish: differentiating a constant with respect to anything gives zero. The determinant reduces to multiplying the diagonal entries, the evaluation becomes trivial to implement, and the multi-dimensional flow problem has a solution. Keep the diagonal entries nonzero and the map stays invertible.

Watch how the structure argument works entry by entry. The top-right block of $J$ asks "how does $z_{1}$ respond to $x_{2}$ ?" — but $z_{1}$ was defined without ever reading $x_{2}$ , so every such entry is exactly zero. The same reasoning empties the entire upper triangle. This is why the autoregressive trick needs no numerical cleverness: the cheap determinant falls out of which inputs each function is allowed to see, before any arithmetic happens.

Worked example — a tiny 3-dimensional autoregressive flow. Define three concrete transformations:

$z_{1} = 2 x_{1}, z_{2} = x_{2} + 0.5 tanh (x_{1}), z_{3} = x_{3} + 0.25 x_{1} x_{2}$

Each $z_{i}$ is increasing in its own $x_{i}$ (slopes 2, 1, and 1 respectively), so each can be undone given the past — invertibility holds. Now differentiate. At the data point $(x_{1}, x_{2}, x_{3}) = (1, 2, 3)$ :

Row 1: $\partial z_{1} / \partial x_{1} = 2$ ; entries against $x_{2}, x_{3}$ are zero because $z_{1}$ never reads them.
Row 2: $\partial z_{2} / \partial x_{1} = 0.5 sech^{2} (1) \approx 0.5 \times 0.4199 \approx 0.21$ ; $\partial z_{2} / \partial x_{2} = 1$ ; against $x_{3}$ it is zero.
Row 3: $\partial z_{3} / \partial x_{1} = 0.25 x_{2} = 0.5$ ; $\partial z_{3} / \partial x_{2} = 0.25 x_{1} = 0.25$ ; $\partial z_{3} / \partial x_{3} = 1$ .

$J = 2 0.21 0.5 01 0.25 001, ∣ det J ∣ = 2 \times 1 \times 1 = 2$

Sense check: the messy off-diagonal numbers (0.21, 0.5, 0.25) never enter the answer — only the three diagonal slopes do. That is the whole payoff of triangularity: however rich we make the cross-dependencies, the determinant stays a short product.

7.3.2 Three Requirements: Invertibility, Structure, Expressiveness

Collect the design checklist for any practical flow:

Invertibility. The transformation must be a bijection so generation can run $f^{- 1}$ .
Structure. An autoregressive arrangement forces the lower-triangular Jacobian, making the determinant cheap.
Expressiveness. The family of transformations must be elaborate enough to warp a genuinely complicated input distribution into the standard Gaussian.

The third item hides a subtle two-part puzzle. Solving the optimization is one thing; the quality of the optimum is another. A transformation family that is too rigid may train smoothly yet land at a mediocre optimum.

Pitfall — solvable but not expressive. The optimal NLL should come out very low — numbers in the range of −200, −400, −800 illustrate the target — rather than hovering near 50, 80, or 90. Those figures are illustrative, not literal thresholds, but they convey the demand. Why negative at all? The NLL averages $- lo g p_{X} (x)$ : when the model concentrates high probability on real data, $lo g p_{X}$ grows large and positive, so the loss drops far below zero. A loss stuck near a small positive number means the model barely beats guessing. Structure gives solvability, expressiveness gives a low optimum, and a good architecture supplies both.

A useful mental split: requirement 1 lives inside each coordinate's own transformation (each $f_{θ_{i}}$ must be undoable in its own $x_{i}$ given the past), requirement 2 lives in the wiring pattern between coordinates, and requirement 3 lives in how rich each $f_{θ_{i}}$ may be internally. When you meet a new flow paper, ask which requirement it strengthens — that question organizes the whole literature.

7.3.3 Named Architectures and Non-Volume-Preserving Flows

Several celebrated architectures amount to particular choices of the transformation functions, each engineered to keep a reasonable determinant while staying expressive. The names to know:

Architecture	One-line identity
RealNVP	Real-valued Non-Volume Preserving; couples transformed halves of the input
Flow++	Refines the coupling recipe with richer per-coordinate transforms
Glow	OpenAI architecture; performs strongly despite its complexity

The "non-volume preserving" label teaches the geometry. The determinant measures how a small blob of probability volume stretches under the map. Preserving volume would pin $∣ det J ∣ = 1$ everywhere — like rotating or shearing dough without squeezing it. Dropping that constraint lets ranges differ too, so the range of $x$ need not match the range of $z$ . RealNVP announces in its name that it gives up exact volume preservation in exchange for expressiveness, while still keeping $∣ det J ∣ 0$ so densities stay finite and invertibility survives.

7.3.4 Dequantization: Handling Discrete Data

One more bridge before applications. Everything so far assumes continuous distributions — densities defined on smooth stretches of the number line — yet microphone samples, text tokens, and image pixel values are discrete. A density value at one exact pixel intensity is not even well-defined for countable data, and worse, a discrete distribution lets a flexible density climb toward infinity by piling up on observed values.

The fix is a dequantization step: nudge the quantized input into a continuous space first — for example, add a little noise to each pixel value so values spread over intervals instead of sitting on isolated grid points — after which the entire flow machinery applies unchanged. Knowing why it exists already explains half of it: flows need continuous densities, dequantization manufactures that continuity from discrete data.

This step is the final topic of the flow module; the detailed treatment arrives next session.

Recap: the autoregressive assumption — each coordinate sees only its own input plus the past — makes the Jacobian lower triangular for free, collapsing the feared determinant into a product of diagonal slopes while invertibility rides on keeping those slopes nonzero. Next we make this concrete in two dimensions, where the transformation of choice is a mixture of Gaussian CDFs.

7.4 Two-Dimensional Autoregressive Flow with Mixture CDF Transformations

7.4.1 Formulation and Training Objective

Take two-dimensional data $(x_{1}, x_{2})$ . The autoregressive map reads

$z_{1} = f_{θ_{1}} (x_{1}), z_{2} = f_{θ_{2}} (x_{1}, x_{2})$

with one key asymmetry: $θ_{1}$ carries the minimum necessary condition of invertibility, because $x_{1}$ passes through it alone — if $f_{θ_{1}}$ were not invertible, no later step could repair it. The dependence inside $f_{θ_{2}}$ may be arbitrarily complex on the $x_{1}$ side: any neural network may decide how to treat $x_{2}$ , because once $x_{1}$ has been handled by an invertible first stage, the second stage only needs its action along the $x_{2}$ direction to remain undoable. Notation duality helps later: writing $z_{1} = f (x_{1})$ is the same statement as $x_{1} = f^{- 1} (z_{1})$ ; it is purely a question of which direction you read.

For completeness, write out the Jacobian of this two-coordinate map:

$J = \frac{\partial z _{1}}{\partial x _{1}} \frac{\partial z _{2}}{\partial x _{1}} 0 \frac{\partial z _{2}}{\partial x _{2}}, ∣ det J ∣ = \frac{\partial z _{1}}{\partial x _{1}} \frac{\partial z _{2}}{\partial x _{2}}$

Expanding the one-dimensional objective to two dimensions gives

Two-dimensional training objective.

$lo g p_{X} (x_{1}, x_{2}) = lo g p_{Z} (z_{1}) + lo g p_{Z} (z_{2}) + lo g ∣ det J ∣$

Now notice which pieces move. The base densities $p_{Z} (z_{1})$ and $p_{Z} (z_{2})$ are known quantities — the base distribution was assumed at the outset, so nothing about them needs to be learned. What remains to be pushed upward is the log-determinant term. Practitioners call this the log-determinant objective: maximize the log of the Jacobian determinant, which requires differentiating the transformation network and evaluating a small determinant. Verbal form: "you know the p of z pieces because the base distribution is assumed, so what you maximize is the log Jacobian part."

A precision note on "drop out as constants": the base-density terms still change from step to step, because $z_{i}$ itself moves as the parameters $θ$ update. They are known functions we can always evaluate exactly — that is the real content of the remark. The name "log-determinant objective" highlights the one factor that makes flow training special compared with any other network loss.

7.4.2 Cumulative Distribution Function Refresher

The most popular two-dimensional transformation leans on the cumulative distribution function (CDF), so recall its definition from an earlier probability and statistics course:

CDF definition and properties.

$F (Θ) = \int_{- \infty}^{Θ} f (t) d t$

where $f (t)$ is the density (PDF) and $F (Θ)$ accumulates probability up to the point $Θ$ . Its properties are worth memorizing:

$F$ rises monotonically (never decreases), from a minimum of 0 to a maximum of 1;
at the mean of a Gaussian distribution, $F$ equals exactly 0.5 (the bell curve is symmetric);
differentiating the CDF hands back the PDF: $F^{'} (Θ) = f (Θ)$ .

These basics deserve permanent recall regardless of professional path — graduate-level work assumes them.

The monotonicity property is not decoration here — it is precisely what makes a CDF usable inside a flow. An invertible scalar map must be strictly monotone, and every CDF passes that test automatically. That observation motivates the practice task from class: derive the CDF of a mixture of Gaussians and state its minimum and maximum. Here is the full derivation so you can check your own attempt.

Worked derivation — CDF of a mixture of Gaussians (the practice task). Let the density be a weighted blend of $M$ Gaussian components,

$f (x) = m = 1 \sum M π_{m} \frac{1}{σ _{m} 2 π} e^{- \frac{1}{2} (\frac{x - μ _{m}}{σ _{m}})^{2}}, π_{m} > 0, m = 1 \sum M π_{m} = 1$

Integrate term by term (the integral of a sum is the sum of the integrals):

$F (Θ) = \int_{- \infty}^{Θ} m = 1 \sum M π_{m} f_{m} (t) d t = m = 1 \sum M π_{m} \int_{- \infty}^{Θ} f_{m} (t) d t = m = 1 \sum M π_{m} Φ (\frac{Θ - μ _{m}}{σ _{m}})$

where $Φ$ is the standard Gaussian CDF and the substitution $u = (t - μ_{m}) / σ_{m}$ produced each inner integral. So the mixture CDF is just the same weights applied to standard CDF values. Now the extremes: as $Θ \to - \infty$ , every $Φ (\cdot) \to 0$ , so $F \to \sum_{m} π_{m} \cdot 0 = 0$ ; as $Θ \to + \infty$ , every $Φ (\cdot) \to 1$ , so $F \to \sum_{m} π_{m} \cdot 1 = 1$ . Numerical taste-test with two components — weight 0.7 at mean 2 spread 0.7, weight 0.3 at mean 3 spread 8 — evaluated at $Θ = 2$ :

$F (2) = 0.7 Φ (0) + 0.3 Φ (\frac{2 - 3}{8}) = 0.7 (0.5) + 0.3 (0.45) \approx 0.485$

which sits sensibly between the two component contributions.

Why is this the right transformer for a flow? A classic result closes the loop: if $X$ comes from a continuous density whose true CDF is $F$ , then the random value $z = F (X)$ follows exactly the uniform distribution on $[0, 1]$ . Applying a CDF to its own data normalizes it — which is why a learnable CDF-shaped transformation, like the mixture above, is such a natural fit for forcing latents toward the base distribution. Its slope is also friendly: differentiating the mixture CDF gives back the mixture PDF, a positive number we can write in closed form, so the diagonal entries of $J$ are cheap and never vanish.

7.4.3 Worked Synthetic Experiments

Why synthetic experiments? Because the ground truth is known in advance, so after training you can check how close the learned distribution came to reality. With real data nobody can say what the "true" density is; with manufactured data the answer key exists. Two setups show the machinery at work.

Experiment A — moon-shaped data. Construct $x_{1}$ as a mixture of five Gaussians with user-chosen means and spreads, and construct $x_{2}$ as another five-Gaussian mixture conditioned on $x_{1}$ . Plotting samples traces a moon shape: the five centers march along an arc, and conditioning makes the arc's width vary along its length. The flow transforms each coordinate through a mixture-CDF block. Here $z_{1}$ is the CDF of a mixture of Gaussians applied to $x_{1}$ :

$z_{1} = F_{MOG} (x_{1}) = m = 1 \sum M π_{m} Φ (\frac{x _{1} - μ _{m}}{σ _{m}})$

where $Φ$ is the standard Gaussian CDF, $π_{m}$ the mixture weights, and $(μ_{m}, σ_{m})$ the component mean and spread. Component weights such as $π_{1}$ and $π_{2}$ enter the sketch for simplicity; the experiment itself uses five components ( $M = 5$ ), each contributing its own mean and spread. A convenient implementation detail: an arbitrary neural network can be trained to output the mixture parameters — the weights, the per-component means, and the five spreads — directly from $x_{1}$ ; the same construction, conditioned on $x_{1}$ , serves $z_{2}$ .

Watch the training trajectory through the latent space. At epoch 0, right after random initialization, the latent density sits far from the target. By epoch 3 the cloud has migrated noticeably toward a uniform-like blob. After 99 iterations the latent density looks like the assumed base, and pushing samples back through $f^{- 1}$ reproduces the original moon — bright regions appear exactly where the data had many points, because brightness tracks sample density.

Read those three snapshots as a movie of the honey being re-poured: initialization pours it badly, early epochs push mass toward the right tray shape, and by iteration 99 the latent picture is the plain base while the data-side inverse map sculpts the moon faithfully.

Experiment B — face-like data. Here the base is uniform, $p_{Z} = 1.0$ on the unit square — every point in $[0, 1] \times [0, 1]$ equally likely — chosen because it keeps life simple: the base density needs no formula beyond the constant 1. The data side again mixes five Gaussians, this time with means and spreads arranged so the scatter plot resembles a face: two eyes and a lip. Starting from the initial configuration, training runs through the same epochs, and after 99 iterations the recovered distribution closely matches the original face pattern.

Both experiments are controlled demonstrations — the transformation choices are known because the data is synthetic, so success or failure is measurable rather than a matter of taste.

7.4.4 Student Questions and Answers

A student question during this segment exposed a genuine confusion worth preserving.

Q: How do we know the plotted data really is a mixture of five Gaussians? A: The data was manufactured on purpose. Code calls a sampling routine with chosen settings — for example, give me a Gaussian sample with mean 2 and spread 0.7 taken with weight 0.7, plus a Gaussian with weight 0.3, mean 3, spread 8 — and repeated calls build the scatter plot. Knowing the recipe means the ground truth is available, so after learning you can measure how close the estimated distribution came to the true one. Otherwise judging quality stays completely subjective. This is a controlled experiment, nothing more.

The takeaway generalizes far beyond this lecture: whenever you evaluate a density estimator, prefer data whose generating process you control. Benchmarks built on synthetic mixtures exist across machine learning for exactly this reason — they turn "does it look right?" into "how far off is the number?".

Recap: a 2D autoregressive flow splits the work — one invertible scalar transform per coordinate, with an arbitrary network choosing the second transform's settings from $x_{1}$ — and training pushes up the log-determinant alongside known base-density terms; mixture-of-Gaussian CDFs supply transforms that are automatically monotone, cheap to differentiate, and bounded between 0 and 1. Next: with training understood, what does generation cost?

7.5 Sampling from a Trained Flow and the Generation Bottleneck

7.5.1 How Sampling Works

After training, the latent distribution is, theoretically speaking, exactly the standard normal. Generation exploits that guarantee:

Generation from a trained flow.

$z \sim N (0, I), x = f^{- 1} (z)$

Draw a latent vector — call the draw $z_{s} = (z_{1}, z_{2}, \dots, z_{n})$ , each coordinate a plain scalar — using a random routine parameterized by the target distribution (here $N (0, I)$ means each $z_{i}$ is an independent standard normal), then pass the vector through the inverse transformation. Out comes an $x$ matching the training distribution. Training minimized the NLL; generation is just sampling followed by inversion.

Notice how cheap this looks next to training: no data, no likelihood sums, no gradients — one random draw and one network pass. At the thirty-thousand-foot level this story is clean, and theory stays clean no matter the altitude. The catch hides in what "one pass through $f^{- 1}$ " means once the map is autoregressive.

7.5.2 Why Sequential Generation Is Impractical

Autoregressive structure entangles the coordinates during generation. To produce $x_{1}$ , use $z_{1}$ alone. To produce $x_{2}$ , you need $z_{2}$ plus the $x_{1}$ you just created. To produce $x_{3}$ , you need $z_{3}$ plus both earlier outputs. Each coordinate waits on its predecessors:

The generation chain.

$x_{1} = f_{1}^{- 1} (z_{1}), x_{2} = f_{2}^{- 1} (x_{1}, z_{2}), x_{3} = f_{3}^{- 1} (x_{1}, x_{2}, z_{3}), \dots$

Every inverse step consumes the outputs of all earlier steps, so the coordinates cannot be computed at the same time even though all the latents $(z_{1}, \dots, z_{n})$ are already sitting in memory. Creating an n-dimensional sample costs n sequential steps.

This is the same dependency pattern that made training easy, read in reverse. During training every $x_{i}$ was already on disk, so all $n$ latent computations ran in parallel; during generation nothing exists yet except the latents, so the chain must unroll one link at a time.

Worked example — audio arithmetic. Run the numbers on speech. One second of audio recorded at 20 kilohertz contains

$20, 000 Hz \times 1 s = 20, 000 samples$

Each sample is one coordinate of $x$ , so generating one second costs 20,000 sequential steps — step 401 cannot start until steps 1 through 400 finish. Now attach a wall-clock guess: even if a single inverse step takes only 1 millisecond, one second of audio needs about $20, 000 \times 1 ms = 20$ seconds to generate. Ten seconds of audio takes over three minutes. Sense check: the cost grows linearly with duration and with sampling rate, so a 44.1 kHz studio stream would be worse still.

Pitfall — real-time systems cannot wait. Live speech translation under such lag simply does not work: a translator that answers twenty seconds after the speaker stops has missed the conversation. The delay makes sequential generation unusable in real scenarios, no matter how good the samples sound. This bottleneck motivated an entire branch of architecture design covered in the next two sections.

The asymmetry stings: autoregressive flows train fast, because all input coordinates sit available at once and the likelihood terms parallelize, yet they generate slowly, one coordinate at a time. Also remember the dimensional fact noted earlier — the latent space matches the original dimension ( $z \in R^{n}$ for $x \in R^{n}$ ), so flows cannot double as compression tools the way autoencoders and embedding vectors do, where the encoded size shrinks far below the input size.

Recap: generation draws $z$ from the base and inverts the flow, but autoregressive inversion is a serial chain — n coordinates cost n dependent steps, which for 20 kHz audio means 20,000 chained evaluations per generated second. Next: the inverse autoregressive flow, which moves that serial chain out of generation and into training.

7.6 Inverse Autoregressive Flows

7.6.1 Trading Training Time for Sampling Speed

The inverse autoregressive flow (IAF) flips the direction of the pain. The functional form mirrors the autoregressive flow, but the phases swap:

Training is sequential and slow. Building the latents from a data vector walks a serial chain: $z_{1}$ comes from $x_{1}$ ; $z_{2}$ uses the freshly made $z_{1}$ together with $x_{2}$ ; $z_{3}$ uses $z_{1}, z_{2}$ and $x_{3}$ . Only after the chain completes can the likelihood be scored and training proceed, so estimating $θ$ goes slowly.
Generation is parallel and fast. Sample the entire latent vector $(z_{1}, \dots, z_{n})$ from the base in one shot, then apply the joint inverse map:

IAF generation in one parallel pass.

$(x_{1}, \dots, x_{n}) = f^{- 1} (z_{1}, \dots, z_{n})$

Because every $z$ is already at hand, all coordinates of $x$ emerge simultaneously, in one parallel pass.

Why is that possible when the plain AF was stuck serializing? Look at what each output coordinate reads. In an IAF, each generated coordinate depends on the latents and the earlier latents — not on previously generated outputs:

$x_{i} = g_{i} (z_{1}, \dots, z_{i})$

Since all latents arrive together from the random draw, no coordinate ever has to wait for another coordinate of $x$ . Pictorially, the data-flow arrows reverse between the two regimes — the serial chain moves from generation time (autoregressive flow) to training time (inverse autoregressive flow).

Trace on a tiny 3-dimensional IAF. Define three generative maps, each reading only latents:

$x_{1} = 2 z_{1}, x_{2} = z_{2} + 0.5 z_{1}^{2}, x_{3} = z_{3} + 0.25 sin (z_{2})$

Draw $(z_{1}, z_{2}, z_{3}) = (0.5, - 1, 2)$ from the standard normal base. All three outputs compute at once:

$x_{1} = 2 (0.5) = 1, x_{2} = - 1 + 0.5 (0.25) = - 0.875, x_{3} = 2 + 0.25 sin (- 1) \approx 1.79$

No coordinate needed any other $x$ . Now try the reverse trip for training — recover the latents from data $(1, - 0.875, 1.79)$ : first $z_{1} = x_{1} /2 = 0.5$ ; only then can we form $z_{2} = x_{2} - 0.5 z_{1}^{2} = - 1$ ; only then $z_{3} = x_{3} - 0.25 sin (z_{2}) \approx 2$ . Each step consumed the previous result, so scoring this one data point cost three chained evaluations — and real training repeats that chain for every example in every batch. Sense check: the same three numbers came back out, confirming invertibility survived both directions.

7.6.2 Comparing AF and IAF Side by Side

Aspect	Autoregressive flow (AF)	Inverse autoregressive flow (IAF)
Training	Parallel, fast — direct maximum likelihood	Sequential, slow — the latent chain blocks progress
Density evaluation	Efficient for any external data point	Poor for external points; efficient for its own generations via caching
Sampling	Sequential over coordinates, slow	All coordinates at once, fast

When to pick which: choose an AF-style model when you mainly need to score data — density evaluation, anomaly detection, compression benchmarks; choose an IAF when you mainly need to generate fast, as in speech synthesis. Note the middle row: an IAF cannot cheaply assign a likelihood to a fresh external data point, because scoring it means walking the slow serial chain — except for points the model itself generated, where the intermediate values were cached along the way.

The comparison begs one more question: can we combine the good halves of both methods — parallel training from AF, parallel sampling from IAF — into something better than either? All strong implementations of flow models pursue exactly that combination, pairing the two ideas in a smart fashion. The next section shows the flagship example.

Recap: an IAF points the autoregressive dependencies at the latents instead of the outputs, so generation collapses into one parallel pass while the serial chain migrates into training-time inversion — the exact mirror image of the plain AF. Next: Parallel WaveNet, which trains a teacher of the fast kind and distills it into a student of the fast-sampling kind.

7.7 Parallel WaveNet: Combining AF and IAF with a Teacher and a Student

7.7.1 The Two-Part Teacher–Student Training Procedure

Recall WaveNet from earlier sessions: an autoregressive audio model whose training is fast but whose sample-by-sample generation crawls. Parallel WaveNet fixes generation by combining the strengths of AF and IAF, and it remains in production today — Google uses it for speech processing work such as speech translation and speech-to-text.

First appreciate why a naive combination fails, because the failure motivates the tricks. If a network genuinely consumed all past context at every output, its depth would grow with the sequence length: a one-megapixel image would demand about a million layers. WaveNet's dilation trick — stacking layers with spacing $d = 1$ , then $d = 2$ , then $d = 4$ , and so on — reaches distant context with few layers, cutting that depth dramatically, and a similar economy applies here.

Engineering pitfall — recomputing what you already know. The second trick, specific to Parallel WaveNet, is storing activations: after training the teacher, push the training inputs through it once and keep the internal states. This mirrors a computational graph, which stores intermediate variables during backpropagation so that derivative evaluation can reuse them — recall that picture from a first deep-learning course. Cached activations then feed the student's training instead of forcing recomputation.

The procedure splits one hard problem into two easy ones:

Teacher–student distillation procedure.

Train the teacher. The teacher is a masked autoregressive flow (MAF) — the autoregressive flow in precise form, where each value depends only on past values, never future ones. Masking makes training efficient by maximum likelihood estimation: minimize the negative log-likelihood of the training data. This phase runs offline and in parallel, so it trains quickly despite being an AF.
Freeze the teacher. Its parameters never change again. The frozen map sends data $x$ to latents $z$ whose distribution is the simple base — meaning the teacher now carries a fully trained density it can evaluate at any point.
Train the student. The student is an IAF. It cannot efficiently score arbitrary external data points, but it can efficiently score its own generations via cached numbers. Initialize it using the teacher's cached activations, then train it to mimic the teacher's distribution by minimizing a divergence (next subsection).
Generate. Draw $z_{1}, \dots, z_{n}$ from the assumed base once and push them through the student's forward map; all of $x_{1}, \dots, x_{n}$ arrive in a single parallel shot.

The division of labor is worth stating in one line: the teacher learns what the data distribution is (using the training-friendly direction), and the student learns to say the same thing quickly (using the sampling-friendly direction).

7.7.2 Matching Distributions with KL Divergence

How do we formalize "the student should say the same thing as the teacher"? Through KL divergence, met in an earlier machine-learning course and revisited heavily in GAN and VAE discussions. Two facts carry the weight here. First, the KL divergence between identical distributions is zero, and it grows as the distributions drift apart — so minimizing it aligns the student with the teacher. Second, it is not symmetric, which disqualifies it as a true distance.

Q: Does the KL divergence behave like a distance, equal in both directions? A: No. Swapping the arguments changes the value, so KL divergence is not symmetric and is not a true distance metric. Metric spaces demand distance(P, Q) = distance(Q, P); a divergence drops that requirement. Zero still means the two distributions coincide.

With $p_{S}$ denoting the density the student assigns and $p_{T}$ the teacher's density, the student minimizes

The student's objective — reverse KL divergence.

$D_{K L} (p_{S} ∥ p_{T}) = E_{x \sim p_{S}} [lo g p_{S} (x) - lo g p_{T} (x)] = \int_{- \infty}^{\infty} p_{S} (x) [lo g p_{S} (x) - lo g p_{T} (x)] d x$

The middle form reads "expectation over samples $x$ drawn from the student"; the right form unpacks the expectation as an integral weighted by the student's own density. Verbal form: draw samples from the student, then compare how strongly the student and the teacher each believe those samples; shrink the gap.

Two sanity checks on this quantity. Domain check: every term of the integral averages a log-ratio under its own weighting, and Gibbs' inequality guarantees $D_{K L} \geq 0$ , with equality exactly when $p_{S} = p_{T}$ almost everywhere — so zero is achievable only by perfect agreement. Direction check: because samples are drawn from $p_{S}$ , regions the student visits often dominate the loss; swapping to $D_{K L} (p_{T} ∥ p_{S})$ would instead weight regions the teacher favors — one reason the two directions behave differently in practice.

Why trust the target? The teacher earned its density from the actual training data through maximum likelihood, so a student that copies the teacher inherits a good model of the data. Throughout this phase the teacher stays frozen; the student merely learns to reproduce what the teacher already knows, starting from the base distribution side. The names are chosen with care: like a classroom, knowledge flows one way during study — from teacher to student — and the teacher never updates itself in response.

7.7.3 Results and Real-World Use

The payoff is dramatic: sampling improves by roughly 1000× over the original WaveNet. This teacher–student distillation is not a curiosity — it powers production speech pipelines at Google to this day, including speech translation and speech transcription. Every spoken response from such systems traces back to exactly this recipe: a maximum-likelihood MAF teacher, frozen; an IAF student pulled toward it with reverse KL; parallel generation at deployment.

The architectural lesson generalizes beyond audio: when one training procedure forces slow generation, consider splitting the work into a parallel-trained teacher plus a parallel-generating student tied together by a distribution-matching loss. The same pattern reappears in knowledge-distillation setups across deep learning.

Recap: Parallel WaveNet gets parallel training (MAF teacher by maximum likelihood) and parallel sampling (IAF student minimizing reverse KL against the frozen teacher), with cached activations bridging the two — about a thousandfold faster generation than sequential WaveNet. Next: with architectures settled, how do you actually use a trained flow day to day?

7.8 Using a Trained Flow: Inference versus Sampling

7.8.1 Scoring Data and Drawing New Data

A trained flow supports two distinct operations, and keeping them separate prevents confusion. They use opposite directions of the same invertible map.

Inference (scoring). Apply the forward map to a candidate point and read off its density:

Scoring a data point.

$p_{X} (x) = p_{Z} (f (x)) det J$

Push the candidate $x$ through the trained transformation, evaluate the known base density at the resulting latent, and correct for volume stretching with the Jacobian determinant. The number answers "how likely is this data?"

Interpretation comes in extremes. A very small value marks rare, unusual data — perhaps an outlier, a corrupted sensor reading, or a flawed generation from another model. A large value marks typical data sitting near the heart of the training distribution.

Sampling (generation). Go the other way: draw $z$ from the base distribution and return $f^{- 1} (z)$ , producing fresh data.

Side by side:

Operation	Direction	Input → Output	Typical question
Inference (scoring)	data → latent ( $f$ )	$x$ → density value	"How likely is this point?"
Sampling (generation)	latent → data ( $f^{- 1}$ )	random $z$ → new $x$	"Show me a new example."

When to reach for which: scoring powers anomaly detection — banks flag transactions whose density falls below a threshold, and factories flag machine vibrations that drift from the learned normal pattern; sampling powers synthesis — speech, images, or synthetic tabular records for testing pipelines.

And a structural reminder: $x$ and $z$ always share dimensionality. No bottleneck exists anywhere in a flow, so unlike autoencoders — and unlike the compact embedding vectors used throughout language processing — a flow offers no compression.

Scope: flows are the wrong tool when your goal is a smaller representation. If you need to shrink 4096 pixel values down to a 512-number code, use an autoencoder or an embedding model; a flow will hand back 4096 latents for 4096 inputs every time. Its value lies entirely in the exact density it models and the samples it draws — those two abilities are what you buy with the invertibility constraint.

Recap: one invertible network serves two operations — forward for scoring likelihoods (outliers get tiny densities), inverse for generating fresh data — and neither direction ever compresses dimensionality. Next: the final architecture of the module gives the autoregressive idea a concrete probabilistic body.

7.9 Masked Autoregressive Flow: A Gaussian Autoregressive Model

7.9.1 Model Formulation

The last architecture gives the autoregressive idea a concrete probabilistic body. Assume the data follows a Gaussian autoregressive model: factor the joint density into conditionals, then give each conditional a Gaussian shape whose settings shift with history.

The factorization is the chain rule of probability, which splits any joint density into a product of one-coordinate conditionals:

$p (x) = i = 1 \prod n p (x_{i} ∣ x_{1}, \dots, x_{i - 1})$

Taking logarithms turns that product into a sum, which is the form training uses:

Autoregressive factorization.

$lo g p (x) = i = 1 \sum n lo g p (x_{i} ∣ x_{1}, \dots, x_{i - 1})$

Each factor gets a Gaussian shape with its own mean and spread, both computed from history:

$p (x_{i} ∣ x_{< i}) = N (x_{i}; μ_{i}, α_{i}^{2}), μ_{i} = f_{μ} (x_{1}, \dots, x_{i - 1}), α_{i} = exp (f_{α} (x_{1}, \dots, x_{i - 1}))$

Here $x_{< i}$ abbreviates the past coordinates $x_{1}, \dots, x_{i - 1}$ . The symbol $μ_{i}$ is the conditional mean produced by a function of the past coordinates; $α_{i}$ is a spread produced exponentially so it stays positive — whatever real number the network $f_{α}$ emits, exponentiating maps it into $(0, \infty)$ , and a Gaussian spread must live exactly there. Both functions are realized as neural networks for $i > 1$ while remaining plain constants for $i = 1$ , since the first coordinate has no history to read. (Some texts write the spread as $σ_{i} = e^{ρ_{i}}$ ; this is the same construction with a different letter for the raw network output.)

Read the model correctly: it is not one Gaussian but a series of Gaussians whose means and spreads slide continuously as the past unfolds. A single multivariate Gaussian can only express one elliptical blob; a chain of conditional Gaussians can express arcs, multi-modal shapes, and heavy tails — each coordinate's bell curve re-centers and re-scales depending on what came before. That is what makes the model elaborate enough to fit rich data while every piece stays simple enough to write down.

7.9.2 Sampling and the Scale-and-Shift Inverse Mapping

Generation composes the standard-normal sampler with a scale-and-shift:

Scale-and-shift sampler and its inverse.

$x_{i} = exp (α_{i}) z_{i} + μ_{i}, z_{i} \sim N (0, 1), i = 1, \dots, n$

Training learns the two functions behind $μ_{i}$ and $α_{i}$ , each consuming only past coordinates $x_{1}, \dots, x_{i - 1}$ . The inverse map is symmetric — scale-and-shift undone:

$z_{i} = (x_{i} - μ_{i}) exp (- α_{i})$

Subtracting $μ_{i}$ recenters, multiplying by $e^{- α_{i}}$ undoes the stretch; composing the two steps in either order returns the other side.

Worked numbers — scale-and-shift round trip. Let the networks output $μ_{i} = 2$ and raw value $f_{α} = 0.693$ at some coordinate, so the spread is $exp (0.693) \approx 2$ . Draw $z_{i} = 1.5$ from the standard normal base.

Forward (generation):

$x_{i} = exp (α_{i}) z_{i} + μ_{i} = 2 \times 1.5 + 2 = 5$

Inverse (scoring): given $x_{i} = 5$ and the same settings,

$z_{i} = (5 - 2) \times e^{- 0.693} = 3 \times 0.5 = 1.5$

the original draw, recovered exactly. Sense check: shifting by 2 and stretching by 2 moved the standard bell to center 2 with spread 2 — precisely the conditional distribution $N (5; 2, 4)$ claims.

Quality comes out high — generated data looks convincingly close to the originals, and the NLL lands at a small value. Speed does not: generation still walks the autoregressive chain one coordinate at a time, because producing $x_{i}$ requires $μ_{i}$ and $α_{i}$ , which are functions of the already-produced past coordinates.

Pitfall — nice samples, slow sampler. This model is the masked autoregressive flow (MAF), and its weakness is inherited, not accidental: any model whose outputs depend on previous outputs must generate serially. Do not confuse quality with speed — MAF earns high scores on likelihood benchmarks and still cannot emit a second of audio quickly.

The inverse direction is where masking shines. Because each equation isolates its own coordinate given the past, MADE-style masked architectures evaluate all coordinates of $z$ in parallel from a data vector — one network pass computes every $(μ_{i}, α_{i})$ at once — and the resulting Jacobian $\partial z / \partial x$ is lower triangular by construction, exactly the structure demanded back in Section 7.2. The IAF twin swaps the speed profile: slow training, fast generation. Between MAF, IAF, and their distillation through Parallel WaveNet, the flow toolbox covers both directions of the compute trade-off.

Closing note for preparation: the remaining variants differ mainly in the chosen form of the transformation — different scale-and-shift recipes reshape the same skeleton — so focus on how each variant arranges invertibility, the triangular Jacobian, and expressive power. The next session finishes these variants in about half an hour and then solves numerical problems from past exams, so review this material beforehand.

Recap: MAF instantiates the autoregressive flow as chained conditional Gaussians — network-predicted means, exponentially-parameterized positive spreads, a scale-and-shift sampler and its exact inverse — gaining expressive power while keeping the lower-triangular Jacobian, at the price of sequential generation. That closes the architecture tour; the appendices next gather exam guidance and industry context in one place.

Exam Guidance Summary

Exam note: everything through the flow-model module counts, including the autoregressive material in these notes; nothing beyond it is in play for this exam cycle.

Assignment-derived questions. No code writing is expected in the exam hall. Instead, expect questions tied directly or indirectly to your assignment: read your own code, explain why a particular library routine was chosen, state what would happen if it were dropped, count the trainable parameters of a declared architecture, and give input and output sizes.
Deviations from past papers are deliberate. Some questions will differ from the released past papers on purpose, to test experiential learning — what you absorbed by doing the assignments.
Practice material. Four past mid-semester papers with worked solutions from an earlier advanced deep-learning offering are available; solving them is strongly advised. The next session closes out the remaining flow variants (about half an hour) and then works numerical problems from past exams.
Conceptual hot spots. Determinant complexity and the triangular structure argument (Section 7.2), the AF versus IAF speed trade-off (Sections 7.5–7.6), and the teacher–student KL procedure (Section 7.7) are prime question territory.

A compact revision list of the math facts to keep ready:

The change-of-variables identity with its absolute value: $p_{X} (x) = p_{Z} (z) \partial z / \partial x$ .
Determinant cost counting: two multiplications at size 2; six paired products at size 3 via Laplace expansion; factorial growth versus cubic-time row-reduction algorithms.
CDF bounds of 0 and 1, with 0.5 exactly at the mean of a Gaussian; the mixture-CDF transform stays monotone between those bounds.
The scale-and-shift sampler $x_{i} = exp (α_{i}) z_{i} + μ_{i}$ and its inverse $(x_{i} - μ_{i}) exp (- α_{i})$ .
The reverse-KL student objective $D_{K L} (p_{S} ∥ p_{T}) = E_{x \sim p_{S}} [lo g p_{S} (x) - lo g p_{T} (x)]$ , remembered as "not a distance".

Practice task carried forward. Derive the CDF of a mixture of Gaussians and state its minimum and maximum values before the final session — the full derivation appears in Section 7.4 to check against.
Study tools. Use the recommended flow write-ups plus AI-assisted drilling with the prompt "explain normalizing flow models and its variants with math formulas"; budget genuine reading time over skimming the output.

Key Industry Applications

Google speech stack. Parallel WaveNet distillation runs in production for speech translation and speech-to-text, delivering roughly 1000× faster sampling than the original WaveNet — the difference between a voice assistant that answers in real time and one that cannot.
Named flow families. RealNVP (Real-valued Non-Volume Preserving), Glow from OpenAI, and Flow++ are the high-performing reference architectures built around tractable Jacobians; they serve as baselines in new generative-modeling papers.
Data shapes in practice. Structured records near 50 columns, text as 100-dimensional bag-of-words or larger, embedding vectors in the hundreds of dimensions, images around 4096 input values, and 20 kilohertz audio streams demanding 20,000 sequential steps per generated second under naive autoregressive sampling. These scales explain why cheap Jacobians and parallel sampling are engineering requirements, not luxuries.
Anomaly and outlier detection. Because a flow scores exact likelihoods (Section 7.8), finance teams flag low-density transactions for review, and industrial monitoring flags sensor readings that drift from learned normal behavior.
AI assistants as study aids. A targeted prompt requesting normalizing flows and their variants with full mathematics returns compact, complete explanations suited for revision and problem practice.
Prototyping samplers. Standard library random routines parameterized by distribution type are all you need to draw base samples when building or testing a flow pipeline — no special hardware required to validate an idea end to end.

UDL Lecture 7 notes · Normalizing Flow Models

Unsupervised Deep Learning· postgraduate· 2026-08-25

Sections Breakdown

1One-Dimensional Normalizing Flows: Recap and Foundations

Base distributions, invertible transformations, the univariate change-of-variables identity derived from scratch, and training by maximum likelihood.

2From One Dimension to Many: Jacobians and Determinants

The multi-dimensional change of variables, what determinants cost to compute, and which matrix structures make them cheap.

3Autoregressive Flows and the Three Design Requirements

Autoregressive structure, the lower-triangular Jacobian, three design requirements, named architectures, and dequantization.

4Two-Dimensional Autoregressive Flow with Mixture CDF Transformations

Two-dimensional flows with mixture-of-Gaussian CDF transforms, CDF properties, worked derivations, and synthetic experiments.

5Sampling from a Trained Flow and the Generation Bottleneck

How sampling from a trained flow works and why sequential generation becomes impractical for audio-scale data.

6Inverse Autoregressive Flows

Trading training time for sampling speed, with a side-by-side AF versus IAF comparison.

7Parallel WaveNet: Combining AF and IAF with a Teacher and a Student

Teacher-student distillation: an offline MAF teacher and an IAF student matched by reverse KL divergence.

8Using a Trained Flow: Inference versus Sampling

Scoring data versus drawing new data, anomaly detection uses, and why flows never compress.

9Masked Autoregressive Flow: A Gaussian Autoregressive Model

Chained conditional Gaussians, scale-and-shift sampling with its exact inverse, and the serial-generation price.

10Exam Guidance Summary

Collected exam scope, assignment-derived question types, practice tasks, and conceptual hot spots.

11Key Industry Applications

Production uses of flow models, reference architectures, and realistic data scales.

Postgraduate students in machine learning and deep learning

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

One-Dimensional Normalizing Flows: Recap and Foundations

Must-know: Change of variables in 1D: pX(x) = pZ(z)|dz/dx|; training maximizes sum of log pZ(zi) + log|dz/dx| at zi = f_theta(xi); sampling draws z from the base and applies f inverse.

$p_{X} (x) = p_{Z} (z) \frac{\partial z}{\partial x}$

⚠️ Top pitfall: Dropping the absolute value: slopes can be negative but densities cannot. Also, flows have no bottleneck, so they never compress data like autoencoders.

Self-check: If z = (x - 2)/3 and pZ is standard normal, what is pX(5)? Answer: about 0.0807 = pZ(1)/3.

Connects to: 7.2, 7.5, 7.8

From One Dimension to Many: Jacobians and Determinants

Must-know: Determinant cost counting: 2 multiplications at size 2, six paired products (3! terms) at size 3 via Laplace, 24 at size 4, factorial growth overall, versus O(n^3) row-reduction algorithms; triangular Jacobian gives det = product of diagonal entries.

$det (T) = i = 1 \prod n T_{ii}$

⚠️ Top pitfall: Trying brute-force determinants on realistic data: factorial-time expansion is impossible even for a 128x128 matrix; also a zero determinant means linearly dependent columns and a non-invertible degenerate map.

Self-check: Why is a lower-triangular Jacobian ideal for flows? Answer: its determinant is just the product of diagonal entries, cheap to compute and nonzero when diagonals are nonzero.

Connects to: 7.1, 7.3

Autoregressive Flows and the Three Design Requirements

Must-know: The autoregressive map z_i = f_theta_i(x_1..x_i) yields a lower-triangular Jacobian because z_i never depends on later inputs; det = product of diagonal slopes, invertible iff diagonals are nonzero. Three requirements: invertibility, structure, expressiveness (low NLL target).

$J_{ij} = 0 for j > i, ∣ det J ∣ = i = 1 \prod n \frac{\partial z _{i}}{\partial x _{i}}$

⚠️ Top pitfall: A rigid transformation family can train smoothly yet land at a mediocre optimum: a good flow needs both solvability (triangular structure) and expressiveness (very low optimal NLL, e.g. far below zero).

Self-check: Why is the upper triangle of the Jacobian exactly zero? Answer: z_i is defined without reading x_j for j > i, so those partial derivatives vanish.

Connects to: 7.2, 7.4, 7.9

Two-Dimensional Autoregressive Flow with Mixture CDF Transformations

Must-know: The mixture-CDF transform z1 = sum_m pi_m Phi((x1-mu_m)/sigma_m) is monotone (hence invertible) with min 0 and max 1; the 2D objective adds log pZ terms (known, evaluable) and the log-determinant; the log-determinant objective is the flow-specific term to maximize.

$z_{1} = F_{MOG} (x_{1}) = m = 1 \sum M π_{m} Φ (\frac{x _{1} - μ _{m}}{σ _{m}}), F \in [0, 1]$

⚠️ Top pitfall: Treating the base-density terms as literal constants: they move with theta because zi depends on theta; the correct statement is that they are always evaluable since the base distribution is assumed.

Self-check: What are the minimum and maximum of a Gaussian-mixture CDF? Answer: 0 as Theta goes to minus infinity and 1 as Theta goes to plus infinity, because every component CDF tends to those bounds with weights summing to 1.

Connects to: 7.3, 7.5

Sampling from a Trained Flow and the Generation Bottleneck

Must-know: Autoregressive flows train fast (all inputs available, likelihood terms parallelize) but generate slowly (one coordinate at a time); one second of 20 kHz audio = 20,000 samples = 20,000 sequential generation steps, unusable for live speech translation.

$x_{1} = f_{1}^{- 1} (z_{1}), x_{2} = f_{2}^{- 1} (x_{1}, z_{2}), \dots$

⚠️ Top pitfall: Assuming parallel sampling because all latents are drawn at once: the inverse chain still forces each xi to wait for its predecessors. Also, flows never compress: latent dimension equals data dimension.

Self-check: Why can't an autoregressive flow generate coordinates in parallel? Answer: each inverse step needs the previously produced coordinates as input, so the computation is inherently serial.

Connects to: 7.3, 7.6, 7.8

Inverse Autoregressive Flows

Must-know: IAF: generation applies the joint inverse map (x1,...,xn) = f^{-1}(z1,...,zn) in one parallel pass because each xi reads only latents; training is sequential and slow; density evaluation is poor for external points but efficient for its own generations via caching.

$(x_{1}, \dots, x_{n}) = f^{- 1} (z_{1}, \dots, z_{n})$

⚠️ Top pitfall: Assuming an IAF can cheaply score arbitrary external data points: scoring requires the slow serial inversion chain unless the point was generated by the model itself.

Self-check: Which flow should you pick for fast speech synthesis, AF or IAF? Answer: IAF, because sampling is parallel; AF trains faster but samples one coordinate at a time.

Connects to: 7.5, 7.7

Parallel WaveNet: Combining AF and IAF with a Teacher and a Student

Must-know: Four-step procedure: train MAF teacher offline by minimizing NLL, freeze it, train IAF student initialized from cached activations to minimize D_KL(p_S || p_T) = E_{x~p_S}[log p_S(x) - log p_T(x)], then generate in one parallel pass. KL is not symmetric, so it is a divergence, not a distance; zero means identical distributions. Sampling speedup about 1000x over WaveNet.

$D_{K L} (p_{S} ∥ p_{T}) = E_{x \sim p_{S}} [lo g p_{S} (x) - lo g p_{T} (x)]$

⚠️ Top pitfall: Calling KL divergence a distance: swapping arguments changes the value, so it is not symmetric and not a metric. Also, without tricks an all-past-context network would need depth growing with sequence length (about a million layers for one megapixel); dilation and activation caching prevent this.

Self-check: Why does the student sample from its own density during training? Answer: reverse KL weights regions the student visits, so generated samples drive the loss; the frozen teacher only supplies target log-densities.

Connects to: 7.6, 7.9

Using a Trained Flow: Inference versus Sampling

Must-know: Inference applies the forward map to score density: tiny value = rare/unusual data, large value = typical data near the training distribution's heart; sampling runs the inverse on base draws. Flows have no bottleneck: latent dimension equals data dimension, so no compression unlike autoencoders/embeddings.

$p_{X} (x) = p_{Z} (f (x)) ∣ det J ∣$

⚠️ Top pitfall: Reaching for a flow when you need dimensionality reduction: with no bottleneck, flows cannot compress data; their value is exact likelihoods plus sampling.

Self-check: A transaction receives a very small density under a trained flow — what does that suggest? Answer: it is rare or unusual relative to training data, a candidate anomaly.

Connects to: 7.1, 7.5

Masked Autoregressive Flow: A Gaussian Autoregressive Model

Must-know: Scale-and-shift sampler x_i = exp(alpha_i) z_i + mu_i with z_i ~ N(0,1); inverse z_i = (x_i - mu_i) exp(-alpha_i); alpha_i = exp(f_alpha(past)) guarantees a positive spread; MADE-style masking evaluates all coordinates in parallel with a lower-triangular Jacobian; generation remains serial.

$x_{i} = exp (α_{i}) z_{i} + μ_{i}, z_{i} = (x_{i} - μ_{i}) exp (- α_{i})$

⚠️ Top pitfall: Confusing sample quality with generation speed: MAF produces convincing samples and low NLL yet still walks the autoregressive chain one coordinate at a time.

Self-check: Why write the spread as exp(f_alpha)? Answer: a Gaussian spread must be positive, and exponentiating any real network output lands in (0, infinity).

Connects to: 7.3, 7.6, 7.7

Previous LectureLecture 06 · PixelCNN Variants and 1D Normalizing Flows

Next LectureLecture 08 · Normalizing Flows: Masked Autoregressive Models

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.