Deep Neural Networks

Convolutional Neural Networks — From Pixels to Patterns

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning and Computer Vision

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Convolutional Neural Networks — Introduction — covered in Lecture 8
CNN Architecture Overview — covered in Lecture 9
Deep Feedforward Networks — covered in Lectures 6 and 7
Activation Functions (ReLU, Sigmoid, Tanh) — covered in Lectures 4, 5, 6
Gradient Descent and Backpropagation — covered in Lectures 3, 4, 5

Convolutional Neural Networks

10.1 Introduction to Convolutional Neural Networks

A one-megapixel photo has a million pixels. A standard feedforward network connecting each pixel to even 1000 hidden units would need a billion weights. How does a CNN manage the same job with a few hundred parameters?

Think of the children's game Where's Waldo. You don't memorize every pixel's exact location. Instead, you sweep a mental "Waldo detector" across the page — glasses, striped shirt, red hat. The detector is the same no matter where Waldo hides. The brain also does this: a small patch of visual cortex fires for edges or textures in one spot. And the same wiring repeats elsewhere. The analogy breaks for deep layers: Later features can become position-dependent A chin detector only makes sense near the bottom of a face.

10.1.1 What is a CNN

A convolutional neural network (CNN) is a deep neural network built for grid-structured data. It replaces general matrix multiplication with the convolution operation. This slides a small kernel (filter) across the input. It computes a local weighted sum at each position. Three design principles drive CNNs:

Sparse connectivity. Each output neuron connects to only a small receptive field A local patch of the input — rather than every input unit. A hidden unit at position $(i, j)$ sees only pixels within a small window around $(i, j)$ .
Parameter sharing. The same kernel weights are reused at every spatial position. Instead of learning a separate set of weights for each patch, the network learns one kernel. That kernel captures a feature (edge, corner, texture) everywhere it appears.
Equivariance to translation. If you shift the input image, the output shifts by the same amount. Formally, a function $f$ is equivariant to a transformation $g$ if $f (g (x)) = g (f (x))$ . Convolution naturally satisfies this for translation.

Parameter count comparison. Suppose a grayscale image is $1000 \times 1000$ pixels. You map it to a hidden layer of the same size with a fully connected approach. You would need a fourth-order tensor $[W]_{i, j, k, l}$ with $(1 0^{3})^{4} = 1 0^{12}$ parameters — impossible to store or learn. A CNN with a $5 \times 5$ kernel reduces this to just $5 \times 5 = 25$ parameters shared across all positions. Result: ~40 billion times fewer parameters, from $1 0^{12}$ down to $25$ .

Edge-detection efficiency. Detecting vertical edges in a $320 \times 280$ image using convolution with a $1 \times 2$ kernel requires $320 \times 280 \times 3 \approx 267, 960$ floating-point operations. Modeling the same transformation with a fully connected matrix would need $320 \times 280 \times 319 \times 280 \approx 8 \times 1 0^{9}$ matrix entries. That makes convolution roughly 60,000 times more efficient computationally.

10.1.2 Locality

A single pixel, by itself, tells you almost nothing about what the image shows. Its meaning depends entirely on its neighbors. A dark pixel could be part of a letter's edge, a shadow, or just noise You only know by looking around it.

The locality principle says that relevant information for a feature at position $(i, j)$ is concentrated in a small neighborhood around $(i, j)$ . Mathematically, we restrict the kernel offsets $a, b$ to a range $[- Δ, Δ]$ . Here $Δ$ is small (e.g., $Δ = 2$ for a $5 \times 5$ kernel).

$[H]_{i, j} = u + a = - Δ \sum Δ b = - Δ \sum Δ [V]_{a, b} \cdot [X]_{i + a, j + b}$

Each output $[H]_{i, j}$ depends only on a $(2Δ + 1) \times (2Δ + 1)$ patch of the input. The receptive field is this $Δ$ -limited window. As you stack more convolutional layers, deeper units accumulate wider receptive fields. A unit in layer 3 may indirectly see the whole image. Yet each layer's direct connections are local.

Scope: Locality is a modeling assumption. It holds for natural images (nearby pixels are correlated). It fails for tasks where long-range dependencies matter immediately. Examples: reading a serial number that wraps around an image edge, or tracking a global lighting condition. Assumption: The small $Δ$ kernel is enough because deeper layers aggregate local features into global context. This breaks if you use too many pooling layers or strides that shrink the representation before global context can form.

Digit 7 recognition. A $28 \times 28$ digit image. A green-highlighted $3 \times 3$ window slides over the number 7. At the top-left patch, the window sees only blank pixels — zeros. When it reaches the diagonal stroke, the window captures a falling sequence of bright values. Each window position feeds into one output neuron. The result: a $26 \times 26$ grid of local feature responses, each summarizing what happened in a small patch.

One-sentence takeaway: locality compresses the image piece by piece instead of trying to understand it all at once.

10.1.3 Translation Invariance

If a cat moves one pixel to the right, should your model's understanding of the scene change completely? Of course not. Yet a fully connected network would treat the shifted image as an entirely new input with no relation to the original.

Translation invariance (more precisely, translation equivariance) means: If the input $X$ shifts by one pixel, then applying convolution gives the same result as shifting the convolution output. Let $X^{'} (i, j) = X (i - 1, j)$ . The kernel weights $[V]_{a, b}$ do not depend on the absolute position $(i, j)$ ; they only depend on the relative offset $(a, b)$ .

$[H]_{i, j} = u + a \sum b \sum [V]_{a, b} \cdot [X]_{i + a, j + b}$

Notice that $[V]$ has no subscript $i, j$ — the same weights apply at every position. This is the key. The kernel learns a pattern (e.g., vertical edge) and detects it anywhere in the image.

Waldo detector. Imagine a $6 \times 6$ image. Waldo's striped shirt occupies a $3 \times 3$ patch at the bottom-left (rows 3-5, columns 0-2). A $3 \times 3$ kernel trained to fire on red-and-white stripes scans the whole image:

At position $(0, 0)$ : no match → output ≈ 0
At position $(3, 0)$ : full match → output = +1
At position $(0, 3)$ : no match → output ≈ 0

If Waldo moves to the top-right, the same kernel still fires +1 — just now at position $(0, 3)$ instead of $(3, 0)$ . The pattern was detected regardless of position because the detector is translation-equivariant.

Pitfalls:

Equivariance, not invariance. Convolution is equivariant to translation (output shifts with input), not invariant (output stays the same). True invariance requires pooling layers that discard location information.
Not equivariant to scale or rotation. Translation invariance alone does not handle zoomed-in or rotated objects. A 7 that is 50% larger or rotated 30 degrees may not match the learned kernel well.
Boundary artifacts. Near image edges, some kernel positions extend beyond the image. Without proper padding, border pixels are underutilized and the model may miss patterns near edges.
Too much weight sharing can underfit. If your task needs position-specific features, full parameter sharing is the wrong inductive bias. For example, detecting a chin near the bottom of a face photo.

CNNs replace dense matrix multiplication with sparse, shared-weight local connections, achieving massive parameter efficiency From billions of weights down to hundreds By using the twin principles of locality and translation equivariance.

This concludes the introduction. In the next section, we will open up the convolution operation itself. This is the exact arithmetic of sliding, multiplying, and summing. It produces feature maps from raw pixel grids.

CNNs power nearly every modern computer vision system. Examples: face ID on your phone, medical image analysis detecting tumors in CT scans. And self-driving cars identifying pedestrians, lane markings, and traffic signs in real time.

10.1.4 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$W$	Kernel/filter weight	$W_{i, j}$	Scalar, learned
$X$	Input pixel value	$X_{i, j}$	Scalar
$b$	Bias term	$b$	Scalar, learned
$H$	Feature map value	$H_{i, j}$	Scalar
$Δ$	Half-kernel offset range	$Δ$	Scalar
$V$	4D kernel tensor	$[V]_{a, b, c, d}$	Learned tensor

10.2 The Convolution Operation

In the 1980s, image-processing engineers hand-designed filters like the Sobel edge detector by manually choosing numbers in a small grid. A CNN does something radically different: it learns those numbers from raw data. How does the computer do it?

Imagine a credit card-sized red transparent film. Hold it over a photograph. Where the photo is bright red, lots of light passes through; where it's blue or dark, almost nothing. Now slide the film inch by inch across the photo, recording the light intensity at each spot. The film is your kernel. The recorded intensities form your feature map. The analogy breaks because our "film" can have negative weights — it can subtract as well as add.

10.2.1 Definition and Explanation

The convolution operation (technically cross-correlation in deep learning parlance) takes two inputs:

Input tensor $X$ — the image or feature map. For a grayscale image, $X \in R^{H \times W}$ . For a color image with $c$ channels, $X \in R^{c \times H \times W}$ .
Kernel $W$ — the learnable filter. $W \in R^{k_{h} \times k_{w}}$ for 2D convolution, where $k_{h}$ and $k_{w}$ are kernel height and width.

The operation slides the kernel across every valid position of the input, computing a dot product at each stop:

$output at (i, j) = m = 0 \sum k_{h} - 1 n = 0 \sum k_{w} - 1 W_{m, n} \cdot X_{i + m, j + n}$

The result is a single scalar per position. The collection of all such scalars forms the output tensor.

10.2.2 How Convolution Works

Place the kernel at the top-left corner of the image. Multiply every kernel weight by the pixel it sits on. Sum all products. That sum is one output value — the response at position $(0, 0)$ .

Slide the kernel one column to the right. Repeat the multiply-and-sum. Keep going across the row until the kernel would extend beyond the image. Then drop down one row and start again from the left edge. This left-to-right, top-to-bottom sweep produces a 2D grid of scalar responses.

The step size between successive kernel positions is the stride, denoted $S$ . By default $S = 1$ . A stride of 2 skips every other position, producing a smaller output.

Padding adds extra rows and columns (usually of zeros) around the input border. Without padding, a $k \times k$ kernel on an $n \times n$ image yields an output of size $(n - k + 1)^{2}$ . With same padding, zeros are added so the output matches the input dimensions.

Padding modes:

Mode	Output size	Description
Valid	$(n_{h} - k_{h} + 1) \times (n_{w} - k_{w} + 1)$	No padding; kernel stays fully inside input. Output shrinks each layer.
Same	$n_{h} \times n_{w}$ (when $S = 1$ )	Enough zero-padding to preserve input dimensions. Border pixels are less represented.
Full	$(n_{h} + k_{h} - 1) \times (n_{w} + k_{w} - 1)$	Maximum padding so every pixel is visited $k$ times.

Convolution on a $3 \times 3$ image with a $2 \times 2$ kernel (valid padding, stride 1).

Input $X$ and kernel $W$ : $X = 036147258 W = [0213]$

Step 1: Kernel at top-left, covering $X$ rows 0-1, cols 0-1: $(0 \times 0) + (1 \times 1) + (3 \times 2) + (4 \times 3) = 0 + 1 + 6 + 12 = 19$

Step 2: Slide right one column, covering rows 0-1, cols 1-2: $(1 \times 0) + (2 \times 1) + (4 \times 2) + (5 \times 3) = 0 + 2 + 8 + 15 = 25$

Step 3: Drop down one row, covering rows 1-2, cols 0-1: $(3 \times 0) + (4 \times 1) + (6 \times 2) + (7 \times 3) = 0 + 4 + 12 + 21 = 37$

Step 4: Slide right, covering rows 1-2, cols 1-2: $(4 \times 0) + (5 \times 1) + (7 \times 2) + (8 \times 3) = 0 + 5 + 14 + 24 = 43$

Output feature map: $Y = [19372543]$

Sense-check: Input is $3 \times 3$ , kernel $2 \times 2$ , valid padding → output size = $(3 - 2 + 1) \times (3 - 2 + 1) = 2 \times 2$ . The output values grow as the kernel moves toward brighter pixels (bottom-right). This makes sense: input values grow from 0 to 8.

Same-padding example. Input $n = 5$ , kernel $k = 3$ , stride $S = 1$ . Pad with one row/column of zeros on all sides → effective size $7 \times 7$ . Output = $5 \times 5$ — same as input. The corner pixel at $(0, 0)$ is now used only when the kernel centers on position $(1, 1)$ , making border pixels slightly underrepresented.

10.2.3 Kernel Size Conventions

Kernels are almost always square and odd-sized: $3 \times 3$ , $5 \times 5$ , $7 \times 7$ . An odd kernel has a well-defined center; symmetric padding on all sides preserves alignment. Even-sized kernels (e.g., $2 \times 2$ ) are valid but require asymmetric padding — half a pixel offset on one side.

Rectangular filters ( $1 \times 4$ , $5 \times 1$ ) appear when data has asymmetric context. For OCR of handwritten text, horizontal context spans multiple characters. But vertical context is just one line. For audio spectrograms, frequency and time axes have different scales.

Q: Can the filter be $2 \times 2$ ? A: Yes, but odd-sized square kernels ( $3 \times 3$ , $5 \times 5$ , $7 \times 7$ ) are conventional. The reason is centrality — you can point to a center and say surrounding regions relate to it. Odd kernels capture symmetry better. Rectangular filters ( $1 \times 4$ , $5 \times 1$ ) are rare but appear when data has more horizontal context than vertical. Examples: OCR of handwritten sentences, or signal images where different width and height makes sense. For most vision tasks, stick with square odd-sized kernels.

10.2.4 Mathematical Note — Cross-Correlation vs Convolution

In pure mathematics, convolution of a function $f$ with kernel $g$ involves flipping the kernel: $(f * g) (x) = \int f (z) \cdot g (x - z) d z$ . The operation used in deep learning — sliding, multiplying, summing without flipping — is properly called cross-correlation:

$cross-correlation: m, n \sum W_{m, n} \cdot X_{i + m, j + n}$

vs.

$true convolution: m, n \sum W_{m, n} \cdot X_{i - m, j - n}$

The distinction is cosmetic for learning. Since kernel weights are themselves learned, the network trained with cross-correlation simply learns a flipped kernel. The output is identical. The output is identical. Deep learning libraries universally use cross-correlation and call it "convolution."

Pre-CNN vs. CNN filters. Traditional image processing uses fixed, hand-engineered kernels. The Sobel filter detects vertical edges. The Gaussian blur kernel smooths noise. No learning occurs — the engineer chooses the numbers. In a CNN, the kernel values start random and are updated by backpropagation. The network discovers that vertical-edge-like patterns are useful, but it may also invent patterns no human would design.

Scope: The convolution formula assumes the kernel is fully contained within the input at every valid position. For same-padding, border positions use fewer real pixels and more zero-padding The output at edges is less reliable than at the center. Assumption: The 2D spatial grid structure is meaningful. Convolution on shuffled pixels (a permuted image) gives the same result. The operation is position-independent. The learned features will be nonsense for the permuted data.

Picture the convolution as a sliding window animation. A $3 \times 3$ blue-bordered kernel scans left-to-right across a $6 \times 6$ grayscale grid. At each stop, multiplication pairs each kernel weight with its underlying pixel. The sum appears as a single number in a growing $4 \times 4$ output grid below. The output grid is smaller (valid padding) and its values are larger in magnitude because the kernel weights are predominantly positive. One-sentence takeaway: convolution is just a sliding dot product that compresses local pixel neighborhoods into feature scores.

Pitfalls:

Forgetting the bias. The raw convolution sum is often followed by adding a bias term $b$ before the activation. Without it, the model has fewer degrees of freedom to shift the activation threshold.
Ignoring padding effects. Repeated valid-padding convolution shrinks the image by $k - 1$ at each layer. After 4 layers of $3 \times 3$ convolution, a $28 \times 28$ image collapses to $20 \times 20$ — losing border information irreversibly.
Confusing stride and pooling stride. Convolution stride skips input positions during the convolution itself. Pooling stride sums over regions after convolution. They achieve different goals.
Using large kernels unnecessarily. Two stacked $3 \times 3$ convolutions have the same receptive field as one $5 \times 5$ but with fewer parameters ( $18$ vs. $25$ ) and more nonlinearity.

Convolution is element-wise multiplication of a small kernel with every local patch of the input, summed to a scalar. Padding and stride control output size. In deep learning, the kernel is learned, not designed.

Next, we apply this operation and see what it produces: feature maps — compressed, learned representations of the input.

10.2.5 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$W$	Kernel/filter weights	$W_{i, j}$	Scalar, learned
$X$	Input pixel or feature map value	$X_{i, j}$	Scalar
$k_{h}, k_{w}$	Kernel height and width	$k_{h}, k_{w}$	Scalar, odd (3, 5, 7)
$S$	Stride	$S$	Scalar, default 1
$n_{h}, n_{w}$	Input height and width	$n_{h}, n_{w}$	Scalar
$S_{h}, S_{w}$	Horizontal and vertical stride	$S_{h}, S_{w}$	Scalar
$p_{h}, p_{w}$	Total padding rows and columns	$p_{h}, p_{w}$	Scalar

10.3 Feature Maps

After convolution, your image is gone — replaced by a grid of numbers. What are these numbers actually saying? A high value at some position means "the pattern this kernel learned appears here, strongly." But how do you interpret a negative value? And what happens when every value is negative?

Think of a feature map like a heat map overlay on the original image. Where the kernel's pattern matches, the heat map glows bright (positive). Where the opposite pattern appears, it glows dark (negative). Where nothing relevant happens, it's transparent (zero). Each kernel paints its own heat map. One for horizontal edges, one for vertical edges, one for corners, and so on. The analogy breaks when kernels learn patterns that aren't human-interpretable. Deeper layer kernels often capture textures or shapes with no simple name.

10.3.1 What is a Feature Map

A feature map (also called an activation map) is the 2D output produced by convolving one kernel across the entire input. After convolution, you no longer call the result an "image." It is a feature map. This is a compressed representation showing where a particular learned pattern appears.

If the input has $c_{in}$ channels and the layer uses $c_{o u t}$ kernels, the output is a stack of $c_{o u t}$ feature maps. One map per kernel. Formally, for a layer with multiple input and output channels:

$[H]_{i, j, d} = a \sum b \sum c \sum [V]_{a, b, c, d} \cdot [X]_{i + a, j + b, c} + b_{d}$

Each output channel $d$ has its own kernel $[V]_{:, :, :, d}$ and its own bias $b_{d}$ .

10.3.2 Why Bias Matters

The convolution sum on its own is a linear measure of how well the kernel matches the local patch:

$f = i, j \sum W_{i, j} \cdot X_{i, j}$

This sum can be positive, zero, or negative. Positive means the pattern matched. Negative means the inverse pattern matched (pixels are bright where the kernel expects dark, and vice versa).

Then a bias term $b$ is added:

$f = (i, j \sum W_{i, j} \cdot X_{i, j}) + b$

The bias $b$ is a learnable scalar that shifts the feature map values. It serves three critical roles:

Survival through ReLU. The next section introduces ReLU — $max (0, x)$ — which kills all negative values. If a feature map's raw convolution outputs are all negative, ReLU turns everything to zero. For example, inputs like $[- 7, - 10, - 9, - 1, - 2, - 1, - 11]$ . Every neuron becomes dead, passing no information forward. The bias learns to shift the distribution upward so that meaningful signals stay positive and survive.

Channel weighting. In a multi-kernel layer, biases let the model emphasize one feature map over another. This works even when both kernels have similar convolution responses.

Threshold offset. The bias sets the baseline activation level. A kernel detecting "horizontal edge" should fire at 0 for flat regions — the bias calibrates this zero-point.

Bias saves the signal. A vertical-edge kernel produces convolution outputs across a $3 \times 3$ image patch: $Without bias: [- 5, - 3, + 8, - 2, + 12, - 1, - 7, + 3, - 1]$

Pass through ReLU $max (0, x)$ : $[0, 0, 8, 0, 12, 0, 0, 3, 0]$

Only three values survive — the strong edge responses. The mild negatives that suggested "edge-like but not quite" are lost.

Now add a learned bias $b = 4$ : $[- 1, + 1, + 12, + 2, + 16, + 3, - 3, + 7, + 3]$

Pass through ReLU: $[0, 1, 12, 2, 16, 3, 0, 7, 3]$

Seven values survive. The bias shifted weakly negative signals across the zero boundary, preserving texture information that would otherwise be discarded.

Scope: The bias is effective only when the convolution outputs span both positive and negative ranges. If all outputs are strongly positive regardless, adding a bias simply inflates them — it adds no discrimination. Assumption: Each bias creates one degree of freedom per output channel. For a layer with 64 output channels, 64 biases are learned — one per kernel. This assumes each kernel needs an independent activation threshold.

Picture a bar chart with nine bars showing raw convolution values. Three bars are tall and positive (+8, +12, +3). Six bars are negative or near-zero. A red horizontal line at $y = 0$ marks ReLU's cutoff. The same chart is shown again with all bars shifted up by +4 (the bias): now only two bars remain below zero. The bars above zero are colored green (surviving features), those below red (dead neurons). One-sentence takeaway: the bias is a vertical lift that rescues borderline features from the ReLU guillotine.

Pitfalls:

Bias is one per channel, not one per position. A single scalar bias is added to every spatial location of a feature map. Don't confuse it with per-pixel offsets.
Zero-initialized bias is fine. Unlike weights, biases are often initialized to zero or a small positive constant. They will quickly adjust during training.
Bias alone can't fix dead ReLUs everywhere. If the raw convolution consistently produces large negative values regardless of input, the kernel has learned nothing useful No bias magnitude can fix a useless kernel.
Don't forget biases in parameter counts. A layer with 32 kernels of size $3 \times 3$ on 3-channel input has $32 \times 3 \times 3 \times 3 = 864$ weight parameters. It also has 32 bias parameters. Biases are small but not zero.

A feature map is the convolution output plus a per-channel bias — a 2D spatial grid showing where each kernel's pattern fires. The bias shifts values so that informative signals survive the ReLU activation that follows.

Next, we introduce ReLU itself: the nonlinearity that gives CNNs their expressive power.

10.3.3 Symbol Registry

Symbol	Meaning	LaTeX	Type
$b$	Bias term (per output channel)	$b$	Scalar, learned
$b_{d}$	Bias for output channel $d$	$b_{d}$	Scalar, learned
$f$	Feature map value (pre-activation)	$f = \sum (W ⊙ X) + b$	Scalar
$[H]_{i, j, d}$	Feature map output at position $(i, j)$ , channel $d$	$[H]_{i, j, d}$	Scalar
$[V]_{a, b, c, d}$	4D kernel tensor (input height, width, in-channel, out-channel)	$[V]_{a, b, c, d}$	Learned tensor
$c_{in}$	Number of input channels	$c_{in}$	Scalar
$c_{o u t}$	Number of output channels (kernels)	$c_{o u t}$	Scalar

10.4 ReLU Activation and Nonlinearity

Convolution is a linear operation: multiply, sum, repeat. Stack 100 linear layers and you still get... a linear function. When does a deep network become deep in anything but name?

Think of a sculptor with a block of marble. Linear operations are like chiseling with a single flat blade You can only create flat planes no matter how many strokes you make. A nonlinear activation is like switching to a curved gouge — suddenly you can carve curves, hollows, and details. Each stroke builds on the last to create a complex shape. ReLU is the simplest curved tool. Any negative material is simply knocked off. Positive material passes through untouched. The analogy breaks because ReLU is not gradual. It's a hard cutoff at zero. Think of a chisel that either removes material completely or leaves it fully intact.

10.4.1 Why Nonlinearity

Convolution (multiply, sum) and bias addition are both linear or affine operations. Composing linear functions yields another linear function. Without nonlinearity, a stack of convolutional layers collapses into a single linear transformation. No matter the depth, the network cannot learn curved decision boundaries or hierarchical abstractions.

A nonlinear activation function $σ (\cdot)$ is applied element-wise after convolution and bias:

$activation = σ (i, j \sum W_{i, j} \cdot X_{i, j} + b)$

The activation introduces the capacity to model complex, non-linear relationships between input pixels and output classes.

10.4.2 ReLU

ReLU (Rectified Linear Unit) is the dominant activation function in CNNs:

$ReLU (x) = max (0, x)$

Behavior: Every negative input becomes 0. Every positive input passes through unchanged. At $x = 0$ the function is non-differentiable (kink). In practice, gradient-based optimizers handle this by using 0 or the right-limit derivative.

Why ReLU over sigmoid/tanh:

Property	ReLU	Sigmoid	Tanh
Range	$[0, \infty)$	$(0, 1)$	$(- 1, 1)$
Gradient for large $∣ x ∣$	1 (no saturation)	~0 (saturation)	~0 (saturation)
Gradient for $x < 0$	0 (hard zero)	Very small	Very small
Computation	$max (0, x)$	$\frac{1}{1 + e ^{- x}}$	$\frac{e ^{x} - e ^{- x}}{e ^{x} + e ^{- x}}$

Key advantage: no saturation on the positive side. Sigmoid and tanh squash large positive values toward their asymptotes. The gradient approaches zero. Weight updates stall. This is the vanishing gradient problem. ReLU's gradient is exactly 1 for any $x > 0$ , so positive signals propagate through arbitrarily deep networks without attenuation.

ReLU in action on a $3 \times 3$ feature map.

After convolution + bias, the pre-activation values are: $- 3.2 + 4.5 + 1.1 + 2.1 - 1.8 - 2.4 - 0.5 + 0.2 - 0.9$

Apply $ReLU (x) = max (0, x)$ element-wise: $0 4.5 1.1 2.1 00 0 0.2 0$

5 out of 9 values survived. The strongest positive (+4.5) passed through at full strength. The weak negative (-0.5) was killed even though it was close to zero — ReLU has no "gray zone."

Sense-check: Sparsity ratio = 5/9 ≈ 55%. Real CNNs often target 50%–80% sparsity per layer. Sparsity makes representations efficient and reduces co-adaptation of neurons — a form of built-in regularization.

Scope: ReLU works best when the layer's pre-activation distribution is roughly centered near zero with meaningful spread on both sides. If the distribution is heavily skewed positive, ReLU acts like the identity function — no nonlinearity benefit. Assumption: The "dying ReLU" problem (neuron outputs zero for all inputs) is acceptable because the network has redundant capacity. A dead neuron that fired on nothing during training can never resurrect. Its gradient through ReLU is zero, so its incoming weights stop updating.

Picture a 2D plot with $x$ on the horizontal axis ( $- 3$ to $+ 3$ ) and $ReLU (x)$ on the vertical. The curve follows the $x$ -axis flat line for negative $x$ . At $x = 0$ it sharply bends 45 degrees upward. It becomes the identity line $y = x$ for all positive $x$ . Below the plot, a gradient bar shows: red flat zone (gradient 0) for $x < 0$ ; green zone (gradient 1) for $x > 0$ . One-sentence takeaway: ReLU is a switch — off for negatives, fully on for positives — that stacks nonlinearities across layers without saturating gradients.

Pitfalls:

Dying ReLUs. A large negative bias or a large gradient update can push a neuron's pre-activation permanently negative. That neuron stops learning forever. Use Leaky ReLU ( $max (0.01 x, x)$ ) or Parametric ReLU to give dead neurons a small pathway back.
Not zero-centered. ReLU outputs are non-negative (mean $\neq = 0$ ), which can cause zigzagging gradient updates. Batch normalization after ReLU largely mitigates this.
Exploding activations. Unlike sigmoid, ReLU has no upper bound. Very deep networks can accumulate large positive values layer after layer, leading to numerical overflow. Weight initialization schemes (He initialization) and batch normalization control this.
Don't place ReLU before the output layer. For classification, the final layer should use softmax, not ReLU. For regression, the output layer should have no activation (identity) unless the target range is constrained.

Exam note: ReLU avoids the saturation problem of sigmoid and tanh. For $x > 0$ , the gradient is always 1 — no vanishing gradient over positive paths. This is the primary reason almost every modern CNN architecture defaults to ReLU.

Next, we address a subtle but critical companion to ReLU: normalization, which keeps activation magnitudes in check across layers.

Self-driving car perception systems use ReLU throughout their CNN backbone. The sparsity from zeroed-out negative values means only a fraction of neurons fire per image. This makes real-time inference faster on embedded hardware. The ImageNet-winning AlexNet (2012) used ReLU and trained 6x faster than an equivalent tanh network.

10.4.3 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$σ$	Activation function	$σ (\cdot)$	Function
$ReLU (x)$	Rectified Linear Unit	$max (0, x)$	Function

10.5 Normalization

After convolution, bias, and ReLU, some feature maps might produce values in the hundreds while others hover near zero. The network will ignore the quiet ones and obsess over the loud ones. How do you make every kernel's voice heard equally?

Imagine a classroom where one student shouts answers at volume 10 while others whisper at volume 1. The teacher (the optimizer) naturally gravitates toward the loud student. It never hears the quiet ones. This happens even if their answers are better. Normalization is like giving every student a microphone calibrated to the same output volume. Each voice is equally audible, and the teacher can evaluate based on content, not volume. The analogy breaks because normalization is often batch-dependent. The "volume calibration" changes slightly with every minibatch. This adds a subtle noise that can actually help generalization.

10.5.1 Purpose of Normalization

Normalization rescales the values in a feature map to a controlled range Typically zero mean and unit variance, or simply [0, 1]. Without normalization, the magnitudes of activations can vary wildly across layers and channels, causing:

Scale imbalance. One feature map's values might be 100x larger than another's. The larger one dominates gradient updates. The smaller one barely learns.
Internal covariate shift. As weights update during training, the distribution of each layer's inputs constantly changes. Later layers must continuously adapt to this shifting distribution, slowing convergence.
Numerical instability. Unbounded positive values (amplified by ReLU's lack of upper bound) can grow layer after layer, eventually overflowing floating-point precision.

Common normalization methods in CNNs:

Method	Normalization scope	Behavior
Batch Norm	Across batch dimension	Normalizes each channel using mean/variance of the minibatch. Adds learnable scale $γ$ and shift $β$ .
Layer Norm	Across features per sample	Normalizes across all channels for a single example. Common in transformers; less common in CNNs.
Instance Norm	Per channel per sample	Normalizes each channel independently per example. Used in style transfer.
Group Norm	Groups of channels per sample	Compromise between layer norm and instance norm. Works well with small batch sizes.

Batch normalization is the most widely used in CNNs. For a feature map with $B$ examples in the batch, $C$ channels, and spatial dimensions $H \times W$ :

$x^_{i} = \frac{x _{i} - μ _{batch}}{σ _{batch}^{2} + ϵ}, y_{i} = γ x^_{i} + β$

where $μ_{batch}$ and $σ_{batch}^{2}$ are computed over the batch and spatial dimensions for each channel. The term $ϵ$ prevents division by zero. The parameters $γ, β$ are learnable and restore the ability to represent any distribution.

Without normalization. A two-channel feature map after ReLU has these value ranges:

Channel 0 (edge detector): $[- 5, + 25]$ , mean ~3.2, variance ~45
Channel 1 (texture detector): $[- 0.5, + 0.8]$ , mean ~0.12, variance ~0.09

The gradient update for Channel 1's kernel will be ~500x smaller than for Channel 0's kernel at the same learning rate. Channel 1 effectively stops learning.

With batch normalization. After applying BN:

Channel 0: values rescaled to mean ~0, variance ~1, then shifted by $γ_{0}, β_{0}$
Channel 1: values rescaled to mean ~0, variance ~1, then shifted by $γ_{1}, β_{1}$

Both channels now update at comparable rates. The learnable $γ, β$ let each channel rediscover its optimal scale and shift.

Scope: Batch normalization is most effective when batch sizes are large (typically 16+). Small batch sizes produce noisy estimates of $μ_{batch}$ and $σ_{batch}^{2}$ , degrading performance. For batch size 1 or 2, use Layer Norm or Group Norm instead. Assumption: The samples in a minibatch are independent and identically distributed. This holds during training with shuffled data but may break during inference, so running averages of mean/variance are used at test time.

Picture a histogram of activation values for a single feature map: A tall cluster near zero with a long right tail extending to +15 (courtesy of ReLU). Below it, the same histogram after batch normalization: a symmetric bell shape centered at 0, spanning roughly $[- 2.5, + 2.5]$ . The $γ$ and $β$ knobs then shift and stretch this bell to the optimal position for the next layer. One-sentence takeaway: normalization re-levels the playing field so every feature map contributes proportionally to learning.

Pitfalls:

Forgetting the train/test mismatch. During training, batch norm uses minibatch statistics. During inference, you must use the running averages accumulated during training. Setting the model to training mode during evaluation silently uses batch statistics and produces incorrect results.
Applying normalization before or after activation. The standard order is: Conv → BatchNorm → ReLU. Some architectures use Conv → ReLU → BatchNorm. Both are valid but produce different behaviors — be explicit about the order.
Normalization removes the bias. If you use BatchNorm immediately after convolution, the bias term becomes redundant Batch norm's $β$ parameter subsumes it. Setting bias=False saves parameters with no loss.
Over-reliance on normalization for stability. Normalization helps but is not a cure for poor initialization or extreme learning rates. Still use proper weight initialization (He, Xavier).

Normalization rescales feature map values to a controlled range, preventing channel-scale imbalances, reducing internal covariate shift. And enabling higher learning rates for faster, more stable training.

This is the last piece of the basic CNN toolkit. With convolution, bias, ReLU, and normalization, you have the four building blocks that repeat in every modern CNN architecture. Next: pooling layers that downsample and introduce translation invariance.

In the ResNet-50 architecture (2015), batch normalization layers are inserted after every convolution and before every ReLU — over 50 normalization layers. This design choice, along with skip connections, enabled training a 50-layer network that was previously impossible due to vanishing/exploding gradients. Medical imaging pipelines (e.g., tumor segmentation in MRI scans) use instance normalization instead because batch sizes are often small (2-4 scans per GPU) and instance-specific contrast normalization is critical for consistent feature extraction across patients.

10.5.2 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$μ_{batch}$	Batch mean	$μ_{batch}$	Scalar
$σ_{batch}^{2}$	Batch variance	$σ_{batch}^{2}$	Scalar
$ϵ$	Small constant for numerical stability	$ϵ$	Scalar
$γ$	Learnable scale parameter	$γ$	Scalar, learned
$β$	Learnable shift parameter	$β$	Scalar, learned
$x^_{i}$	Normalized activation	$x^_{i}$	Scalar

10.6 Pooling

10.6.1 Why Pooling

Hook: A 28×28 feature map has 784 numbers. Most of them are noise — only a handful actually matter. How do you keep the signal and throw away the noise without learning any new parameters at all?

Intuition + Analogy: Imagine you summarize a book page by page. For each page, you write down only the most important sentence (max pooling). Or you write a one-sentence paraphrase that blends everything (average pooling). Either way, the summary fits on fewer pages than the original. The analogy breaks where real pooling uses a fixed-size sliding window (2×2, 3×3). These are not page boundaries. Regions overlap or skip depending on stride $S$ .

10.6.2 Max Pooling

Formalize — Max Pooling: Let the pooling window be a $2 \times 2$ patch. At each position, max pooling selects the single largest value:

$y = max {H_{i, j}, H_{i, j + 1}, H_{i + 1, j}, H_{i + 1, j + 1}}$

For a feature map $H$ of size $N \times N$ , a window of size $K_{pool} \times K_{pool}$ , and stride $S$ , the output size is:

$O = ⌊ \frac{N - K _{pool}}{S} ⌋ + 1$

When $S = K_{pool}$ (no overlap), the spatial area shrinks by a factor of $K_{pool}^{2}$ . A $2 \times 2$ window with stride 2 compresses 4 values into 1.

Worked Example — Max Pooling (2×2, stride 2):

Given a $4 \times 4$ feature map $H$ :

$H = 7 - 1 40 9 - 5 12 23811026$

Step 1: First patch (top-left $2 \times 2$ ): ${7, 9, - 1, - 5}$ → max = 9

Step 2: Second patch (top-right $2 \times 2$ ): ${2, 1, 3, 0}$ → max = 3

Step 3: Third patch (bottom-left $2 \times 2$ ): ${4, 1, 0, 2}$ → max = 4

Step 4: Fourth patch (bottom-right $2 \times 2$ ): ${8, 2, 1, 6}$ → max = 8

Pooled output ( $2 \times 2$ ):

$output = [9438]$

Sense-check: 16 values compressed to 4. The strongest activations (9, 3, 4, 8) survived. The negative and near-zero values were discarded. This is exactly sparse learning — only strong signals propagate forward.

10.6.3 Average Pooling

Formalize — Average Pooling: Replace the max with the arithmetic mean:

$y = \frac{1}{∣ R ∣} (p, q) \in R \sum H_{p, q}$

where $R$ is the set of indices in the pooling window.

Average Pooling (same input):

$Patch 1: Patch 2: Patch 3: Patch 4: \frac{7 + 9 + ( - 1 ) + ( - 5 )}{4} = \frac{10}{4} = 2.5 \frac{2 + 1 + 3 + 0}{4} = \frac{6}{4} = 1.5 \frac{4 + 1 + 0 + 2}{4} = \frac{7}{4} = 1.75 \frac{8 + 2 + 1 + 6}{4} = \frac{17}{4} = 4.25$

Avg-pooled: $[[2.5, 1.5], [1.75, 4.25]]$ . Notice how each output is a smoothed blend — no single value dominates.

Q: Is average pooling the average of a row or of a patch?

A: Of a patch. Just like a kernel, you define a pooling window size (2×2, 3×3) and consolidate all values from that patch into one. The window slides in two dimensions — height and width — same as convolution.

Q: When should you pick max pooling over average pooling?

A: Max pooling is preferable when features are sparse. A strong edge or corner appears in only a few pixel positions. You want that signal to dominate. Average pooling is better when the entire region carries meaningful texture information (e.g., sky color, grass texture). And you want a smooth representative. In practice, max pooling is the default for most CNN architectures (AlexNet, VGG, ResNet). Average pooling appears in the very last pooling layer before the classifier. For example, global average pooling in ResNet-50.

10.6.4 Pooling Properties

Key properties:

Pooling has zero learnable parameters — no weights, no bias.
Pooling never changes the number of channels — it only reduces spatial height and width.
Gradients flow through pooling during backpropagation, but nothing is updated at the pooling layer.

Scope:

Pooling assumes that the precise spatial location of a feature matters less than its presence. This holds for classification tasks (Is there a face?) but breaks for dense prediction tasks like semantic segmentation. In segmentation, you need pixel-level precision. For example: is pixel (17, 23) a road or a sidewalk? For those tasks, architectures like U-Net skip pooling or use transposed convolution to recover resolution.
Pooling also assumes the pooling window is a regular grid. Irregularly sampled data (point clouds, graphs) need different aggregation — see graph neural networks.

Visual Intuition: Picture a heat map overlaid on the original image — bright red where a vertical edge is detected, faint blue elsewhere. Apply a $2 \times 2$ max-pooling grid over it. Each grid cell keeps only the hottest pixel and discards the rest. The resulting image is half the width, half the height, but all the bright spots survive. If you shift the input image by one pixel, the heat map shifts too. But the brightest point in each $2 \times 2$ region often stays the same value. Even its exact position changed. This is local translation invariance: the pool output is stable under small shifts. (Goodfellow et al., Section 9.3, Figure 9.8 illustrates this precisely.)

Pitfalls:

Underfitting from over-pooling: If you pool too aggressively (large windows, large stride) on a task where fine spatial detail matters, the network cannot learn discriminating features. Training error goes up, not just test error. When in doubt, pool less on early layers where details are fine.
Information loss in max pooling: Max pooling discards position information. You know a strong edge existed somewhere in the 2×2 patch, but you no longer know exactly where. For tasks requiring precise localization, consider alternatives like dilated convolution (no pooling, expanded receptive field) or strided convolution instead.
Confusing pooling stride with convolution stride: They are independent. A convolution with stride 1 followed by pooling with stride 2 differs from a convolution with stride 2 and no pooling. The former applies the full feature detector everywhere, then summarizes. The latter skips positions before detecting. The difference is statistically meaningful.
Pooling window smaller than stride (fractional pooling): Not all frameworks handle this cleanly Some regions of the feature map may be skipped entirely. Standard practice uses $S = K_{pool}$ or $S < K_{pool}$ with overlap.

Recap + Bridge: Pooling downsamples feature maps with no learned parameters — max for sparse signals, average for smooth consolidation. It never changes channel count and is the third stage in every typical CNN layer (Convolution → ReLU → Pool). Next: how padding keeps edges from being neglected during convolution.

Exam note: Pooling has zero learnable parameters. Convolution has learnable kernels and biases. Normalization (e.g., BatchNorm) has learnable $γ$ and $β$ . Fully connected layers have learnable weight matrices and biases. This chain of "has parameters / doesn't have parameters" is a standard exam question.

Real-World & Domain Connection: Max pooling's translation invariance was directly inspired by the primate visual cortex (Riesenhuber & Poggio, 1999). Complex cells pool responses from simple cells. This achieves position-invariant object recognition. The same principle drives face detection in smartphone cameras today: A face detector activates on a 2×2 max-pooled feature map, so a face shifted by a few pixels still triggers the same detector output.

10.6.5 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$K_{pool}$	Pooling window size	$K_{pool}$	Scalar
$O$	Output spatial size after pooling	$O$	Scalar
$R$	Set of indices in pooling window	$R$	Set
$y$	Pooled output value	$y$	Scalar
$H_{i, j}$	Feature map value at $(i, j)$	$H_{i, j}$	Scalar

10.7 Padding

10.7.1 Why Padding

Hook: Convolve a 5×5 image with a 3×3 kernel, and you get a 3×3 output. Repeat 10 times, and your image shrinks to... nothing. How do CNNs survive dozens of layers without vanishing?

Intuition + Analogy: A magnifying glass (kernel) can only bring the center of a page into sharp focus. The edges are blurry because the glass frame hits the paper boundary before the lens reaches the margin. Padding adds a wide paper border — now the lens can slide all the way to the original edge without falling off. The analogy breaks because the border in CNNs is filled with zeros (not blank paper). Zero-padding adds no signal. It just prevents the kernel from missing real pixels.

10.7.2 Same-Size Padding

For same padding with an odd kernel and stride 1, the padding on each side is:

$P = \frac{K - 1}{2}$

For a $3 \times 3$ kernel: $P = (3 - 1) /2 = 1$ — one layer of zeros added to all four sides.

10.7.3 Padding Types

Let input size be $N_{h} \times N_{w}$ (height × width) and kernel size be $K_{h} \times K_{w}$ . With no padding (valid padding), the output shrinks:

$O_{h} = N_{h} - K_{h} + 1, O_{w} = N_{w} - K_{w} + 1$

Add $P_{h}$ total rows of padding. Split roughly half on top, half on bottom. Add $P_{w}$ total columns. Split half on left, half on right. The output size becomes:

$O_{h} = N_{h} - K_{h} + P_{h} + 1, O_{w} = N_{w} - K_{w} + P_{w} + 1$

If we also include stride $S_{h}, S_{w}$ , the formula generalizes:

$O_{h} = ⌊ \frac{N _{h} - K _{h} + P _{h} + S _{h}}{S _{h}} ⌋, O_{w} = ⌊ \frac{N _{w} - K _{w} + P _{w} + S _{w}}{S _{w}} ⌋$

Type	Formula	Effect
Valid	$P = 0$	Output smaller than input
Same	$P_{h} = K_{h} - 1, P_{w} = K_{w} - 1$	Output equals input (with stride 1)
Full	$P_{h} = 2 (K_{h} - 1)$	Output larger than input

Worked Example — Same Padding:

Input: $N = 5 \times 5$ image. Kernel: $K = 3 \times 3$ . Goal: same-size output.

Step 1: Compute padding per side: $P = \frac{3 - 1}{2} = 1$

Step 2: Add one row of zeros on top, one on bottom; one column of zeros on left, one on right. The padded input is now $7 \times 7$ .

Step 3: Convolve with stride 1: $O = ⌊ \frac{5 - 3 + 2 + 1}{1} ⌋ = ⌊ \frac{5}{1} ⌋ = 5$

Output: $5 \times 5$ , matching the input. The corner pixel (0, 0) now has the same chance to be fully covered by the kernel. It matches the center pixel (2, 2).

Sense-check: A $3 \times 3$ kernel centered on the original top-left corner originally couldn't fit — two of its weights would hang off the edge. With one zero-row/column added to each side, the kernel's center can sit directly on every original pixel, including corners. Every pixel gets used equally often.

Counter-example — No padding with stride > 1:

Input: $8 \times 8$ , kernel: $3 \times 3$ , stride: 2, valid padding: $O = ⌊ \frac{8 - 3 + 0 + 2}{2} ⌋ = ⌊ \frac{7}{2} ⌋ = 3$

The output is $3 \times 3$ — less than half the original size, in just one layer.

Scope:

Padding works because input images typically have meaningful content in the center and blank/background near edges. If your images are edge-anchored (e.g., UI screenshots where critical buttons sit at screen borders), zero-padding may wash out important signals. Consider mirror padding or reflection padding in those cases.
Same-padding with odd kernels is clean: $P$ is an integer, so you can distribute padding symmetrically. Even kernels require asymmetric padding — one side gets one more row/column than the other. This is why $3 \times 3$ , $5 \times 5$ , $7 \times 7$ are the standard.

Visual Intuition: Picture a $5 \times 5$ grid of lit pixels on a dark background. Draw a one-cell-thick dark border around the entire grid — the grid is now $7 \times 7$ but the border contributes nothing. Slide a $3 \times 3$ spotlight over it. With the border, the spotlight's center reaches every original cell. Without the border, the spotlight's center only reaches the inner $3 \times 3$ . On a graph of output size vs. layer depth, without padding the line drops linearly toward zero; with same-padding, it stays flat — the output dimensions hold constant across layers.

Pitfalls:

Confusing same vs. valid in frameworks: In TensorFlow, padding='SAME' computes the padding automatically to make output = ceil(input/stride). But the exact padding distribution (top vs. bottom) may be uneven — the framework pads bottom/right first. In PyTorch, you must specify padding manually. Always verify the output shape after the first layer.
Padding with stride > 1 breaks same-size guarantee: Same padding with stride 2 does not produce same-size output. It produces roughly half-size output. The "same" descriptor only holds for stride 1. With stride $S$ , the output is roughly $⌈ N / S ⌉$ .
Assuming padding is free: Padding adds no computation for zeros. But the kernel visits padded positions. These produce nonzero outputs when convolved with nonzero neighbors. Edge effects can propagate. Be aware that padded-layer outputs at the border contain information from fewer real pixels.

Q: Why are odd kernel sizes the standard?

A: Odd kernels (3, 5, 7) have a well-defined center pixel, so symmetric padding produces integer padding amounts. An even kernel (4×4) requires asymmetric padding Say, 1 row on top and 2 on bottom — which shifts the spatial correspondence. The clerical simplicity of odd kernels is a major practical reason they dominate CNN design.

Recap + Bridge: Padding adds zero-filled borders so the kernel can reach edge pixels. It prevents output shrinkage. It ensures all pixels contribute equally. The general output formula combines kernel size, stride, and padding. Next: how all these pieces fit together into the full CNN pipeline.

Exam note: The padding formula $P = (K - 1) /2$ applies only for stride 1, odd kernel, and symmetric same-padding. For strided convolutions, use the full ceiling formula. You may be asked to compute the output size given $N$ , $K$ , $P$ , and $S$ — memorize the general formula.

Real-World & Domain Connection: Zero-padding is the default in every major CNN library because it is computationally trivial. No extra memory allocation is needed. The convolution operator implicitly treats out-of-bounds reads as zero. Satellite imagery pipelines (agriculture, meteorology) use padding to preserve spatial structure across hundreds of spectral channels. Where every pixel's geospatial location must remain aligned through the entire ConvNet.

10.7.4 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$N_{h}, N_{w}$	Input spatial dimensions	$N_{h}, N_{w}$	Scalar
$K_{h}, K_{w}$	Kernel spatial dimensions	$K_{h}, K_{w}$	Scalar
$P_{h}, P_{w}$	Total padding rows and columns	$P_{h}, P_{w}$	Scalar
$O_{h}, O_{w}$	Output spatial dimensions	$O_{h}, O_{w}$	Scalar
$S_{h}, S_{w}$	Stride in each dimension	$S_{h}, S_{w}$	Scalar

10.8 Putting It Together: The CNN Flow

Hook: You've seen convolution, ReLU, normalization, pooling, and padding — each as separate ideas. How do they chain together so that a network with no explicit rules about what a digit 7 looks like can still tell a 7 from a 3 with >99% accuracy?

Purpose: A CNN pipeline transforms raw pixels into a class label. It uses a cascade of operations. These progressively reduce spatial resolution. They also increase the number of learned feature detectors. The pipeline detects local patterns first, then composes them into global shapes, finally feeding a compact feature vector into a classifier.

Inputs: An image tensor of shape $C_{in} \times H \times W$ (channels × height × width). For MNIST: $1 \times 28 \times 28$ . For CIFAR-10: $3 \times 32 \times 32$ .

Outputs: A probability vector over $M$ classes (e.g., softmax output for digits 0–9).

### 10.8.1 The Full Pipeline

Steps — The Full Pipeline:

Stage	Operation	Input Shape → Output Shape	Parameters
1	Convolution + Bias + ReLU	$C_{in} \times H \times W$ → $C_{out} \times H^{'} \times W^{'}$	$C_{out} \times C_{in} \times K \times K$ weights + $C_{out}$ biases
2	Normalization (BatchNorm)	$C_{out} \times H^{'} \times W^{'}$ → same shape	$C_{out}$ $γ$ 's + $C_{out}$ $β$ 's
3	Pooling	$C_{out} \times H^{'} \times W^{'}$ → $C_{out} \times H^{''} \times W^{''}$	None (zero learnable parameters)
4	Repeat stages 1–3 $N$ times	Increasing channels, decreasing spatial	Depends on each layer
5	Flatten	$C \times H \times W$ → $C \cdot H \cdot W$ vector	None
6	Fully Connected + Softmax	Vector length $L$ → $M$ class probabilities	$L \times M + M$ weights + biases

Detailed walkthrough for digit 7 recognition (LeNet-style):

Convolution + ReLU: Apply multiple kernels (horizontal-line detector, vertical-line detector, diagonal-edge detector, etc.) to the $28 \times 28$ input. Each kernel produces a feature map. Bias $b$ shifts activation thresholds. ReLU zeroes out negative responses — keeping only where the pattern matched. Output: 6 feature maps of size, say, $28 \times 28$ (with same-padding) or $26 \times 26$ (with valid-padding).

Normalization: Scale activations across the batch so no single channel dominates. Every feature map now has mean ≈ 0 and variance ≈ 1 across the batch.

Pooling: Apply $2 \times 2$ max-pooling with stride 2. Each $28 \times 28$ feature map becomes $14 \times 14$ . The presence of a horizontal line survives, though its exact pixel location within each $2 \times 2$ patch is lost. This is intentional — you care that a horizontal line exists, not whether it starts at pixel (5, 12) vs. (5, 13).

Repeat: A second Conv-ReLU-BN-Pool block applies 16 kernels to the $14 \times 14 \times 6$ volume. The kernels now operate across all 6 input channels. They detect combinations. For example, a horizontal edge next to a vertical edge at a specific offset. Output: $5 \times 5 \times 16$ feature maps. Early-layer features (edges, corners) have been combined into mid-level features (curves, junctions).

Flatten: Reshape the $5 \times 5 \times 16 = 400$ values into a flat vector of length 400.

Fully Connected: Feed the 400-length vector through one or more dense layers (e.g., 120 → 84 → 10). The final softmax layer outputs probabilities: "95% digit 7, 2% digit 1, 1% digit 9..."

Trace — A single digit 7 through the pipeline:

Input: $1 \times 28 \times 28$ grayscale image of a handwritten 7.

Conv1 (6 kernels, 5×5, valid padding, stride 1):

Output: $6 \times 24 \times 24$ (6 feature maps). The "horizontal bar at top" kernel activates strongly on row 3–4 of the 7. The "diagonal stroke" kernel activates on the middle-right region. The "vertical line" kernel weakly activates — a 7 has no vertical, so this channel is near-zero. ReLU zeros out the vertical-kernel channel almost entirely.

Pool1 (2×2, max, stride 2):

Output: $6 \times 12 \times 12$ . The horizontal-bar activation survives; its spatial extent is halved.

Conv2 (16 kernels, 5×5, valid padding, stride 1):

Output: $16 \times 8 \times 8$ . A kernel sensitive to "horizontal bar ABOVE a diagonal stroke" fires on channels 1 and 3. This kernel learned that 7s have a specific spatial arrangement of edge types.

Pool2 (2×2, max, stride 2):

Output: $16 \times 4 \times 4$ .

Flatten + FC → softmax:

256 numbers → 120 → 84 → 10. The output vector is [0.001, 0.002, ..., 0.001, 0.940, 0.003, 0.005, 0.030] Class index 7 (the 8th position, 0-indexed) has probability 0.94.

Complexity & Cost:

Computational: Each Conv-BN-ReLU-Pool block costs roughly $O (C_{in} \cdot C_{out} \cdot H \cdot W \cdot K^{2})$ operations. The fully connected portion at the end costs $O (L \cdot M)$ where $L$ is the flattened vector length. For a small LeNet on MNIST ( $28 \times 28$ ), this is ~60K multiply-adds per image. For ResNet-50 on ImageNet ( $224 \times 224$ ), this is ~3.8 billion per image.
Memory: The largest tensors are the intermediate feature maps (not the parameters). A batch of 32 images at $64 \times 112 \times 112$ intermediate resolution uses ~100 MB of GPU memory. The FC layers store most parameters. But pooling reduces the flattened vector length dramatically. This is the whole point of the spatial reduction.

When to Use / Alternatives:

Use this pipeline for classification tasks: ImageNet, CIFAR-10, MNIST — any task where the output is a single label per image.
Alternatives for other tasks:
Semantic segmentation: Replace pooling with dilated convolution to preserve spatial resolution, then use a decoder (transposed convolution) to upsample. Architecture: U-Net, DeepLab.
Object detection: Keep spatial structure through the pipeline, output bounding boxes + class labels per region. Architecture: YOLO, Faster R-CNN.
Generative modeling: Reverse the pipeline — start from a latent vector, upsample through transposed convolutions. Architecture: DCGAN generator.

Q: When we reduce dimensions, don't we lose details?

A: It depends on the task. For image classification (cat vs. dog), you do not need pixel-level detail. Detecting whiskers, tail shape, and ear position is enough — those features survive pooling. For semantic segmentation — classifying every pixel as foreground or background — you cannot afford to lose resolution. In those cases, skip aggressive pooling and use transposed convolution or dilated convolution to upsample back to the original size. The pattern detection (convolution) stays the same; you just reconstruct spatial dimensions afterward using a decoder network.

Pitfalls:

Over-pooling before the FC layer: If you aggressively pool to, say, $1 \times 1$ per feature map before flattening, you've thrown away all spatial information. The FC layers can only learn from channel-level stats. This is called global average pooling and is fine only as the very last pooling before softmax Doing it mid-network destroys spatial structure.
Forgetting to flatten differently for batch: After the last pool layer, the tensor shape is [batch_size, channels, height, width]. You must reshape to [batch_size, channels × height × width] — preserving the batch dimension. A common bug flattens the batch into the feature vector.
Mixing training-only operations (dropout, batch norm) in inference: The FC layers often include dropout during training. During inference, dropout must be turned off and batch norm must use running statistics (not batch statistics). Forgetting to switch to eval mode is one of the most frequent deployment bugs.
Assuming deeper is always better: More Conv-BN-ReLU-Pool blocks give more hierarchy. But on small datasets (e.g., 1000 images), too many layers overfit. Match depth to dataset size — shallow LeNet for MNIST (60K images), deep ResNet for ImageNet (1.2M images).

Recap + Bridge: The CNN flow chains Conv-ReLU-Norm-Pool into blocks. It repeats them to build a feature hierarchy. It flattens the result. Finally, FC layers classify it. Spatial resolution falls; channel count rises. Next: the architectural principles — depth, width, and hierarchical detection.

Real-World & Domain Connection: The Conv→ReLU→Pool→Repeat pipeline defined the winning entry of the 2012 ImageNet competition (AlexNet, Krizhevsky et al.). It reduced top-5 error from 26.2% to 15.3%. This >10% absolute improvement triggered the modern deep learning era. ATMs worldwide still run LeNet-5 (LeCun et al., 1998) for check digit recognition using exactly this pipeline. Medical imaging pipelines (CT scan classification, X-ray abnormality detection) use the same flow because the feature hierarchy Edges → textures → organ structures → pathology patterns Mirrors how radiologists read scans.

10.8.2 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$C_{in}$	Number of input channels	$C_{in}$	Scalar
$C_{out}$	Number of output channels (kernels)	$C_{out}$	Scalar
$M$	Number of output classes	$M$	Scalar
$L$	Flattened vector length	$L$	Scalar

10.9 CNN Architecture: Depth, Width, and Hierarchy

Hook: A single convolutional layer detects edges. How many layers does it take to go from edges to faces? The answer reveals the difference between two networks. A shallow one never quite "gets" the image. A deep one recognizes you from any angle.

Intuition + Analogy: Think of a detective squad investigating a crime scene. Rookie detectives (early layers) gather raw evidence — fingerprints, footprints, broken glass shards. Senior detectives (middle layers) combine evidence into partial theories "there was a struggle here" or "the intruder entered through the window." The chief inspector (final layers) synthesizes everything into a verdict "Person A committed the crime." All rookies work independently (no need to coordinate — parallel processing), each scans a small area at a time (local connectivity), and the same evidence-collection procedure works in any room (parameter sharing — translation equivariance). The analogy breaks because in a CNN, all layers are learned end-to-end. This is not rule-based. Also, "chief inspectors" (FC layers) lose spatial awareness entirely.

### 10.9.1 Hierarchical Detection

Formalize — Hierarchical Detection:

The hierarchy emerges from depth. Let $H_{ℓ}$ be the hidden representation after layer $ℓ$ . Each successive layer sees a larger receptive field — the region of the original input image that influences one output pixel:

$RF_{ℓ} = RF_{ℓ - 1} + (K_{ℓ} - 1) \cdot i = 1 \prod ℓ - 1 S_{i}$

where $K_{ℓ}$ is the kernel size at layer $ℓ$ and $S_{i}$ are the cumulative strides of all previous layers (including pooling strides).

Layer 1 (shallow): Receptive field = $K_{1}$ pixels. Sees ~3×3 to 7×7 pixel patches. Detects edges, corners, color blobs.
Layer 3 (mid): Receptive field covers ~15–30 pixels. Detects textures, simple shapes (curves, circles).
Layer 5+ (deep): Receptive field covers ~60+ pixels. Detects object parts (eyes, wheels, door handles) and eventually full objects (faces, cars).

### 10.9.2 Depth vs. Width

Dimension	What it controls	When to increase
Depth (more layers)	Level of abstraction How complex the detected patterns become	Your dataset has hierarchical structure (object parts → objects → scenes)
Width (more kernels per layer)	Number of distinct features detected at the same level	Your data has many visually distinct sub-categories at a given granularity

Adding width without depth gives you many shallow detectors (50 different edge orientations). Adding depth without width gives you deep abstraction but few parallel pathways (one type of edge detector, one curve detector, etc.). In practice, modern architectures increase both — but depth is prioritized because hierarchical composition beats brute-force enumeration.

### 10.9.3 Key Principles

Key Principles:

Local connectivity: Each neuron in a feature map connects only to a small $K \times K$ window of the previous layer, not to every neuron. This enforces the prior that nearby pixels are more related than distant pixels.

Parameter sharing: The same kernel weights $W$ slide across all spatial positions. If a pattern can appear anywhere, you should use the same detector everywhere. This reduces parameters from $O (N^{4})$ (fully connected) to $O (K^{2})$ (convolutional), where $N$ is the image width.

Translation equivariance (not full invariance): If you shift the input image by $Δ$ pixels, the feature map shifts by $Δ$ pixels. The output moves exactly with the input. Full invariance (output unchanged) comes from pooling alone. This distinction matters: equivariance preserves spatial information; invariance discards it.

Parallel processing: Each spatial position in a feature map can be computed independently. Applying the kernel at position (0, 0) does not depend on the result at position (0, 1). This enables GPU parallelism at the pixel level.

### 10.9.4 Parameter Efficiency

Worked Example — Parameter Efficiency:

Setup: A $10 \times 10$ grayscale image (100 pixels). Goal: classify into 10 categories using a network with 2 hidden nodes in the FC layer.

Without CNN — fully connected directly:

Input layer to first FC: $100 \times 2 = 200$ weights + 2 biases = 202 parameters
This is already manageable. But scale up.

With a real image — $256 \times 256$ pixels:

Direct FC: $25 6^{2} \times 2 = 131, 072$ weights. That's just for 2 hidden nodes. For a reasonable 128-node hidden layer: $25 6^{2} \times 128 \approx 8.4$ million parameters — from one layer alone. Train that on a dataset with 60K images, and you have 140× more parameters than examples — guaranteed overfitting.

With a CNN pipeline:

Conv layer: $C_{out} \times C_{in} \times K^{2} = 6 \times 1 \times 3^{2} = 54$ weights + 6 biases.
Conv + Pool reduces the $256 \times 256$ input to, say, $4 \times 4 \times 16 = 256$ flattened values after several blocks.
FC layer: $256 \times 10 = 2, 560$ weights + 10 biases.
Total: ~2,630 parameters — 3,200× fewer than the direct FC approach. And the convolutional weights are actually more expressive because they capture spatial structure.

Scope:

The hierarchical assumption holds for images, speech spectrograms, and video — data with compositional spatial structure. It does not hold for tabular data (spreadsheet rows) where column order is arbitrary. CNNs applied to shuffled tabular columns lose all their inductive bias.
Translation equivariance assumes image statistics are roughly stationary A cat looks like a cat whether it's in the top-left or bottom-right. This assumption is reasonable for natural images. It fails for medical X-rays where anatomy is in fixed positions. It fails for satellite imagery where corners may contain different terrain types than the center.

Visual Intuition: Picture a pyramid diagram — the base is the input image (wide, flat). As you move up, each layer is a new horizontal slice through the pyramid. The slices get narrower (spatial dimension shrinks from pooling) but thicker in channel depth (more feature maps). Label the left side "Spatial Resolution" (decreasing upward) and the right side "Feature Complexity" (increasing upward). The bottom slice shows edge-detection activations; the middle shows shape-detection; the top shows object-part detection. The key takeaway: information flows from where (spatial location) to what (feature identity) as depth increases.

Pitfalls:

Confusing depth with number of layers vs. number of blocks: In common parlance, "101-layer ResNet" counts only weighted layers (conv + FC), not pooling or ReLU. Know what your framework counts when you claim depth.
Stacking depth without skip connections: Pure sequential depth (Conv → Conv → Conv → ...) hits the degradation problem. Beyond ~20 layers, training error increases. Gradients vanish. Optimization gets stuck in poor local minima. Residual connections (ResNet, 2015) solve this by adding identity shortcuts: $H_{ℓ + 1} = F (H_{ℓ}) + H_{ℓ}$ . Without skip connections, depth beyond 20 layers is counterproductive.
Over-widening early layers: Adding 256 kernels to the first conv layer on a small input (28×28) wastes parameters. The receptive field is tiny. Each kernel can only detect trivial patterns. Width should grow as spatial resolution shrinks.
Forgetting that FC layers lose all spatial structure: Once you flatten, the network no longer knows which features were adjacent. This is intentional for classification. But if you ever need to go back to spatial reasoning (e.g., for attention maps), you must preserve the tensor shape or use global average pooling instead of flatten+FC.

Q: How do you decide the number of layers for a new task?

A: Start with a known architecture designed for a similar data type and image size. For 32×32 images, try 3–5 conv layers (CIFAR-10 baseline). For 224×224, try ResNet-18 or ResNet-50. If you must design from scratch, use a heuristic: Keep adding Conv-BN-ReLU-Pool blocks until the spatial dimension drops to ≤ 4×4, then add 1–2 FC layers. Monitor validation accuracy If it plateaus while training accuracy climbs, you may have too many parameters. If both plateau low, you may need more depth or width.

Q: What's the difference between "translation equivariance" and "translation invariance"?

A: Equivariance means the output shifts exactly with the input — convolution alone is equivariant. Invariance means the output stays the same despite input shifts — pooling adds approximate invariance. You want equivariance in early layers so you know where edges are. You want increasing invariance in later layers so you care what objects are present. Not their exact coordinates.

Recap + Bridge: CNN architecture balances depth (feature hierarchy) and width (feature diversity) using local connectivity, parameter sharing, and progressive spatial compression. The four principles — locality, sharing, equivariance, parallelism — enable parameter counts orders of magnitude smaller than fully connected nets. Next: how channels unify color inputs, feature maps, and multi-kernel outputs under one consistent concept.

Exam note: You must distinguish "convolution has learnable parameters" from "pooling has zero parameters." You must also distinguish "convolution is equivariant" from "pooling adds invariance." Both are high-frequency exam distinctions.

Real-World & Domain Connection: The hierarchical depth principle was validated at scale by AlexNet (2012). Which stacked 5 convolutional layers (compared to LeNet-5's 2 conv layers) and won ImageNet by a landslide. VGG (2014) pushed this to 16–19 layers by showing that many small 3×3 kernels in sequence match a larger receptive field with fewer parameters. ResNet-152 (2015) exploited skip connections to reach 152 layers, achieving 3.6% top-5 error on ImageNet — beating humans at the task. Today, Vision Transformers (ViT, 2020) challenge this paradigm by replacing convolution with self-attention. But the depth/width tradeoff and hierarchical design principles remain universal.

10.9.5 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$H_{ℓ}$	Hidden representation at layer $ℓ$	$H_{ℓ}$	Tensor
$RF_{ℓ}$	Receptive field size at layer $ℓ$	$RF_{ℓ}$	Scalar
$K_{ℓ}$	Kernel size at layer $ℓ$	$K_{ℓ}$	Scalar
$S_{i}$	Stride at layer $i$	$S_{i}$	Scalar
$ℓ$	Layer index	$ℓ$	$Z^{+}$
$F$	Learned residual transformation	$F (x)$	Function

10.10 Channels

Hook: A grayscale image has 1 number per pixel. A color image has 3. But after one convolutional layer, your "image" has 64, 128, or even 256 numbers per spatial position. What are all these extra numbers, and how does one kernel handle many channels at once?

10.10.1 Color Channels

Intuition + Analogy: Think of a color image as a stack of three transparent sheets. Red, green, blue — all perfectly aligned. You view them together. The combined intensities form the final color. After convolution, each sheet becomes a feature map — a sheet showing where edges are, where textures exist, where brightness changes. The word "channel" refers to each individual sheet in the stack at any stage — input channels (RGB) or feature map channels. The analogy breaks because feature maps are not independent color separations. They are learned end-to-end. They are jointly optimized to be useful to the next layer. They do not reconstruct the original image.

Formalize — Channel Terminology:

Position in Network	What "Channel" Means	Notation
Input layer	Color components (R, G, B) or sensor bands	$C_{in} = 1$ (grayscale) or $3$ (RGB)
After convolution	Feature maps — one per kernel applied	$C_{out}$ = number of kernels in that layer
After pooling	Same feature maps, spatially reduced	$C_{out}$ unchanged — pooling never alters channels

10.10.2 Feature Map as Channels

A CNN tensor always has the shape:

$C \times H \times W or [batch] \times C \times H \times W$

where $C$ (channels) comes first in PyTorch style and last in TensorFlow style. Know your framework's convention.

10.10.3 Multi-Channel Convolution

Multi-Channel Convolution:

When the input has $C_{in}$ channels and the layer has $C_{out}$ kernels, the full weight tensor has shape:

$W \in R^{C_{out} \times C_{in} \times K \times K} plus b \in R^{C_{out}}$

For one output channel $d$ (one kernel), the operation is:

$H_{i, j}^{(d)} = b_{d} + c = 1 \sum C_{in} a = 0 \sum K - 1 b = 0 \sum K - 1 X_{i + a, j + b}^{(c)} \cdot W_{a, b}^{(c, d)}$

Steps: (1) convolve each input channel with its corresponding 2D sub-filter, (2) sum all the per-channel results element-wise, (3) add the bias $b_{d}$ . This produces one output channel. Repeat for each of the $C_{out}$ kernels to get $C_{out}$ output channels.

Total weights per layer: $C_{out} \times C_{in} \times K \times K$ .

Pooling operates independently per channel — no cross-channel summation.

Worked Example — Multi-Channel Convolution:

Setup: Input $X$ of shape $4 \times 4 \times 2$ (two input channels, each $4 \times 4$ ). One kernel with $2$ sub-filters of size $3 \times 3$ (so kernel shape: $2 \times 3 \times 3$ ). Stride 1, valid padding.

Channel 1 data $X^{(1)}$ : $X^{(1)} = 0481215913261014371115$

Channel 2 data $X^{(2)}$ : $X^{(2)} = 15913261014371115481216$

Sub-filter 1 (for channel 1) $W^{(1)}$ : $W^{(1)} = 100010001$

Sub-filter 2 (for channel 2) $W^{(2)}$ : $W^{(2)} = 200020002$

Step 1: Convolve $X^{(1)}$ with $W^{(1)}$ (valid, $4 \to 2$ ):

Top-left: $0 \cdot 1 + 1 \cdot 0 + 2 \cdot 0 + 4 \cdot 0 + 5 \cdot 1 + 6 \cdot 0 + 8 \cdot 0 + 9 \cdot 0 + 10 \cdot 1 = 0 + 5 + 10 = 15$
Top-right: $1 + 6 + 11 = 18$
Bottom-left: $4 + 9 + 14 = 27$
Bottom-right: $5 + 10 + 15 = 30$

Result 1: $[15271830]$

Step 2: Convolve $X^{(2)}$ with $W^{(2)}$ :

Top-left: $1 \cdot 2 + 5 \cdot 2 + 9 \cdot 2 = 2 + 10 + 18 = 30$
Top-right: $2 + 12 + 22 = 36$
Bottom-left: $10 + 18 + 26 = 54$
Bottom-right: $12 + 20 + 28 = 60$

Result 2: $[30543660]$

Step 3: Sum element-wise: $[15271830] + [30543660] = [45815490]$

Step 4: Add bias $b = 0$ (omitted for simplicity).

Final output (one channel): $2 \times 2$ matrix.

Sense-check: If the kernel has $C_{out} = 6$ such three-dimensional filters, the output would be $6 \times 2 \times 2$ — six channels. Each filter has $C_{in} = 2$ sub-filters. This is exactly the $28 \times 28 \times 6$ output described in the lecture's architecture flow after Conv1.

Worked Example — Architecture Flow with Channel Tracking:

Stage	Input Shape	Operation	Output Shape	Note
Input	—	RGB image	$32 \times 32 \times 3$	$C_{in} = 3$ input channels
Conv1	$32 \times 32 \times 3$	6 kernels, $5 \times 5$ , valid, stride 1	$28 \times 28 \times 6$	Output channels = 6 feature maps
Pool1	$28 \times 28 \times 6$	Max $2 \times 2$ , stride 2	$14 \times 14 \times 6$	Channels unchanged (still 6)
Conv2	$14 \times 14 \times 6$	16 kernels, $5 \times 5$ , valid, stride 1	$10 \times 10 \times 16$	Each kernel has $6 \times 5 \times 5$ weights
Pool2	$10 \times 10 \times 16$	Max $2 \times 2$ , stride 2	$5 \times 5 \times 16$	Channels unchanged (still 16)
Flatten	$5 \times 5 \times 16$	Reshape	400-length vector	$5 \times 5 \times 16 = 400$
FC	400	$400 \times 120$ matrix	120-length vector	Fully connected
Output	120	$120 \times 10$ + softmax	10 probabilities	10-class classification

Key observations:

Spatial dimensions shrink: $32 \to 28 \to 14 \to 10 \to 5$ . Down by a factor of ~6.4×.
Channel count grows: $3 \to 6 \to 16$ . A 5.3× increase.
The network trades spatial resolution for feature diversity — a universal CNN pattern.

Scope:

The multi-channel formulation assumes dense per-channel connectivity: every output channel sums over all input channels. This is the standard. An alternative — depthwise separable convolution — splits the work. One spatial filter per input channel, no cross-channel summation. Then a $1 \times 1$ convolution mixes channels. This reduces parameters drastically. It is the backbone of MobileNet.
Channel ordering conventions differ: PyTorch uses [batch, channels, height, width] ("NCHW"), TensorFlow uses [batch, height, width, channels] ("NHWC"). GPU kernels are typically optimized for one or the other.

Visual Intuition: Picture a thick book where each page is a $32 \times 32$ grid. Page 1 is red intensities, page 2 green, page 3 blue. After Conv1, the book now has 6 pages — each page shows a different pattern (edges, corners, color blobs). After Pool1, each page is half the size ( $14 \times 14$ ). After Conv2, the book has 16 pages — new patterns combining edges + textures. After Pool2, only $5 \times 5$ per page, but 16 pages deep. The "book" is getting physically smaller (spatial dimensions) but thicker (channels). Plot channel count on the y-axis against spatial resolution on the x-axis. You see an inverse curve. This is the hallmark of every deep CNN.

Pitfalls:

Forgetting to account for input channels when defining kernels: If your input has 3 channels, each kernel must have 3 sub-filters. Frameworks handle this automatically in high-level APIs (nn.Conv2d). But if you write low-level code, mismatched channel counts produce errors.
Treating channels as independent: In modern CNNs, channels are not independent. The weighted sum across channels in the next layer means channels are learned to be jointly useful. A channel that detects "vertical edges" coexists with one that detects "horizontal edges." The next layer can combine them to detect "corners." Do not interpret channels in isolation. Think in terms of channel subspaces.
Confusing feature map indices with physical meaning: The 3rd feature map after Conv1 does not correspond to "green" anymore. The network learned its own representation — channel 3 might detect diagonal edges at 45°, or bright spots, or a specific texture. Attribution tools (grad-CAM, feature visualization) can help interpret, but the mapping is learned, not designed.
Pooling channel independence: Pooling processes each channel separately — no cross-channel pooling. If you think pooling is mixing information across channels, you will misunderstand the shape of the output (channels unchanged).

Q: Why do CNNs increase channels as they go deeper?

A: Early layers detect few, general features (edges of a few orientations, a few color contrasts). Later layers need many specialized detectors because the combinatorial space grows: there are many more possible combinations of edges than edges themselves. Increasing channels while decreasing spatial resolution keeps activations roughly balanced. The tensor volume $C \times H \times W$ tends to stay within a factor of 2–4 across layers. This prevents any one layer from becoming a memory bottleneck.

Recap + Bridge: Channels are the depth dimension of CNN tensors — RGB at input, feature maps after convolution. Multi-channel convolution uses per-channel sub-filters whose results are summed, producing one output channel per kernel. Pooling treats channels independently; only convolution mixes across channels. This completes all the building blocks of CNNs.

Exam note: The shape of a convolution kernel is $C_{out} \times C_{in} \times K \times K$ . If asked "how many weights in layer 2," multiply: $C_{out} \cdot C_{in} \cdot K^{2}$ . Plus $C_{out}$ biases. Pooling has zero parameters — channels are unchanged. These are standard computation questions.

Real-World & Domain Connection: The increasing-channels pattern appears in virtually every production CNN: AlexNet (3→96→256→384→384→256 channels), ResNet-50 (3→64→256→512→1024→2048), and EfficientNet (where channel width is scaled by a compound coefficient). The reverse pattern — decreasing channels via $1 \times 1$ convolutions — is used in bottleneck layers (ResNet) to reduce computation. Satellite imagery can have 100+ input channels (hyperspectral bands) — each capturing a narrow wavelength slice, and the channel concept extends naturally. In medical imaging, 3D convolutions treat the third spatial dimension (depth/slices) identically to channels, using a 4D tensor $C \times D \times H \times W$ .

10.10.4 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$C$	Number of channels	$C$	Scalar
$H$	Spatial height	$H$	Scalar
$W$	Spatial width	$W$	Scalar
$X^{(c)}$	Input channel $c$	$X^{(c)}$	Tensor
$W^{(c, d)}$	Sub-filter for input channel $c$ , output channel $d$	$W_{a, b}^{(c, d)}$	Tensor, learned
$H_{i, j}^{(d)}$	Output feature map at $(i, j)$ , channel $d$	$H_{i, j}^{(d)}$	Scalar
$b_{d}$	Bias for output channel $d$	$b_{d}$	Scalar, learned

10.11 The Output Size Formula

10.11.1 Hook

You slide a magnifying glass over a map. Sometimes you skip rows to move faster. Sometimes the map has a white border so the glass can reach the edges. How big is the final annotated map?

10.11.2 Intuition + Analogy

Picture a window cleaner working on a skyscraper. They stand on a scaffold that slides down the building. The cleaner can only work where the scaffold lands fully on the glass If the scaffold hangs over the edge, that area gets skipped. Padding is like extending the building with temporary panels so the scaffold never overhangs. Stride is how many floors the scaffold drops between stops.

You have four knobs:

Input size $N$ — how tall the building is.
Kernel size $K$ — how big the scaffold is.
Padding $P$ — how many extra panels you bolt on each side.
Stride $S$ — how many floors you jump between stops.

The output is the number of stops where the scaffold fully fits.

10.11.3 Formalize

The output size $O$ from any convolution operation is:

$O = ⌊ \frac{N + 2 P - K}{S} + 1 ⌋$

Every symbol:

$N$ — input spatial dimension (width or height), a positive integer.
$P$ — padding per side — the number of zero-rows or zero-columns added to each of the four borders. So the total padding across the dimension is $2 P$ .
$K$ — kernel size — the width/height of the sliding filter (assume square).
$S$ — stride — the step size between successive kernel positions.
$⌊ \cdot ⌋$ — floor function — round down to the nearest integer. You apply it to the entire expression, not the intermediate fractions.

The floor deals with cases where the kernel does not divide the padded input evenly. Any positions where the kernel would overhang are simply dropped.

Equivalent form from standard texts: $⌊ \frac{N - K + 2 P + S}{S} ⌋$

Both are the same. Multiply the professor's form by $\frac{S}{S}$ : $\frac{N + 2 P - K}{S} + 1 = \frac{N + 2 P - K + S}{S} = \frac{N - K + 2 P + S}{S}$ .

The professor's version separates the "+1" so you can see that the first valid position counts as position 1. And each stride adds one more row of outputs. Use whichever form you prefer — they compute the same $O$ .

10.11.4 Worked Examples

Example 1: $N = 32$ , $P = 0$ , $K = 3$ , $S = 2$

$O = ⌊ \frac{32 + 0 - 3}{2} + 1 ⌋ = ⌊ \frac{29}{2} + 1 ⌋ = ⌊ 14.5 + 1 ⌋ = ⌊ 15.5 ⌋ = 15$

Sense-check: a 32×32 image with a 3×3 kernel and stride 2. The kernel slides by 2 each step. After 15 stops it covers the full image — no positions left over.

Example 2: $N = 6$ , $P = 0$ , $K = 3$ , $S = 2$

$O = ⌊ \frac{6 + 0 - 3}{2} + 1 ⌋ = ⌊ \frac{3}{2} + 1 ⌋ = ⌊ 1.5 + 1 ⌋ = ⌊ 2.5 ⌋ = 2$

A 6×6 input with stride 2. Only two complete kernel placements fit.

Example 3: $N = 6$ , $P = 0$ , $K = 3$ , $S = 3$

$O = ⌊ \frac{6 + 0 - 3}{3} + 1 ⌋ = ⌊ \frac{3}{3} + 1 ⌋ = ⌊ 1 + 1 ⌋ = 2$

Stride equals kernel size, so the kernel positions are non-overlapping. Two patches fit across 6 pixels.

10.11.5 Scope

Scope: The formula assumes square inputs and square kernels ( $N = N_{h} = N_{w}$ , $K = K_{h} = K_{w}$ ). For rectangular inputs, compute $O_{h}$ and $O_{w}$ separately using the same formula with their respective dimensions. The formula also assumes zero-padding — padding with a constant value. If you use reflection, mirror, or learnable padding, the spatial arithmetic still holds but border effects differ.

10.11.6 Visual Intuition

Imagine a number line from 0 to $N + 2 P$ . The kernel of size $K$ starts at position 0, slides right by $S$ each step, and stops when its right edge reaches position $N + 2 P$ . Each valid start position produces one output entry. The floor cuts off any partial-window positions at the far right. The "+1" counts the very first placement. A plot of $O$ versus $S$ for fixed $(N, P, K)$ is a decreasing step function — larger strides reduce output size in steps, not smoothly.

10.11.7 Pitfalls

Flooring too early: Computing $⌊(N + 2 P - K) / S ⌋ + 1$ gives the same answer as flooring the whole expression only when the fraction inside the floor is not an integer. If $(N + 2 P - K) / S$ happens to be an integer, both approaches match. But to be rigorous, apply floor to the entire result: $⌊(N + 2 P - K) / S + 1 ⌋$ .

Confusing total padding vs. per-side padding: The professor's $P$ means padding on one side. Total padding added is $2 P$ . The standard textbook $p_{h}$ often means total padding (both sides combined). If a source says $p = 2$ for a $3 \times 3$ kernel, check whether that's 1 per side or 2 per side — the output differs.

Using stride > input size: If $S > N + 2 P$ , the denominator dominates and you get $O = 1$ (just the starting position). But if $S$ is so large that the kernel overhangs the input from the start, you get 0 valid placements The convolution is undefined.

Ignoring spatial extent: The formula gives $O$ for one dimension. The output feature map is $O \times O$ for square inputs. Multiply to get total output pixels.

10.11.8 Exam Drill

Exam note: This formula is one of the most tested in CNN papers. Given any three of $(N, P, K, S, O)$ , you can solve for the other two. With $S = 1$ and $P = 0$ , $K = N - O + 1$ . Example: "A 32×32 input becomes 28×28 after convolution. What was the kernel size if stride = 1 and no padding?" Answer: $K = 32 - 28 + 1 = 5$ .

10.11.9 Recap + Bridge

$O = ⌊(N + 2 P - K) / S + 1 ⌋$ predicts every feature map dimension in your CNN. Next we explore why deeper layers "see" more of the original image — the receptive field.

10.11.10 Real-World & Domain Connection

Every deep learning framework (PyTorch, TensorFlow, JAX) computes output dimensions using this formula internally. When you write nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=2, padding=1), the framework silently runs this arithmetic to allocate the next tensor. In object detection with Feature Pyramid Networks (FPN), you stack multiple conv branches and must match output sizes — the formula tells you which padding and stride combinations align feature maps of different depths for fusion.

10.11.11 Symbol Registry

Symbol	Meaning	LaTeX	Domain
$N$	Input size (spatial dimension, height or width)	$N$	$Z^{+}$
$P$	Padding layers (per side)	$P$	$N_{0}$
$K$	Kernel size	$K$	$Z^{+}$
$S$	Stride	$S$	$Z^{+}$
$O$	Output size	$O$	$Z^{+}$

10.12 Receptive Field Growth

10.12.1 Hook

A single neuron in layer 4 of a CNN "sees" a patch of the original image. How big is that patch? And why does it keep growing as you go deeper?

10.12.2 Intuition + Analogy

Stand one meter from a wall and look through a small window. You see a single brick. Step back to three meters — now you see a section of the wall. Wait — the window didn't grow. But your effective view on the wall did, because the glass now covers a wider angle relative to the wall.

In a CNN, the "window" is always the kernel. But each layer's inputs are themselves windows on earlier layers. A layer-3 neuron connects to a 3×3 patch of layer 2. Which connects to a 5×5 patch of layer 1, which connects to a 7×7 patch of the input image. The view widens even though every kernel is the same size.

This is the receptive field — the region of the original input image that influences a single unit in a given layer. Deeper = wider view = more context for each decision.

10.12.3 Formalize

The receptive field $RF_{i}$ at layer $i$ is:

$RF_{i} = RF_{i - 1} + (K_{i} - 1) \times j = 1 \prod i - 1 S_{j}$

Every symbol:

$RF_{i}$ receptive field at layer $i$ : the side length of the square patch of the original input that one unit at layer $i$ depends on.
$RF_{i - 1}$ — receptive field of the previous layer.
$K_{i}$ — kernel size at layer $i$ .
$S_{j}$ — stride at layer $j$ . The product $\prod S_{j}$ is the cumulative stride — how many input pixels one step at layer $i$ skips over in the original image.

For the first layer: $RF_{1} = K_{1}$ . A 3×3 kernel at layer 1 sees exactly a 3×3 input patch.

The term $(K_{i} - 1)$ captures the expansion: each extra kernel element beyond the first covers one more input pixel. The cumulative stride product stretches that expansion back to the original image scale. If all layers have stride 1, the product is always 1 and the field grows by $(K_{i} - 1)$ per layer.

10.12.4 Worked Example

Assume all layers use $3 \times 3$ kernels with stride 1.

Layer $i$	$RF_{i - 1}$	$K_{i}$	$(K_{i} - 1)$	$\prod S_{j}$	$RF_{i}$
1	—	3	—	—	3
2	3	3	2	1	5
3	5	3	2	1	7
4	7	3	2	1	9

After 4 layers of 3×3 convs with stride 1, each unit at layer 4 sees a 9×9 patch of the original image.

Sense-check: 4 layers of 3×3 each add $2 + 2 + 2 = 6$ pixels of expansion beyond the first 3×3 kernel. Total: $3 + 6 = 9$ . Matches.

Example with strides: Layer 1: $K_{1} = 3, S_{1} = 2$ . Layer 2: $K_{2} = 3, S_{2} = 1$ .

$RF_{1} = 3$
$RF_{2} = 3 + (3 - 1) \times 2 = 3 + 4 = 7$

The stride=2 in layer 1 doubles the expansion contribution of layer 2 Each step in layer 2 covers 2 pixels in the original image.

10.12.5 Scope

Scope: The formula tracks one-dimensional growth (square inputs). The receptive field is a square region for isotropic architectures. For networks with dilation (see Section 10.15), the effective kernel size $K_{eff}$ replaces $K_{i}$ in the formula. The formula also assumes no padding. Padding does not affect which input pixels a unit sees. It only adds zeros that don't change the dependency chain.

10.12.6 Visual Intuition

Picture a pyramid viewed from above. The top layer (deepest) is a single square tile. Draw lines downward That tile connects to a 3×3 region in the layer below. Which fans out to 5×5, then 7×7, then 9×9 at the base (input). The pyramid widens at a rate of $K_{i} - 1$ per step in the stride-normalized coordinate system. If strides are all 1, the growth is linear. If strides exceed 1, the growth accelerates because each down-sampling magnifies the spatial footprint of subsequent layers.

10.12.7 Pitfalls

Forgetting stride product: A stride of 2 in layer 1 doubles the receptive field growth of every subsequent layer. If you just add $(K_{i} - 1)$ without multiplying by the cumulative stride, you underestimate the field size.

Mixing receptive field with output resolution: The receptive field tells you how many original pixels influence one unit. The output resolution tells you how many units exist. These are different concepts. After pooling, resolution shrinks but receptive field keeps growing.

Assuming first-layer RF = 1: The receptive field starts at $K_{1}$ , not 1. A 3×3 kernel at layer 1 sees 9 pixels, not 1.

Ignoring that dilation changes effective K: If a layer uses dilation rate $R$ , replace $K_{i}$ with $K_{eff} = K + (K - 1) (R - 1)$ in the RF formula. Using the raw $K$ will underestimate the true field.

10.12.8 Exam Drill

Exam note: "A CNN has 5 conv layers, all 3×3 kernels, all stride 1. What is the receptive field at layer 5?" Answer: $RF_{5} = 3 + 4 \times (3 - 1) = 3 + 8 = 11$ . Each extra 3×3 layer adds 2 pixels of field width when stride stays at 1.

10.12.9 Recap + Bridge

The receptive field grows by $(K_{i} - 1)$ times the cumulative stride product at each layer. Deeper layers see more context without bigger kernels. Next: how many learnable numbers (parameters) drive these layers?

10.12.10 Real-World & Domain Connection

Receptive field computation is essential when designing object detection networks. In SSD (Single Shot MultiBox Detector), different layers have different fields. Shallow layers with small RF detect small objects like traffic signs. Deep layers with large RF detect large objects like buses and trucks. In semantic segmentation with U-Net, the encoder's deepest layer must have a receptive field large enough to cover the largest structures in the image (e.g., whole organs in medical scans). If the RF is too small, the network guesses pixel labels from not enough context and blobs appear at boundaries.

10.12.11 Symbol Registry

Symbol	Meaning	LaTeX	Domain
$RF_{i}$	Receptive field at layer $i$	$RF_{i}$	$Z^{+}$
$K_{i}$	Kernel size at layer $i$	$K_{i}$	$Z^{+}$
$S_{j}$	Stride at layer $j$	$S_{j}$	$Z^{+}$
$\prod_{j = 1}^{i - 1} S_{j}$	Cumulative stride — magnification from previous layers	$\prod S_{j}$	$Z^{+}$

10.13 Parameter Counting

10.13.1 Hook

A fully connected layer for a 64×64 image at 1000 hidden units needs 4+ billion parameters. A conv layer with the same input, 64 filters, and 3×3 kernel: just 577. How does that math work?

10.13.2 Intuition + Analogy

Imagine you hire a team of 100 surveyors to inspect every window on a 100-story building. If each surveyor needs a unique checklist tailored to each specific window, you need 100×100 = 10,000 checklists. That's a fully connected layer — every input-output pair gets its own weight.

But if all types of windows are structurally the same, you hand out 100 copies of the same checklist. Every surveyor uses the identical template. You now need only the checklist itself — maybe 10 items — plus one pen per surveyor (bias). Total items: $10 \times 100 + 100 = 1100$ . That's a conv layer — parameter sharing across spatial positions.

Convolution saves parameters because one kernel template is reused at every spatial location. The number of learnable numbers depends on the kernel size, the number of input channels. And how many kernels (output channels) you deploy not on the spatial dimensions of the input.

10.13.3 Formalize

The number of learnable parameters for one convolutional layer:

$params = (K \times K \times C_{in} + 1) \times C_{out}$

Every symbol:

$K$ — kernel size (assume square, so $K \times K$ is the spatial area of one kernel).
$C_{in}$ — input channels — the depth of the input tensor. For an RGB image, $C_{in} = 3$ . For a later layer, this is the number of kernels in the previous layer.
$C_{out}$ — output channels — how many distinct kernels (filters) this layer learns. Each kernel produces one output channel (one feature map).
$+ 1$ — one bias term per kernel. Each kernel adds its weighted sum across all input channels and then shifts the result by a scalar bias.

Why $K \times K \times C_{in}$ ? One kernel is a 3D filter: $K$ rows, $K$ columns, across all $C_{in}$ input channels. It slides across space, but the weights themselves are shared — so the filter size counts parameters, not the input's height/width.

Then multiply by $C_{out}$ because you have that many distinct 3D kernels, each with its own bias.

Contrast with fully connected layers: A dense layer connecting a flattened input of size $D$ to $M$ outputs uses $D \times M + M$ parameters. For a 64×64×3 image, $D = 12, 288$ . A conv layer with $C_{out} = 64$ and $K = 3$ needs only $(3 \times 3 \times 3 + 1) \times 64 = 1, 792$ parameters — roughly 6800× fewer.

10.13.4 Worked Examples

Example 1: $C_{in} = 64$ , $K = 3$ , $C_{out} = 1$ (single kernel):

$params = (3 \times 3 \times 64 + 1) = 576 + 1 = 577$

This one kernel produces one output feature map from 64 input channels.

Example 2: $C_{in} = 64$ , $K = 3$ , $C_{out} = 6$ :

$params = (3 \times 3 \times 64 + 1) \times 6 = 577 \times 6 = 3462$

Six distinct kernels, each with 577 learnable numbers, scanning the same 64-channel input to produce 6 output channels.

Example 3: A common architecture pattern: first conv layer on RGB image, $C_{in} = 3$ , $K = 3$ , $C_{out} = 32$ :

$params = (3 \times 3 \times 3 + 1) \times 32 = (27 + 1) \times 32 = 28 \times 32 = 896$

Less than 1000 parameters to transform a raw image into 32 learned feature maps.

10.13.5 Scope

Scope: The formula counts learnable parameters only. It does not count hyperparameters (kernel size, stride, padding values — these are chosen by you, not learned). For layers with biases disabled ( $bia s = F a l se$ in PyTorch), drop the $+ 1$ . The formula also assumes standard 2D convolution with full channel connectivity — each output channel connects to every input channel. In depthwise separable convolution (used in MobileNet), connectivity is restricted: each channel connects to only one input channel, drastically reducing parameter count.

10.13.6 Visual Intuition

Picture a bookshelf. Each shelf ( $C_{out}$ shelves, one per kernel) holds a 3D block — $K$ rows, $K$ columns, $C_{in}$ pages deep. Each entry in the block is one weight. Plus one bookmark (bias) per shelf. The input image is a tall stack of papers Pick a 3D brick, slide it across every position. And compute one output page. Repeat with the next shelf's brick for the next output page. The spatial size of the input stack doesn't matter — the brick's dimensions are the only cost.

10.13.7 Pitfalls

Forgetting the bias: The $+ 1$ per kernel is small for large $K$ or large $C_{in}$ , but in very narrow layers (e.g., $K = 1$ , $C_{in} = 1$ ), the bias can be a significant fraction of the parameter count.

Confusing parameters with FLOPs: Parameter count is static memory (storage). FLOPs (floating-point operations) is runtime computation. A conv layer with 577 parameters may perform millions of FLOPs because those same 577 numbers are reused at thousands of spatial positions. Parameter count ≠ compute cost.

Missing the channel multiplier: Beginners often remember $K \times K + 1$ and forget $C_{in}$ and $C_{out}$ . A 3×3 conv with 128 input channels and 256 output channels has $(9 \times 128 + 1) \times 256 = 295, 168$ parameters — not 2,560. The channel dimensions dominate the count in deep layers.

Assuming parameter count determines memory: The activations (intermediate feature maps) often consume far more GPU memory than the parameters. A 577-parameter conv layer processing a 256×256 image stores 256×256×64 ≈ 4.2 million activation values in its output.

10.13.8 Exam Drill

Exam note: Parameter counting questions are common. A typical question: "A conv layer has 64 input channels, 128 output channels, and 3×3 kernels. How many learnable parameters?" Answer: $(3 \times 3 \times 64 + 1) \times 128 = (576 + 1) \times 128 = 577 \times 128 = 73, 856$ . Always multiply by $C_{out}$ last — the $+ 1$ is per kernel, not per layer.

10.13.9 Recap + Bridge

$params = (K^{2} \cdot C_{in} + 1) \cdot C_{out}$ — one 3D filter per output channel, reused across all spatial positions. Next: how these parameters actually get updated — forward and backward propagation in CNNs.

10.13.10 Real-World & Domain Connection

Parameter counting drives architecture design decisions. VGG-16 has ~138 million parameters — most in the fully connected layers, not the convolutions. ResNet-50 has ~25 million, all convolutional. MobileNet-v2 squeezes to ~3.5 million using depthwise separable convolutions. When deploying to a phone with 2 GB RAM, every million parameters matters The parameter budget determines whether your model fits in the app binary and runs without swapping to disk. Counting parameters is the first feasibility check for any deployment target.

10.13.11 Symbol Registry

Symbol	Meaning	LaTeX	Domain
$C_{in}$	Input channels	$C_{in}$	$Z^{+}$
$C_{out}$	Output channels (number of kernels)	$C_{out}$	$Z^{+}$
$K$	Kernel size (square assumed)	$K$	$Z^{+}$
$+ 1$	Bias term — one scalar per kernel	$+ 1$	Scalar

10.14 CNN Learning: Forward and Backward Propagation

10.14.1 Forward Pass

You've built a deep CNN. Convolution detects edges, pooling compresses, ReLU keeps things non-negative. But how do the kernels learn which patterns to detect — and how do wrong guesses correct themselves?

10.14.2 Backward Pass

A CNN learns by iterating two passes: forward (predict) then backward (correct). The forward pass runs the image through all layers to produce a class prediction and a loss. The backward pass traces the loss back through every operation, computing gradients that nudge each weight toward lower error. Convolution layers update their kernel weights; pooling layers pass gradients through without updating anything. This repeats for thousands of batches until the kernels converge on useful pattern detectors.

#### Inputs & Outputs

Pass	Input	Output
Forward	An image tensor (batch × channels × height × width)	Class probabilities + scalar loss value
Backward	Scalar loss value	Gradients for every learnable parameter (kernel weights, biases)

#### Steps

Forward pass:

Receive input image tensor.
For each conv layer: slide each kernel across the input, compute weighted sum, add bias, apply activation (e.g., ReLU).
Pool: reduce spatial dimensions via max or average pooling.
Repeat steps 2–3 for all conv+pool blocks.
Flatten the final feature maps into a vector.
Feed through fully connected layers to produce class logits.
Compute loss (e.g., cross-entropy) against the ground-truth label.

Backward pass:

Compute gradient of loss with respect to output logits ( $\partial L / \partial logits$ ).
Propagate gradients backward through FC layers using standard backpropagation.
At each conv layer: Compute gradients of the loss with respect to kernel weights ( $\partial L / \partial W$ ) and biases ( $\partial L / \partial b$ ), then compute $\partial L / \partial input$ to pass to the previous layer.
At each pooling layer: Route the gradient to the "winning" position for max pooling (gradient passthrough mask), or distribute evenly for average pooling. No weight updates occur here.
Apply optimizer update: $W \leftarrow W - η \cdot \partial L / \partial W$ for each learnable parameter.
Repeat until the first conv layer — gradients flow through all layers back to the raw input.

#### Key Behaviors

Conv layers: Learnable. Weights are kernels. Biases are per-channel offsets. Both update every backward pass.
Pooling layers: No learnable parameters. Gradient routes through to the input positions that contributed to the pooled output.
ReLU / activation layers: No parameters. Gradient is zero where the pre-activation was $\leq 0$ , and passes through unchanged where the pre-activation was $> 0$ .

#### Student Q&A

Q: Does backpropagation apply all the way through to the first convolution layer?

A: Yes. Gradients flow from the loss through every operation. FC layers, pooling, ReLU, convolution — all the way back to the first kernel's weights. Every conv layer where weights exist receives a gradient update. Pooling and activation layers have no weights, so the gradient merely passes through to the preceding layer.

Q: Can a weight become zero after backpropagation?

A: Yes. If the input contains only one type of pattern, other kernels drift toward zero. For example, only horizontal edges are present. Kernels for vertical edges or diagonals receive small gradients. Their weights drift toward zero. The network automatically allocates representational capacity to the patterns actually present in the data. This is analogous to L1 regularization in traditional ML, where irrelevant features get zero or near-zero coefficients. The learning process itself acts as a feature selector — you don't tell the network which patterns to look for.

Q: How do kernels know what patterns to learn?

A: They don't "know" in any human-interpretable sense. Kernels start as random numbers. During training, gradient descent adjusts them to minimize the loss. The resulting weights respond strongly to certain visual patterns. Edges at specific orientations. Textures. Color blobs. But the machine has no concept of "horizontal line detector." The weights are just numbers that minimize error for the training distribution. At deeper layers, interpretation becomes nearly impossible because the kernels detect abstract combinations of earlier-layer features that have no simple human name.

10.14.3 Vanishing Gradients in Deep CNNs

With 150 convolution layers plus 120 fully connected layers, the gradient signal shrinks dangerously. By the time it reaches early conv layers, it is nearly zero. The weight $\partial L / \partial W_{1}$ may be $1 0^{- 15}$ — so small that $W_{1}$ barely budges even after thousands of iterations.

Early architectures worsened this problem. LeNet (1990s) used sigmoid and tanh activation functions. These saturate — their derivatives are near zero for inputs far from zero, killing the gradient at every layer. AlexNet (2012) used ReLU instead. Its derivative is 1 for all positive inputs. The gradient passes through undiminished on the positive path. This one change made training 8-layer networks feasible and kicked off the deep learning revolution.

Later innovations — batch normalization, residual connections (ResNet), and careful initialization — further addressed vanishing gradients, enabling networks with hundreds of layers.

#### Complexity & Cost

Forward pass cost: Proportional to $O (C_{in} \cdot C_{out} \cdot K^{2} \cdot H \cdot W)$ per conv layer — dominated by the spatial output size.
Backward pass cost: Roughly 2× the forward pass (one pass for weight gradients, one for input gradients).
Memory: Activations from every layer must be stored for the backward pass. A 150-layer network processing 224×224 images can consume 10+ GB of GPU memory just for intermediate activations.
Gradient checkpointing trades computation for memory: recompute activations during the backward pass instead of storing them.

#### When to Use / Alternatives

Scenario	Approach
Shallow CNN (≤ 20 layers)	Standard SGD with momentum, no special gradient tricks needed
Deep CNN (50–150 layers)	Add batch normalization, use residual connections
Very deep CNN (150+ layers)	ResNet-style skip connections essential; consider DenseNet
Limited GPU memory	Gradient checkpointing, mixed-precision training (FP16)
Small dataset, deep model	Transfer learning — freeze early conv layers, fine-tune only the classifier

#### Pitfalls

Dead ReLUs: If a large negative gradient pushes all activations for a kernel below zero, the ReLU derivative is 0 and that kernel receives no further updates. This "dying ReLU" permanently kills the kernel. Using LeakyReLU or proper learning rate scheduling mitigates it.

Exploding gradients in deep CNNs: If weight initialization is too large, gradients compound multiplicatively and overflow (NaN). Use He/Kaiming initialization for ReLU networks.

Forgetting that pooling has no parameters: You cannot "train" a pooling layer — its behavior is fixed by design. If you want learnable downsampling, use strided convolution instead.

Storing the entire computation graph: PyTorch's autograd retains all intermediate activations by default. For large images and deep networks, use torch.no_grad() during inference and call .detach() to free memory.

#### Exam Note

Exam note: Remember the historical progression: LeNet → early CNN, used sigmoid/tanh, suffered from vanishing gradients. AlexNet → introduced ReLU, which does not saturate on the positive side, enabling deeper networks and starting the ImageNet era. Vanishing gradients are the key challenge of depth; ReLU, batch norm, and residual connections are the three main solutions.

#### Recap + Bridge

Forward pass predicts, backward pass corrects. The same chain rule that trains MLPs trains CNNs. Conv layers receive weight updates. Pooling layers route gradients through. Next: a technique that enlarges the receptive field without adding parameters — dilated convolution.

#### Real-World & Domain Connection

Backpropagation through a CNN is the computational backbone of every modern vision system. Tesla's Autopilot trains on millions of driving clips. The backward pass runs on clusters of GPUs for days. It updates kernels that learn to detect lanes, pedestrians, and traffic signs. Instagram's content moderation CNNs use the same backward pass to learn what violates policy. In medical imaging (e.g., detecting tumors in CT scans), the backward pass is what teaches the network that a subtle texture change in a 512×512 slice is cancerous versus benign From thousands of labeled examples.

#### Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$\partial L / \partial W$	Gradient of loss w.r.t. kernel weights	$\partial L / \partial W$	Tensor
$\partial L / \partial b$	Gradient of loss w.r.t. bias	$\partial L / \partial b$	Tensor
$η$	Learning rate	$η$	Scalar
$L$	Loss value	$L$	Scalar

10.15 Dilated (Atrous) Convolution

10.15.1 Motivation

You want a wider field of view — bigger receptive field — but adding layers costs compute and parameters. What if you could zoom out your existing kernel instead?

10.15.2 How It Works

You're looking through a pair of binoculars. At normal magnification ( $R = 1$ ), you see a small patch of the field in focus. You adjust the zoom ring ( $R = 2$ ). Now you see twice the width of the field. But the image has small blank gaps between the lens elements. The gaps are like the zeros inserted into a dilated kernel. They don't add new glass (parameters). They just spread the existing glass apart to cover more area.

Striding is different: striding moves the binoculars faster across the field, skipping some patches. Dilation keeps you checking every patch — just with a wider lens each time. Same number of checks, broader view per check.

10.15.3 Effective Kernel Size

Dilated convolution (also called atrous convolution, from the French à trous — "with holes") inserts zeros between kernel elements. Given a dilation rate $R$ :

$R = 1$ : Standard convolution — no gaps. The kernel behaves as usual.
$R = 2$ : Insert $R - 1 = 1$ zero between every pair of adjacent kernel elements (rows and columns).
$R = 3$ : Insert 2 zeros between every pair.

The effective kernel size — the spatial extent the dilated kernel covers:

$K_{eff} = K + (K - 1) (R - 1)$

Every symbol:

$K$ — original kernel size (number of actual learnable weights per dimension).
$R$ — dilation rate, how many positions apart to place consecutive kernel elements.
$K_{eff}$ — the side length of the region this kernel "reaches" across.

The learnable parameters stay at $K \times K$ — the zeros are non-learnable placeholders. They contribute no gradients and are never updated.

A $2 \times 2$ kernel with $R = 2$ :

Original kernel: $[w_{1} w_{3} w_{2} w_{4}]$

After dilation ( $R = 2$ , spacing = 1): $w_{1} 0 w_{3} 000 w_{2} 0 w_{4}$

$K_{eff} = 2 + (2 - 1) (2 - 1) = 3$ . The effective kernel is $3 \times 3$ , but only 4 weights are learnable. The zeros fill the gaps without consuming parameter budget.

10.15.4 Properties

Zeros are inert: They multiply input pixels by zero — equivalent to skipping those input positions. They receive zero gradient and never change.
Receptive field without parameters: Dilation increases $K_{eff}$ without increasing the parameter count. A $3 \times 3$ kernel with $R = 4$ covers a $9 \times 9$ region but still has only 9 learnable weights.
Spatial resolution preserved: Unlike stride, dilation does not downsample. If you match padding to $K_{eff}$ , the output height and width equal the input. You get a wider view without losing spatial detail.
Sparse coverage: As $R$ increases, the kernel samples input positions farther apart. This works well when features are coarse or spread out, but misses fine-grained local structure.

10.15.7 Comparison with Strided Convolution

Property	Dilated ( $R > 1$ )	Strided ( $S > 1$ )
Output size	Can match input (with padding)	Smaller than input
Receptive field	Expanded via kernel gaps	Expanded via layer stacking
Parameter count	Unchanged	Unchanged
Position coverage	Every input position checked	Some positions skipped
Mechanism	Spaces kernel elements apart	Jumps kernel over input positions
Best for	Dense prediction, segmentation	Downsampling, reducing compute

Q: The comparison above assumes no padding, correct?

A: Yes. With padding, strided convolution can also maintain the output size. The core conceptual difference: dilation inserts zeros inside the kernel to spread its view. Striding jumps the kernel over the input to cover ground faster. With dilation you still visit every position — just with a wider lens. With stride you visit fewer positions, trading coverage for speed.

#### Visual Intuition

Picture the kernel as a comb. With $R = 1$ , the teeth are tight together — you comb through every strand. With $R = 4$ , the teeth have wide gaps. One pass of the comb covers as much width as four passes of the tight comb. Some strands slip through untouched. Stacking multiple dilated convolutions with different rates catches patterns at all scales. For example, rates of 1, 2, 4. This is like combing with progressively wider combs.

10.15.5 When to Use

Scope: Dilation is effective only when the features of interest span across the dilated gaps. For very fine-grained, localized patterns (e.g., individual pixel defects in a manufacturing inspection task), $R > 1$ may skip key detail between sample points. Dilation also increases the intermediate tensor size in the forward pass. The $K_{eff} \times K_{eff}$ effective window stores more activation values. This is more memory than a $K \times K$ window would need. In frameworks, dilated convolution uses the actual convolution operation with expanded sampling. It does not physically construct the zero-padded kernel. The zeros are implicit in the sampling pattern.

#### Applications

Semantic segmentation (DeepLab): Every pixel gets a class label (road, car, sidewalk). Dilated convolutions in DeepLab-v3 use rates $(6, 12, 18)$ in parallel to capture multi-scale context Nearby road texture, mid-range lane markings, and far-field building outlines — all at full spatial resolution.
Dense prediction (object detection): Feature Pyramid Networks pair dilation with skip connections so each level maintains high resolution while aggregating wide context.
Multi-scale feature fusion (InceptionNet / GoogLeNet): Multiple parallel branches with different dilation rates capture patterns at several scales from the same layer Small rates for fine textures, large rates for broad shape context.
Audio and time-series: Dilated 1D convolutions (WaveNet) use exponentially increasing rates $(1, 2, 4, 8, \dots)$ to capture extremely long-range temporal dependencies without deep stacking.

10.15.6 Limitations

Fine detail loss: If the input pattern is smaller than the gap between sample points, dilation misses it entirely.
Gridding artifacts: When stacking many layers with the same dilation rate, the effective sampling creates a sparse grid pattern Certain input positions are never sampled. Use rates that are coprime (e.g., $(1, 2, 3)$ instead of $(2, 4, 8)$ ) or vary rates across layers.
Memory cost: The larger effective window means more multiplications per output position, even though the parameter count is unchanged.
Not a replacement for depth: Dilation expands the receptive field at a single layer, but deep stacking builds hierarchical feature composition. A single dilated layer at $R = 16$ covers a $33 \times 33$ region but cannot compose edges into shapes into objects — that requires multiple non-linear layers.

#### Pitfalls

Confusing R=2 with stride=2: Dilation at $R = 2$ spreads the kernel. Stride at $S = 2$ skips positions. They produce different output shapes and different feature coverage patterns.

Using same dilation rate in every layer: Stacking three layers all at $R = 2$ creates a checkerboard sampling pattern The effective combined field exhibits periodic gaps. Vary rates: $(1, 2, 4)$ or $(1, 2, 3)$ prevents gridding.

Forgetting to increase padding: With $K_{eff} = 3$ and no padding, a $32 \times 32$ input becomes $30 \times 30$ . Despite "dilation preserves resolution" claims, that's only true when you add padding proportional to $K_{eff}$ . Compute padding as $P = ⌊ K_{eff} /2 ⌋$ to maintain output size.

Applying dilation where features are tiny: If your task is counting cells in a microscope image and cells span 5 pixels, a dilation rate of 4 (sampling every 5th pixel) completely misses cell boundaries. Match the dilation to the feature scale.

#### Exam Drill

Exam note: $K_{eff} = K + (K - 1) (R - 1)$ . A $3 \times 3$ kernel at $R = 3$ has $K_{eff} = 3 + 2 \times 2 = 7$ . Parameter count stays at $3 \times 3 \times C_{in} + 1$ per kernel regardless of $R$ . Dilation is most associated with semantic segmentation (DeepLab) and multi-scale feature extraction (Inception modules).

#### Recap + Bridge

Dilation is the zoom lens of CNNs — same glass, wider view, preserved resolution. $K_{eff} = K + (K - 1) (R - 1)$ with unchanged parameter count. Use it when you need large receptive fields at full spatial resolution, especially in segmentation and dense prediction. This concludes the core operations of convolutional neural networks.

#### Real-World & Domain Connection

Google's DeepLab-v3+ achieves state-of-the-art semantic segmentation by combining dilated convolutions at multiple rates (Atrous Spatial Pyramid Pooling — ASPP). Each rate captures a different scale of context A rate of 6 sees nearby pixels (pavement texture), rate 12 captures mid-range structure (car shape), rate 18 captures global context (road layout). All operate at the original resolution, so the final pixel-level predictions are sharp. In medical imaging, dilated convolutions in U-Net variants let radiologists see both micro-calcifications (tiny, local) and organ boundaries (large, global) in the same output. In audio generation, WaveNet's dilated causal convolutions with rates $(1, 2, 4, \dots, 512)$ give each output sample a receptive field covering thousands of past samples Enough to generate realistic speech waveforms.

#### Symbol Registry

Symbol	Meaning	LaTeX	Domain
$R$	Dilation rate	$R$	$Z^{+}, R \geq 1$
$K$	Original kernel size	$K$	$Z^{+}$
$K_{eff}$	Effective kernel size after dilation	$K_{eff} = K + (K - 1) (R - 1)$	$Z^{+}$

10.16 Transposed Convolution

Hook: Convolution shrinks images. But semantic segmentation needs pixel-by-pixel answers — you must zoom back in. How do you reverse the shrinking? Transposed convolution.

10.16.1 What It Does

Intuition: Imagine you photocopied a 4×4 grid down to a 2×2 summary. Now you need the 4×4 back. You spread the 2×2 values apart — inserting zeros between them — then apply a learned filter. The filter fills the gaps. That is transposed convolution.

Transposed convolution (also called deconvolution) is the upsampling operation. It adds zeros between data points to increase spatial dimensions. While regular convolution slides a kernel over the input, transposed convolution does the reverse. It spreads input values out. It applies a learnable kernel to fill the upsampled grid.

Formal Relationship: A regular convolution with kernel size $K$ , stride $S$ , padding $P$ maps:

$N_{in} \to N_{out} = ⌊ \frac{N _{in} + 2 P - K}{S} + 1 ⌋$

Transposed convolution reverses this mapping. Given $N_{in}$ , stride $S$ , kernel $K$ , padding $P$ :

$N_{out} = (N_{in} - 1) \times S - 2 P + K$

The factor $(N_{in} - 1) \times S$ accounts for the zeros inserted between input elements. The kernel and padding terms determine how the boundary is handled.

Worked Example: Input 2×2, kernel 3×3, stride $S = 2$ , padding $P = 1$ :

$N_{out} = (2 - 1) \times 2 - 2 (1) + 3 = 2 - 2 + 3 = 3$

Output: 3×3. You zoom from 2×2 back to 3×3.

Visual intuition: Picture a 2×2 grid of pebbles on a table. Push them apart — insert empty space between each pebble. Now slide a 3×3 magnifying glass over the expanded table. At each position, the glass weights the pebbles it covers and outputs a value. The result is a denser, larger grid.

Scope:

Transposed convolution has learnable parameters (the kernel) — unlike simple nearest-neighbor or bilinear upsampling.
It does not perfectly invert convolution. Information lost during downsampling cannot be recovered — the kernel learns to plausibly fill gaps.
The term "deconvolution" is misleading. True deconvolution is the mathematical inverse of convolution. Transposed convolution is only the shape inverse.

Pitfalls:

Checkerboard artifacts: When stride > kernel size, the zero-insertion pattern creates uneven coverage. Some output positions get more kernel overlaps than others, producing a grid-like pattern. Use kernel sizes divisible by stride to mitigate this.
Trainable but noisy: The upsampled output is only as good as the learned kernel. Without enough downstream constraints (e.g., a segmentation loss), the network may produce blurry outputs.

Recap: Transposed convolution upsamples by inserting zeros and applying a learnable kernel. It is the reverse-shape counterpart to regular convolution. Use it when you need pixel-level outputs — semantic segmentation, super-resolution, autoencoders.

Real-world connection: In medical image segmentation (e.g., U-Net), the encoder downsamples CT scans to extract features. The decoder uses transposed convolutions to upsample back to the original resolution, producing a pixel-by-pixel tumor mask. The kernel learns which patterns to restore at each scale.

10.16.2 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$N_{in}$	Input spatial dimension	$N_{in}$	Scalar
$N_{out}$	Output spatial dimension	$N_{out}$	Scalar
$K$	Kernel size	$K$	Scalar
$S$	Stride	$S$	Scalar
$P$	Padding per side	$P$	Scalar

10.17 Multi-Scale Feature Capture

Hook: A 3×3 kernel sees a small neighborhood. But images contain large objects (a bus) and tiny details (a license plate) in the same frame. How does one network detect both?

10.17.1 The Problem

A single fixed kernel size may not suffice when images contain objects of varied dimensions and shapes. You may need to detect large patterns (wheels, windows), medium patterns (headlights, doors), and fine patterns (badges, text) simultaneously.

10.17.2 The Approach: Parallel Multi-Scale Kernels

Intuition: Instead of choosing one kernel size, run multiple kernels of different sizes in parallel on the same input. A large kernel catches big shapes. A small kernel catches fine texture. Their outputs are concatenated — the next layer sees all scales at once.

InceptionNet (a Google architecture) uses inception blocks Groups of convolutional paths with different kernel sizes (e.g., 1×1, 3×3, 5×5) and dilation rates applied in parallel within the same layer.

How Dilation Enables Multi-Scale: A dilated kernel inserts gaps (zeros) between its weights. A 3×3 kernel with dilation rate $d = 2$ covers a 5×5 receptive field but still uses only 9 parameters. By varying dilation rates, different branches of an inception block see different spatial contexts without increasing parameter count.

Branch	Kernel	Dilation	Effective Receptive Field	Parameters
Fine detail	1×1	—	1×1	1
Local texture	3×3	1	3×3	9
Mid-range	3×3	2	5×5	9
Wide context	5×5	1	5×5	25

All branches run on the same input. Their outputs are stacked depthwise.

Visual intuition: Imagine three photographers shooting the same scene. One uses a macro lens (fine texture), one uses a normal lens (objects), one uses a wide-angle lens (scene layout). You stitch their photos together. The combined image captures everything from pores to panoramas. That is what an inception block does.

Worked Example: Input 32×32×128. An inception block with three parallel paths:

1×1 conv → 32×32×64 (pointwise detail)
3×3 conv, dilation 1 → 32×32×64 (local patterns)
3×3 conv, dilation 3 → 32×32×64 (wider context)

Concatenated output: 32×32×192. Each spatial position now encodes fine, local, and wide-range features.

Scope:

Multi-scale parallelism increases layer width (more output channels). This means more computation and memory.
Inception blocks typically use 1×1 convolutions as bottlenecks to reduce channel count before expensive 3×3 and 5×5 convolutions A practical detail the original lecture defers to later content.

Pitfalls:

Naive stacking is expensive: Running a large 5×5 kernel directly on many input channels is costly. Always bottleneck with 1×1 first.
Scale mismatch: If the object scale in your dataset is known to be relatively uniform, multi-scale blocks add unnecessary complexity.

Recap: Multi-scale feature capture uses parallel kernels of different sizes and dilation rates. Each branch sees a different spatial scale. Concatenating their outputs gives the next layer a richer, multi-resolution representation.

Real-world connection: Autonomous driving perception systems must simultaneously detect distant traffic lights (small) and nearby pedestrians (large). Multi-scale architectures like Feature Pyramid Networks (FPNs) extend this idea hierarchically across layers, enabling detection at every scale.

10.18 Applications Beyond Static Images

Hook: CNNs work on images. But the world is not just 2D photos — there is video (time), medical scans (depth), and higher-dimensional data. What changes when you add dimensions?

10.18.1 Video Processing

Video has both spatial correlation (nearby pixels are related) and temporal correlation (nearby frames are related). CNNs handle video by treating time as extra channels — depth dimensions added to the input tensor.

Input dimensions for a video clip:

$Height \times Width \times Channels \times Frames$

Worked Example: 128×128 resolution, 30 fps, 10-second clip, RGB:

$128 \times 128 \times 3 \times (30 \times 10) = 128 \times 128 \times 3 \times 300$

The 300 frames act as more channels. Just as RGB contributes 3 channels of color, temporal frames contribute 300 channels of motion.

Intuition: Think of a flipbook. Each page is a frame. Stacking all pages gives you a 3D block — width, height, and time-as-depth. The CNN kernel can now slide through time as well as space, learning motion patterns (e.g., "hand moving left" across frames).

10.18.2 Medical Scan Processing

For a CT or MRI scan, the data is volumetric: a stack of 2D slices through the body.

Worked Example: 256×256 image with 20 slices. One volume instance:

$256 \times 256 \times 20$

The 20 slices act as channels. The notion of "channel" adapts to the data type — color channels for RGB, frames for video, slices for medical scans.

The Flexible Channel: A "channel" in a CNN is simply a depth dimension carrying related measurements at each spatial position. The meaning changes by domain:

Domain	Channel Interpretation	Example Dimensions
RGB Images	Color bands	$H \times W \times 3$
Hyperspectral	Many wavelength bands	$H \times W \times 200$
Video	Temporal frames	$H \times W \times 3 \times T$
CT/MRI	Anatomical slices	$H \times W \times D$
Multi-sensor	Different sensor readings	$H \times W \times S$

The CNN does not care what the channels represent — it learns spatial filters that operate across all channels simultaneously.

10.18.3 Symbol Registry: Data Dimensionality

Every piece of data in deep learning lives at some dimensionality. The terminology is standardized:

Term	Meaning	Math Notation	Example
Scalar	0D — a single number	$R$	Temperature: 23.5
Vector	1D — a list of numbers	$R^{n}$	RGB pixel: $[128, 64, 32]$
Matrix	2D — a grid of numbers	$R^{n \times m}$	Grayscale image: 28×28
Tensor	3D+ — arbitrary high-dimensional data	$R^{d_{1} \times d_{2} \times \dots}$	Batch of RGB images: $B \times H \times W \times 3$
4D Tensor	Batch + spatial + channels	$R^{B \times H \times W \times C}$	32 RGB images: $32 \times 256 \times 256 \times 3$

Visual intuition for tensor: A scalar is a point. A vector is a line of points. A matrix is a sheet of numbers. A tensor is a stack of sheets — a 3D block. A 4D tensor is a shelf of such blocks (batch dimension). Each added dimension extends the structure orthogonally.

Scope:

The term "tensor" here follows deep learning convention: any multidimensional array of numbers. This is looser than the mathematical definition (a geometric object with specific transformation rules).
3D convolutions (sliding a 3D kernel through $H \times W \times D$ volumes) are computationally expensive. The "treat-slices-as-channels" approach is a practical simplification for modest depth counts.

Pitfalls:

Channel explosion: Treating 300 frames as 300 channels works for short clips. For longer video, this becomes impractical. Real video CNNs use 3D convolutions or two-stream (spatial + temporal) architectures instead.
Memory cost: A batch of 128×128×3×300 tensors, even with batch size 1, requires $128 \times 128 \times 3 \times 300 \times 4$ bytes ≈ 59 MB just for one example. Scale-aware design is critical.

Recap: CNNs extend beyond 2D images by adding dimensions as channels. Video uses frames as channels. Medical scans use slices as channels. The core convolution operation remains identical — it learns spatial kernels that operate over all provided depth dimensions.

Real-world connection: In self-driving cars, a CNN might process 10 consecutive video frames as 30 channels (10 frames × RGB) to detect motion-based hazards like a pedestrian stepping onto the road. In radiology, a CNN might process 64 CT slices as 64 channels to classify a lung nodule as benign or malignant in a single forward pass.

10.19 Historical Architectures

Hook: CNNs did not leap from theory to AlexNet overnight. Each architecture solved a specific problem that blocked progress. Tracing this lineage reveals why modern networks look the way they do.

10.19.1 LeNet (1990s)

LeNet, developed by Yann LeCun, was the first successful CNN. It showed that convolutional networks could recognize handwritten digits at commercial scale. By the late 1990s, LeNet systems were reading over 10% of all checks in the United States.

Key traits:

Used sigmoid and tanh activation functions
Shallow by modern standards (5–7 layers)
Trained on CPUs with limited data

Limitation: Sigmoid and tanh saturate — their gradients approach zero for large positive or negative inputs. This causes vanishing gradients: during backpropagation, gradient signals shrink exponentially as they flow backward through layers. Deeper LeNets failed to learn on large datasets.

10.19.2 AlexNet (2012)

AlexNet won the ImageNet Large Scale Visual Recognition Challenge 2012 by a large margin, achieving a top-5 error rate of 15.3% (vs. 26.2% for the second-place entry). This was the watershed moment that launched the deep learning era.

Key innovation: ReLU activation. Replacing sigmoid/tanh with ReLU ( $max (0, x)$ ) solved the vanishing gradient problem. ReLU's gradient is 1 for all positive inputs — no saturation, no exponential decay of the gradient signal. This enabled training of 8-layer networks on the massive ImageNet dataset (1.2 million images, 1000 classes).

Other advances:

GPU training (two NVIDIA GTX 580s)
Dropout regularization to prevent overfitting
Data augmentation (flipping, cropping, color jittering)
8 layers: 5 convolutional + 3 fully connected
60 million parameters, trained for 6 days

10.19.3 InceptionNet (Google, 2014)

InceptionNet (also called GoogLeNet) introduced inception blocks — multi-branch parallel convolutions within the same layer. Instead of choosing one kernel size, an inception block runs multiple convolutions in parallel. These are 1×1, 3×3, and 5×5 convolutions (plus pooling). The outputs are concatenated.

Key innovation: Multi-scale parallel kernels. Each branch captures a different spatial scale. 1×1 bottlenecks reduce computational cost before expensive convolutions. The network is deeper (22 layers) but uses 12× fewer parameters than AlexNet.

Why it matters: InceptionNet showed that network design could be modular. The inception block is a reusable architectural pattern. It is not a hand-tuned sequence of layers.

10.19.4 Residual Connections — Preview of ResNet

When networks reach 150+ layers, even ReLU cannot fully prevent vanishing gradients. Gradients must pass through every layer; at extreme depth, the signal still attenuates.

Skip connections (also called residual connections) solve this: the input to a block is added directly to its output. Formally, a residual block computes:

$output = F (x) + x$

where $F$ is a learned transformation (e.g., two convolutional layers). The gradient now has a direct shortcut path. It can flow through $F$ . Or it can bypass entirely via the identity connection $x$ . This enables training of networks with hundreds of layers.

Architecture Comparison Table:

Architecture	Year	Depth	Key Contribution	Activation	Parameters (approx.)
LeNet-5	1998	7	First working CNN; proof of concept	Sigmoid/Tanh	60K
AlexNet	2012	8	ReLU; GPU training; deep learning era begins	ReLU	60M
VGG-16	2014	16	Repeated 3×3 blocks; depth matters	ReLU	138M
InceptionNet	2014	22	Multi-scale parallel kernels; bottlenecks	ReLU	5M
ResNet-152	2015	152	Skip connections; extreme depth possible	ReLU	60M

Why ReLU Won: Compare two networks — identical except one uses sigmoid, one uses ReLU. After 5 layers, the sigmoid network's gradient at layer 1 has been multiplied by 5 sigmoid derivatives (each ≤ 0.25). The gradient is reduced by a factor of ~ $0.2 5^{5} \approx 0.001$ . The ReLU network's gradient is multiplied by 5 values of 1 (for active neurons). It arrives intact.

Visual intuition: LeNet is a bicycle — it proved the concept of wheeled transport. AlexNet is a motorcycle — same principle, but more power (ReLU, GPUs). InceptionNet is a Swiss Army knife — multiple tools in one compact package. ResNet is a highway with express lanes — skip connections let information bypass traffic (layers).

Scope:

LeNet was not "obsolete" — it showed the principles that AlexNet scaled up. The architecture lineage is evolutionary, not revolutionary.
AlexNet's success depended on three simultaneous advances: large datasets (ImageNet), GPU hardware, and algorithmic improvements (ReLU, dropout). Each was necessary; none alone was enough.
Deeper networks may not always be better. ResNet-152 outperforms ResNet-1001 on some benchmarks — extreme depth can overfit or degrade.

Pitfalls:

Confusing architecture with era: LeNet was state-of-the-art for its time. Judging it by modern benchmarks misses that data, hardware, and regularization techniques all co-evolved with architecture.
Over-attributing to ReLU: ReLU solved vanishing gradients for moderate depths. But depth beyond ~20 layers revealed new problems. These were degradation issues, not just vanishing. They required residual connections.

Exam note: Know what each architecture contributed:

LeNet — first CNN; proved convolution works for images.
AlexNet — ReLU activation; started the deep learning revolution.
InceptionNet — multi-scale kernels in parallel within a block.
ResNet (preview) — skip connections enable training of very deep networks.

Deeper architectural discussion follows in later course content.

Real-world connection: The pattern of architectural innovation — modular blocks, activation function improvements, and depth-enabling techniques — repeats across domains. Transformer architectures in NLP mirror the same trajectory: attention replaced recurrence (like ReLU replaced sigmoid). And residual connections became standard in both vision and language models.

10.19.5 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$max (0, x)$	ReLU activation	$max (0, x)$	Function
$F (x)$	Learned residual transformation	$F (x)$	Function

10.20 Stride vs. Pooling for Downsampling

Hook: You need to shrink your feature maps. You have two tools: stride (jump further during convolution) and pooling (aggregate within a window). They achieve the same spatial reduction. So when do you pick which?

10.20.1 Two Paths to Downsampling

Both striding and pooling reduce spatial dimensions. The difference is how they combine information:

Strided convolution — the kernel jumps by $S$ pixels instead of 1. Each output value is a weighted sum of a larger input region (the kernel's learned weights do the combining). Has learnable parameters.

Pooling — a fixed window aggregates values with a deterministic rule (max or average). Each output value is simply the strongest signal (max) or the mean signal (average) from that region. Zero learnable parameters.

Comparison Table:

Property	Strided Convolution	Pooling (Max/Avg)
How it shrinks	Kernel jump size $S > 1$	Window stride $S > 1$
Aggregation rule	Learned kernel weights	Fixed: max() or mean()
Learnable parameters	Yes	No
Output meaning	Weighted feature combination	Dominant feature (max) or smoothed estimate (avg)
Computational cost	Higher (dot products)	Lower (comparisons or sums)
Gradient flow	Through kernel weights	Routes through max location (max pool) or averages (avg pool)
Best when	Features need learned spatial reduction	Features need invariance to small translations

Intuition: Strided convolution says: "I want to learn how to combine a 5×5 region into one number." Pooling says: "Just give me the strongest signal in that 5×5 region. I don't need to learn how to combine."

10.20.2 When to Stride Instead of Pool

If features are sparse — the meaningful patterns are scattered and not densely packed — a large stride can replace pooling. Sparse features do not need dense pooling because there is less risk of losing information by skipping spatial positions.

Striding gives each neuron a larger receptive field — the kernel sees more of the input because it jumps further between applications. A 3×3 kernel with stride 2 covers the same spatial extent as a max-pooling layer with a 2×2 window and stride 2. But with learned weighting.

Worked Example: Input 64×64. You want 32×32 output.

Pooling path: 2×2 max-pooling, stride 2 → 32×32. Each output is the max of a 2×2 patch. No learning, fast, translational invariance.

Strided convolution path: 3×3 convolution, stride 2, padding 1 → 32×32. Each output is a learned dot product over a 3×3 patch. Learned, more expressive, more computation.

Visual intuition: Max-pooling is like a "best-of" highlight reel — you keep only the strongest signal from each region. Strided convolution is like a summary writer who learns which details matter They read the whole region and compose a weighted summary.

Scope:

Both techniques trade fine-grained spatial resolution for computational efficiency. Once you downsample, you cannot recover the discarded detail (unless you use skip connections like U-Net).
The choice depends on domain knowledge. For structured patterns (edges, textures), learned striding may capture more. For noisy signals where only the dominant activation matters, max-pooling suffices.

Pitfalls:

Information loss from large stride: A stride of 3 with a 3×3 kernel skips 2 out of every 3 positions. If features are dense and fine-grained, you lose critical spatial detail.
Checkerboard artifacts (strided convolution): When kernel size and stride are not aligned, output pixels receive uneven numbers of contributions, creating grid patterns. Prefer kernel sizes that are multiples of the stride.
Max-pooling discards sub-maximal activations: The second-strongest activation in a window might carry valuable information. Max-pooling throws it away. Average pooling preserves more information but blurs strong features.

Recap: Strided convolution and pooling both downsample. Strided convolution learns how to aggregate — use it when feature combinations matter. Pooling applies a fixed rule — use it for efficiency and translational invariance. The choice is domain-dependent; modern architectures often use strided convolution for the first downsampling step and pooling for later ones.

Real-world connection: In object detection (e.g., YOLO), the backbone network uses strided convolutions to aggressively reduce spatial dimensions while preserving learned features. In classification (e.g., VGG), max-pooling between convolutional blocks provides translational invariance The network does not care exactly where the cat's ear is, only that the ear pattern exists somewhere in the region.

10.20.3 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$S$	Stride	$S$	Scalar
$K$	Kernel size	$K$	Scalar
$P$	Padding per side	$P$	Scalar

Exam Guidance Summary

This section consolidates the key formulas, concepts, and question types you must know for assessment. All derive from Lecture 10 content.

Core Formulas

Output size formula: $O = ⌊(N + 2 P - K) / S + 1 ⌋$ . Memorize it. You can rearrange to find $K$ , $P$ , or $S$ . Always take the floor.

Parameter counting (convolutional layer): $params = (K \times K \times C_{in} + 1) \times C_{out}$ . The "+1" accounts for the bias term per output channel. Total operations = params × output positions.

These two formulas appear in nearly every exam question on CNN architectures. Write them down first when you see a CNN problem.

Expected Question Types

Given an architecture: Find kernel size, output size, or number of parameters at any specified layer.
Work backward: Given input size and output size, determine the kernel size, padding, or stride.
Sequential layers: Trace an input through multiple convolution-pooling layers, computing dimensions at each stage.
Parameter comparison: Compare fully connected vs. convolutional parameter counts for the same input.

Conceptual Knowledge

Concept	What to Know
Pooling	Zero learnable parameters. Aggregates by max or average. Shrinks spatial dimensions.
Convolution	Has learnable kernels (weights) and biases. Number of parameters grows with $K^{2} \times C_{in} \times C_{out}$ .
ReLU	Introduces nonlinearity. Avoids vanishing gradients in moderate-depth networks. Gradient = 1 for all positive inputs.
Padding	Preserves spatial dimensions when stride = 1. "Same" padding adds zeros equally around borders.
Dilation	Increases receptive field without new parameters. Insert zeros between kernel weights. Rate $d = 2$ on 3×3 → effective 5×5 field.
Transposed convolution	Upsamples by inserting zeros and applying a learnable kernel. Used for pixel-level outputs.

Historical Contributions

LeNet (1998): First working CNN. Sigmoid/Tanh activations. Proved convolution works for digit recognition.
AlexNet (2012): Replaced sigmoid with ReLU. Enabled deep training on ImageNet. Launched the deep learning era.
InceptionNet (2014): Multi-scale parallel kernel blocks. Captures features at multiple scales within one layer.

What Is NOT Required

You will not be asked to compute gradients through convolution layers by hand.
You will not be required to derive backpropagation formulas for pooling or convolution.
Architecture minutiae (exact layer counts of every variant) are less important than understanding the design principles and innovations.

Strategy: If you see an architecture diagram, first note $N, C_{in}, K, S, P$ for each layer. Compute $O$ layer by layer. Count parameters per layer using the formula. This covers the vast majority of exam questions.

Key Industry Applications

CNNs power most modern computer vision systems. This appendix catalogs the major application domains and the specific CNN techniques each employs.

Classification and Recognition

Digit and image classification: CNNs classify handwritten digits (MNIST), objects (ImageNet), and scenes. The standard pipeline: convolutional feature extraction → pooling → fully connected classifier.

Optical Character Recognition (OCR): Rectangular (non-square) filters handle the horizontal flow of handwritten text. Horizontal kernels capture letter strokes; vertical kernels capture line separation.

Dense Prediction Tasks

Semantic segmentation: Pixel-level classification — every pixel gets a label (foreground/background, or one of many classes). Uses dilated convolutions to maintain wide receptive fields without losing resolution and transposed convolutions to upsample back to the original image size. U-Net and DeepLab are prominent architectures.

Object detection: Locates and classifies multiple objects in a single image. Handles cluttered scenes, occlusions, and traffic-dense environments. Architectures like YOLO and Faster R-CNN combine region proposal with CNN classification.

Temporal and Volumetric Data

Video analysis: Action recognition, frame-by-frame event detection. Temporal frames are treated as more channels (for short clips) or processed with 3D convolutions (for longer sequences). Applications include surveillance, sports analytics, and autonomous driving.

Medical imaging: CT and MRI slice analysis — each anatomical slice becomes a channel dimension. CNNs detect tumors, fractures, and anomalies. Key difference from natural images: medical images are grayscale and have precise physical scale (mm per pixel).

Traditional vs. Learned Approaches

Traditional computer vision: Fixed Sobel filters, Canny edge detector, and Gabor kernels detect edges, corners, and textures. These kernels are hand-designed based on mathematical models of image structure. No learning — the kernel values are constants. Still used in real-time systems where deep learning is too expensive.

Deep learning frameworks: TensorFlow and PyTorch automate padding calculations via built-in modes:
'valid' — no padding; output shrinks
'same' — pad to keep output size equal to input (when stride = 1)
'full' — pad enough that every kernel position overlaps the input at least partially

The same CNN building blocks — convolution, pooling, ReLU, dilation, transposed convolution — serve all these applications. The difference is how you assemble them: Classification uses pooling to shrink, segmentation uses transposed convolution to expand, detection uses both in a single network.

Beyond Vision

CNNs have also been successfully applied to:

Audio processing: 1D convolutions over time-series waveforms (speech recognition, music genre classification)
Natural language processing: 1D convolutions over word sequences (text classification, sentiment analysis)
Graph data: Graph convolutional networks (GCNs) generalize convolution to irregular graph topologies

These applications use the same core principle: a learnable filter slides over structured data and detects local patterns.

DNN Lecture 10 notes · Convolutional Neural Networks — From Pixels to Patterns

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

1Introduction to Convolutional Neural Networks

Three design principles of CNNs: sparse connectivity, parameter sharing, and translation equivariance

2The Convolution Operation

Sliding kernel dot product, padding modes (valid, same, full), stride, and kernel size conventions

3Feature Maps

Feature map as convolution output, role of bias in preserving signals through ReLU

4ReLU Activation and Nonlinearity

Why nonlinearity is essential, ReLU properties, comparison with sigmoid/tanh

5Normalization

Batch normalization rescales activations, prevents scale imbalance, enables higher learning rates

6Pooling

Max pooling and average pooling for downsampling with zero learnable parameters

7Padding

Same, valid, and full padding modes with output size formulas

8Putting It Together: The CNN Flow

Complete Conv-ReLU-Norm-Pool pipeline traced through digit recognition

9CNN Architecture: Depth, Width, and Hierarchy

Hierarchical feature detection, depth vs width tradeoff, four key principles

10Channels

Input channels, feature map channels, multi-channel convolution, channel tracking through layers

11The Output Size Formula

Formula derivation, worked examples, floor function handling

12Receptive Field Growth

RF formula with cumulative stride product, worked examples

13Parameter Counting

Formula for learnable parameters in convolutional layers, comparison with fully connected

14CNN Learning: Forward and Backward Propagation

Forward pass pipeline and backward pass gradient flow through convolution and pooling

15Dilated (Atrous) Convolution

Effective kernel size formula, comparison with strided convolution, applications

16Transposed Convolution

Upsampling with learnable kernels, output size formula

17Multi-Scale Feature Capture

Parallel kernels of different sizes/dilations for multi-scale representation

18Applications Beyond Static Images

Video, medical scans, tensor dimensionality concepts

19Historical Architectures

LeNet, AlexNet, InceptionNet, ResNet — contributions and innovations

20Stride vs. Pooling for Downsampling

Comparing strided convolution and pooling for spatial reduction

Postgraduate students in Machine Learning and Computer Vision

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Convolution Operation

Must-know: Convolution slides a learnable kernel across the input computing a dot product at each position. Parameter sharing (same weights everywhere) and sparse connectivity (local receptive fields) reduce parameters from $O (N^{4})$ to $O (K^{2})$ .

$output at (i, j) = m = 0 \sum k_{h} - 1 n = 0 \sum k_{w} - 1 W_{m, n} \cdot X_{i + m, j + n}$

⚠️ Top pitfall: Forgetting the bias term or confusing valid padding (output shrinks) with same padding (output preserved for stride 1).

Self-check: A $32 \times 32$ input with a $5 \times 5$ kernel and stride 1 produces what output size with valid padding?

Connects to: Feature Maps, Output Size Formula, Parameter Counting

Feature Maps and Bias

Must-know: A feature map is the 2D output after convolving one kernel across the input plus a per-channel bias. The bias shifts activation values so borderline signals survive ReLU.

$[H]_{i, j, d} = a \sum b \sum c \sum [V]_{a, b, c, d} \cdot [X]_{i + a, j + b, c} + b_{d}$

⚠️ Top pitfall: Bias is one per channel, not per spatial position. A single scalar shifts the entire feature map.

Self-check: If a feature map after convolution has values $[- 5, - 3, + 8, - 2]$ , how does adding bias $b = 4$ change the signal after ReLU?

Connects to: ReLU Activation, Convolution Operation

ReLU Activation

Must-know: ReLU ( $max (0, x)$ ) is the default CNN activation. Its gradient is 1 for all positive inputs — no saturation. This avoids the vanishing gradient problem that plagued earlier sigmoid/tanh networks.

$ReLU (x) = max (0, x)$

⚠️ Top pitfall: Dying ReLU — a neuron that outputs zero for all inputs stops learning permanently. Use Leaky ReLU or proper learning rates to mitigate.

Self-check: Why did AlexNet's use of ReLU over sigmoid enable training of 8-layer CNNs on ImageNet?

Connects to: Vanishing Gradients, AlexNet, Normalization

Normalization (BatchNorm)

Must-know: Batch normalization rescales activations to zero mean and unit variance across the batch, then applies learnable scale $γ$ and shift $β$ . It prevents scale imbalance and enables higher learning rates.

$x^_{i} = \frac{x _{i} - μ _{batch}}{σ _{batch}^{2} + ϵ}, y_{i} = γ x^_{i} + β$

⚠️ Top pitfall: Batch norm uses batch statistics during training but running averages during inference. Forgetting to switch to eval mode is a common deployment bug.

Self-check: What happens to batch normalization when batch size is 1 or 2? Which alternative should you use?

Connects to: ReLU Activation, Parameter Counting

Pooling

Must-know: Pooling downsamples feature maps with zero learnable parameters. Max pooling preserves the strongest signal; average pooling smooths. Pooling never changes channel count.

$Max: y = max {H_{i, j}, H_{i, j + 1}, H_{i + 1, j}, H_{i + 1, j + 1}} Avg: y = \frac{1}{∣ R ∣} (p, q) \in R \sum H_{p, q}$

⚠️ Top pitfall: Pooling has zero parameters — it is a fixed operation. Confusing pooling stride with convolution stride is a common mistake.

Self-check: A $4 \times 4$ feature map with $2 \times 2$ max pooling, stride 2 produces what output size?

Connects to: CNN Flow, Padding, Stride vs Pooling

Padding and Output Size Formula

Must-know: The output size formula is the most frequently tested CNN equation. Given any three of $N, P, K, S, O$ , solve for the others. Same padding preserves output size (stride 1, odd kernel).

$O = ⌊ \frac{N + 2 P - K}{S} + 1 ⌋ P = \frac{K - 1}{2} (same padding, stride 1)$

⚠️ Top pitfall: Confusing total padding (both sides) with per-side padding. The formula uses per-side $P$ ; total added is $2 P$ .

Self-check: A $32 \times 32$ input becomes $28 \times 28$ with stride 1, valid padding. What was the kernel size?

Connects to: Convolution Operation, Receptive Field Growth

Parameter Counting

Must-know: Convolutional layer parameters = $(K \times K \times C_{in} + 1) \times C_{out}$ . Parameter sharing makes CNNs dramatically more efficient than fully connected layers.

$params = (K \times K \times C_{in} + 1) \times C_{out}$

⚠️ Top pitfall: Forgetting the $+ 1$ bias term or missing the $C_{in}$ multiplier. A $3 \times 3$ conv with 128 input and 256 output channels has $(9 \times 128 + 1) \times 256 \approx 295 K$ params.

Self-check: How many parameters in a conv layer with 64 input channels, 128 output channels, $3 \times 3$ kernels?

Connects to: CNN Architecture Depth/Width, Channels

Receptive Field Growth

Must-know: The receptive field grows by $(K_{i} - 1) \times \prod S_{j}$ per layer. Deeper layers see wider context without larger kernels. With all stride 1, each $3 \times 3$ layer adds 2 pixels to the field.

$RF_{i} = RF_{i - 1} + (K_{i} - 1) \times j = 1 \prod i - 1 S_{j}$

⚠️ Top pitfall: Forgetting the cumulative stride product. A stride of 2 in layer 1 doubles the field growth contribution of every subsequent layer.

Self-check: After 5 conv layers, all $3 \times 3$ , stride 1, what is the receptive field at layer 5?

Connects to: Dilated Convolution, CNN Architecture

Dilated (Atrous) Convolution

Must-know: Dilated convolution inserts zeros between kernel elements to expand the receptive field without adding parameters. Effective kernel size: $K_{eff} = K + (K - 1) (R - 1)$ .

$K_{eff} = K + (K - 1) (R - 1)$

⚠️ Top pitfall: Using the same dilation rate in every layer creates gridding artifacts. Vary rates (e.g., 1, 2, 3) to prevent checkerboard sampling patterns.

Self-check: A $3 \times 3$ kernel with dilation rate $R = 4$ has what effective receptive field size?

Connects to: Receptive Field Growth, Semantic Segmentation

Historical Architectures

Must-know: LeNet — first CNN, proved convolution works for digit recognition. AlexNet — introduced ReLU, won ImageNet 2012, launched deep learning era. InceptionNet — multi-scale parallel kernels. ResNet — skip connections enable 150+ layer networks.

$ResNet skip connection: output = F (x) + x$

⚠️ Top pitfall: Over-attributing breakthroughs to single innovations — AlexNet's success required ReLU, GPU training, and large datasets (ImageNet) simultaneously.

Self-check: What problem did skip connections (ResNet) solve that ReLU alone could not?

Connects to: ReLU, Vanishing Gradients, CNN Architecture

Previous LectureLecture 09 · Convolutional Neural Networks

Next LectureLecture 11 · CNN Architectures and Applications

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.