Deep Neural Networks

CNN Architectures and Applications

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Deep Learning and Computer Vision

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Convolution operation, kernels, padding, and pooling — covered in Lecture 10 (Convolutional Neural Networks). These are the raw building blocks every architecture in Lecture 11 stacks and modifies.
Channels and feature maps — covered in Lecture 10. Essential for understanding 1×1 convolutions, the channel-doubling in VGG blocks, and inception concatenation.
The output-size formula and parameter counting — covered in Lecture 10. This is the highest-weight exam skill used throughout Lecture 11 (LeNet-5, AlexNet, VGGNet parameter tables).
Receptive field growth and multi-scale feature capture — covered in Lecture 10. Directly motivates GoogLeNet's parallel multi-scale inception module.
Transposed convolution and applications beyond images — covered in Lecture 10. Needed for the segmentation / upsampling discussion and CNNs beyond vision.
Introduction to CNNs and historical architectures — covered in Lecture 9 and briefly in Lecture 10. Sets up why LeNet-5, AlexNet, and Inception were necessary steps.

CNN Architectures and Applications

This lecture takes you on a tour of the most important CNN architectures ever built. It starts in 1998 with LeNet-5 — the network that first proved machines could read handwritten digits. Then it moves to AlexNet in 2012. AlexNet showed the world that deep learning on GPUs could beat decades of hand-crafted computer vision. From there, you will see how each architecture solved a specific problem. VGGNet found power in simplicity. GoogleNet captured features at multiple scales. ResNet made extreme depth possible. And transfer learning made all of it reusable. By the end, you will understand not just what each architecture looks like. You will also see why each innovation was necessary and what problem it solved.

11.1 LeNet-5 — The Foundational CNN Architecture

How does a machine read a handwritten cheque? How does the post office sort letters by ZIP code? Before 1998, these tasks needed humans. LeNet-5 changed that — it was the first neural network that could look at a handwritten digit. It could reliably tell you what number it was.

Analogy: A postal worker scanning envelopes. Imagine a postal worker who looks at an envelope through a small magnifying glass. She slides the glass across the envelope one small patch at a time. At each patch, she looks for curves, loops, and straight lines. After scanning the whole envelope twice, she has a mental map of features. She scanned once with a coarse lens and once with a finer one. She then runs through her mental checklist: "Does this pattern match a 3? A 7?" LeNet-5 works the same way — small windows (kernels) slide across the image. They detect local patterns and pass the findings to a decision-making network. The magnifying-glass analogy breaks in one key way: the postal worker already knows what a curve looks like. The network has to learn what counts as a useful feature from scratch.

11.1.1 Definition and Explanation

LeNet-5 was introduced by Yann LeCun and his team at AT&T Bell Labs in 1998. It was the first CNN to successfully combine convolution layers with pooling layers for handwritten digit recognition. The "5" in LeNet-5 refers to the five trainable layers: two convolution layers and three fully connected layers. Pooling layers count as non-trainable (they have no learned weights), which is why the total is 5, not 7.

The architecture follows a simple alternating pattern: convolution, then pooling, repeated twice. Then two fully connected hidden layers. Then an output layer. The input is a 32×32 grayscale image of a handwritten digit or character. The total learnable parameters across the entire network is roughly 60,000.

LeNet-5 is a shallow network by modern standards. Its depth is limited. It uses sigmoid activation functions throughout to introduce nonlinearity. Later architectures replaced sigmoid with ReLU because sigmoid causes vanishing gradient problems in deeper networks.

The original LeNet-5 was trained on the MNIST dataset and deployed in ATMs to read cheque amounts. Some ATMs still run code that Yann LeCun and Leon Bottou wrote in the 1990s. Before LeNet-5, the best methods for digit recognition were support vector machines (SVMs). They were the dominant approach in supervised learning at the time. LeNet-5 matched SVM performance — achieving less than 1% error per digit — using learned features instead of hand-crafted ones.

Notation note: In the standard literature, LeNet-5 uses 28×28 input images (the original 32×32 MNIST scans were trimmed to save space). The first convolution then uses padding of 2 to keep the output at 28×28. The professor's version uses the untrimmed 32×32 input with no padding — both approaches produce the same convolution output size. The exam will follow the professor's setup.

11.1.2 Symbol Registry

Symbol	Meaning	Type / Domain
$H_{in}$	Input height/width	integer, pixels
$H_{out}$	Output height/width	integer, pixels
$K$	Kernel size (square kernel, $K \times K$ )	integer
$S$	Stride	integer, $\geq 1$
$P$	Padding	integer, $\geq 0$
$C_{in}$	Number of input channels	integer
$C_{out}$	Number of output channels (kernels)	integer
$b$	Bias term (one per output channel)	scalar

11.1.3 Architecture Walkthrough

Layer-by-layer breakdown with parameter counts:

Input: $32 \times 32 \times 1$ grayscale image. A single number at each pixel — the brightness from black (0) to white (1).

First convolution layer (Conv1): 6 kernels, each $5 \times 5$ . The kernel slides across the 32×32 image with stride $S = 1$ and no padding ( $P = 0$ ). Using the output size formula:

H_{out} = \frac{H _{in} - K + 2 P}{S} + 1 = \frac{32 - 5 + 0}{1} + 1 = 28

The output is $28 \times 28 \times 6$ . Each of the six 28×28 maps is a feature map — one per kernel. These six maps act as six separate channels for the next layer.

Parameters in Conv1: Each kernel has $5 \times 5 \times 1 = 25$ weights (the "×1" is because input has one channel). With 6 such kernels: $5 \times 5 \times 6 = 150$ weights. Plus one bias per kernel: 6 biases. Total = 156 learnable parameters.

First pooling layer (Pool1): Average pooling with a $2 \times 2$ window and stride 2. Each $28 \times 28$ feature map shrinks to $14 \times 14$ . Channels stay at 6. No weights are learned here — pooling does not have parameters.

Second convolution layer (Conv2): 16 kernels, each $5 \times 5$ , applied to the 6-channel input. With stride 1 and no padding:

H_{out} = \frac{14 - 5 + 0}{1} + 1 = 10

Output is $10 \times 10 \times 16$ .

Parameters in Conv2: Each kernel spans all 6 input channels: $5 \times 5 \times 6 = 150$ weights per kernel. With 16 kernels: $5 \times 5 \times 6 \times 16 = 2, 400$ weights. Plus 16 biases. Total = 2,416 learnable parameters.

Second pooling layer (Pool2): Average pooling, $2 \times 2$ , stride 2. Each $10 \times 10$ map becomes $5 \times 5$ . Channels remain 16. Zero learnable parameters.

Flatten: The 16 feature maps of $5 \times 5$ are flattened into a vector of $16 \times 5 \times 5 = 400$ values. These 400 numbers are the distinguishing features that the convolutional kernels have extracted from the original digit image.

First fully connected layer (FC1): 120 neurons. Every one of the 400 input features connects to every one of the 120 neurons. Parameters: $400 \times 120 = 48, 000$ weights + 120 biases = 48,120.

Second fully connected layer (FC2): 84 neurons. Parameters: $120 \times 84 = 10, 080$ weights + 84 biases = 10,164.

Output layer: 10 neurons (digit classes 0–9). Parameters: $84 \times 10 = 840$ weights + 10 biases = 850.

Total learnable parameters: $156 + 2, 416 + 48, 120 + 10, 164 + 850 = 61, 706$ (about 60,000).

11.1.4 Worked Example: Parameter Counting Exercise

Reverse-engineering kernel size from dimensions. You are told: an input of $32 \times 32$ passes through a convolution layer with stride 1 and no padding. The output is $28 \times 28$ . Find the kernel size.

Apply the formula:

H_{out} = \frac{H _{in} - K + 2 P}{S} + 1

Plug in the known values: $H_{out} = 28$ , $H_{in} = 32$ , $P = 0$ , $S = 1$ .

28 = \frac{32 - K + 0}{1} + 1 = 32 - K + 1 = 33 - K

K = 33 - 28 = 5

Answer: the kernel is $5 \times 5$ .

Sense-check: A 5×5 kernel placed at the top-left corner of a 32×32 image covers pixels 1–5. After sliding all the way to the right with stride 1, the kernel covers pixels 28–32 at the final position. That is exactly 28 positions — the math checks out.

The same logic applies to the second convolution layer. Input: $14 \times 14$ . Output: $10 \times 10$ . With $S = 1$ , $P = 0$ , you get $K = 5$ .

Exam tip: "Sometimes you might be given an architecture and you might be given the kernel size and the parameters. Then you need to pinpoint as to where exactly the error is. Or you might be given an architecture, you need to find the kernel size."

11.1.5 Student Questions and Answers

The parameter-counting discussion was teacher-led with interactive verification. For example:

"Can you tell me how many parameters are being learned? 6 parameters. What? 6 parameters. No, no. What are parameters here? The kernel weights are the parameters, is it not?"

This exchange highlights a common beginner trap. It confuses the number of output channels with the number of parameters. There are 6 feature maps. But there are 25 weights × 6 kernels = 150 weights, plus biases.

11.1.6 Assumptions and Scope

Scope: The convolution output size formula $H_{out} = \frac{H _{in} - K + 2 P}{S} + 1$ assumes:

Square input and square kernel (same height and width)
Stride is the same in both directions
Padding is symmetric

When padding is not symmetric, or when strides differ horizontally and vertically, compute height and width separately using the same formula.

Visual Intuition: The shrinking image. Picture a 32×32 grid on graph paper. You have a 5×5 transparent window. You place the window at the top-left corner — that is position 1. Slide it right one square at a time. After 28 slides, the window's right edge touches the grid's right edge — you cannot go further. Now start the next row, one square down. Do this 28 times vertically. The result: a 28×28 grid of window positions. Each position produces one number (one value in the feature map). The shrinking is real — every convolution without padding eats away the edges.

11.1.7 Pitfalls

Confusing feature maps with parameters. Six output channels (feature maps) after Conv1 does NOT mean six parameters. Each kernel has $5 \times 5 = 25$ weights. Six kernels = 150 weights. In Conv2, each kernel spans all 6 input channels. Each has $5 \times 5 \times 6 = 150$ weights — much more than just $5 \times 5$ .
Counting pooling parameters. Pooling layers have zero learnable parameters. They do not learn weights — they just aggregate (average or max). If you count pooling as a "layer with parameters" in an exam, you will get the total wrong.
Flattening does not change the number of values. The 16×5×5 tensor has 400 values before and after flattening. Flatten just reshapes — it does not add or remove information.
Forgetting the input channel dimension in Conv2. Suppose Conv2 gets 6-channel input. Each 5×5 kernel spans all 6 channels. So the weight count is $5 \times 5 \times 6$ . Not $5 \times 5$ . This is the single most common counting error.

11.1.8 Limitations of LeNet-5

LeNet-5 is a shallow network — despite appearing deep at first glance, it lacks the depth of later architectures. It handles simple image classification like handwritten digit recognition well. But it cannot handle very large or complex datasets (e.g., ImageNet with 1000 classes of natural images).

The use of sigmoid activation functions causes the vanishing gradient problem. At many points during backpropagation, the gradient becomes extremely small because sigmoid saturates at 0 and 1. Earlier layers learn very slowly or not at all. This is why nearly every architecture that followed switched to ReLU.

LeNet-5 paired convolutions with pooling for the first time in a way that actually worked at scale. It proved that a network could learn features automatically — without hand-engineering. Every CNN you use today traces its lineage back to this 1998 design. The next lecture covers AlexNet, which took this blueprint and added three game-changers: ReLU, dropout, and data augmentation.

Real-world connection: LeNet-5 was deployed in ATMs across the United States to read handwritten cheque amounts. The US Postal Service also used it to sort mail by reading handwritten ZIP codes. The core idea is simple: slide small learned filters across an image. This same mechanism drives every modern computer vision system. It powers face unlock on your phone and self-driving car perception systems. LeNet was the proof that machines could see.

11.1.9 Exam Notes

Exam note: Expect questions on counting parameters layer by layer in a given CNN architecture. You may also need to reverse-engineer kernel size from input and output dimensions. Use the formula $H_{out} = \frac{H _{in} - K + 2 P}{S} + 1$ . Or identify errors in architectural descriptions. For example: "Given this kernel size and these output dimensions, find the mistake." The Conv2 parameter-counting trap is a favorite exam pitfall. It involves forgetting the 6 input channels.

11.2 AlexNet — ReLU, Dropout, and Data Augmentation

In 2012, a neural network looked at 1.2 million photographs. It learned to tell apart 1,000 different kinds of objects — goldfish, dalmatians, espresso makers, volcanoes. It beat every other computer vision system by a huge margin. The winning margin was so large (top-5 error of 15.5% vs 26.2% for second place). It convinced the entire computer vision community to abandon hand-crafted features and switch to deep learning. That network was AlexNet.

Analogy: Learning to cook by tasting variations. Imagine learning to cook a dish not by following one recipe. Instead, you taste hundreds of slightly different versions — more salt here, less heat there, a different spice. Each variation teaches you something about what matters. AlexNet uses three tools that work the same way. ReLU simplifies what you learn so it does not get stuck. Dropout forces you to forget random ingredients each time. This prevents memorizing the recipe. Data augmentation creates variations of the same dish. You rotate the photo or change the lighting to learn what stays the same. The analogy breaks in that a chef would go mad with random memory loss. AlexNet needs it precisely to avoid becoming too specialized.

11.2.1 Definition and Explanation

AlexNet, developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, won the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC). It introduced three innovations that remain standard practice. (1) ReLU activation replaced sigmoid to fix vanishing gradients. (2) Dropout randomly shuts off neurons during training to prevent overfitting. (3) Data augmentation artificially expands the training set with transformed copies of images.

A fourth innovation was just as critical: GPU training. AlexNet ran on two NVIDIA GTX 580 GPUs, each with 3 GB of memory. Training took 5–6 days. Without GPUs, no one could have trained a network this large.

Compared to LeNet-5, AlexNet is much deeper. It has 5 convolution layers, 2 fully connected hidden layers (each with 4096 neurons), and a 1000-class output layer. The total is about 60 million parameters — about 1,000 times more than LeNet-5.

Notation note: The professor describes AlexNet as having 3 fully connected hidden layers. The standard architecture in the original paper uses 2 hidden FC layers (4096 → 4096) plus the 1000-class output layer. The professor counts the output layer as the third FC layer. Also, the professor uses 227×227 input; the standard implementation uses 224×224. Both are valid — the key point is that your input images must match the pre-trained model's expected size exactly.

11.2.2 Symbol Registry — AlexNet

Symbol	Meaning	Type / Domain
$p$	Dropout probability (fraction of neurons dropped)	scalar, $0 \leq p < 1$
$ReLU (x)$	Rectified Linear Unit: $max (0, x)$	scalar, $\geq 0$

11.2.3 Architecture Details

AlexNet layer-by-layer:

Input: $227 \times 227 \times 3$ RGB image (the professor's variant; standard is $224 \times 224$ )
Conv1: 96 kernels, $11 \times 11$ , stride 4, ReLU → MaxPool $3 \times 3$ , stride 2
Conv2: 256 kernels, $5 \times 5$ , padding 2, ReLU → MaxPool $3 \times 3$ , stride 2
Conv3: 384 kernels, $3 \times 3$ , padding 1, ReLU
Conv4: 384 kernels, $3 \times 3$ , padding 1, ReLU
Conv5: 256 kernels, $3 \times 3$ , padding 1, ReLU → MaxPool $3 \times 3$ , stride 2
Flatten → FC1: 4096 neurons + ReLU + Dropout (0.5)
FC2: 4096 neurons + ReLU + Dropout (0.5)
Output: 1000 neurons (softmax for ImageNet classes)

The convolution kernels shrink as the network goes deeper. The first layer uses $11 \times 11$ to capture large structures in the high-resolution input. Then $5 \times 5$ , then $3 \times 3$ everywhere else. The channel count grows from 3 (RGB) to 96, 256, and 384, so deeper layers can detect richer feature combinations.

The two fully connected layers (4096 neurons each) are enormous. Each alone contains about 26 million parameters when connecting from the flattened conv output. This is why AlexNet needs so much memory.

11.2.4 Data Augmentation

The problem: Deep networks like AlexNet have tens of millions of parameters. With too little data, they will memorize the training set instead of learning general patterns. But collecting more labeled images is expensive.

The solution: Take each training image and apply small, label-preserving transformations.

Data augmentation creates more training data by transforming existing images without changing what the image shows. For image data, common transformations include:

Rotation: spin the image a few degrees
Scaling: zoom in or out slightly
Translation: shift the image a few pixels
Horizontal flipping: mirror left-right
Color jittering: adjust brightness, contrast, saturation

The professor's example: take a handwritten "7" and rotate it slightly. The rotated version still shows a "7." By applying varied transformations to every training sample, you multiply the effective dataset size. The model sees more variety and becomes less capable of memorizing (overfitting).

Why augmentation works: The model learns that rotation, scaling, and small shifts do not change the label. This builds invariance — the model stops caring about irrelevant variations.

AlexNet used extensive augmentation: random crops, horizontal flips, and PCA-based color augmentation (adding multiples of the principal components of RGB pixel values).

11.2.5 Student Q&A — Data Augmentation Pitfalls

Q: If we mirror a "6," it becomes a "9." Does data augmentation cause issues with such digits?

A: Yes. Blind augmentation creates problems when the transformation changes the label. Class "6" mirrored becomes class "9" — a different answer. For digits, rotation may also be dangerous. In these cases, you pick augmentations that preserve identity: scaling, zooming, deformation. You always control which augmentations to apply. The goal is complementary data based on existing samples, not truly synthetic data that crosses class boundaries. A practical rule: if a human would hesitate about the label after a transformation, do not use that transformation for that dataset.

11.2.6 Dropout

Dropout is a regularization technique — a logical switch, not a new layer. When you set dropout = 0.5, the network randomly shuts off 50% of the neurons in that layer during each training iteration. Gradients do not flow through shut-off neurons. Those neurons do not update their weights during that step.

How it works step by step:

At the start of each training iteration, each neuron in a dropout layer flips a coin.
With probability $p$ (the dropout rate, e.g., 0.5), the neuron is temporarily disabled.
Forward pass: the disabled neurons output zero. They contribute nothing to the next layer.
Backward pass: no gradient flows to disabled neurons. Their weights stay unchanged.
Next iteration: repeat with a fresh random subset of disabled neurons.

Why it helps: The network cannot rely on any single neuron being present. Every neuron must learn to be useful in many different contexts. This is like forcing every member of a team to be able to perform every role. The team becomes stronger as a whole. It also prevents co-adaptation. This is where two neurons learn to work together on a very specific pattern. It is a sign of memorization, not generalization.

During testing: All neurons are active, but their outputs are scaled down by $(1 - p)$ to compensate. In practice, modern frameworks handle this scaling automatically.

Q: Do we change the weights for dropout, or introduce another feature like bias?

A: Dropout is not a set of weights or a new feature. It is a few lines of logic in the code. If a layer has 10 neurons and dropout is 0.5, the framework randomly picks 5. It shuts them off for that iteration. It does not delete the network or remove connections permanently. The gradient simply does not flow through the disabled neurons. In the next iteration, a different random set of 5 is disabled. In code, you call something like Dropout(p=0.2) and 20% of nodes in that layer switch off each step.

Q: Is dropout active during testing?

A: No. Dropout is only for training. During testing, all neurons are active and their outputs are scaled appropriately. The entire purpose of dropout is to prevent overfitting during training. It is not needed when you evaluate the finished model.

11.2.7 Comparison: LeNet-5 vs AlexNet

Feature	LeNet-5 (1998)	AlexNet (2012)
Layers	2 conv + 3 FC	5 conv + 2 FC + output
Activation	Sigmoid	ReLU
Regularization	None (weight decay only)	Dropout
Input size	32×32 grayscale	227×227 RGB
Output classes	10	1000
Parameters	~60,000	~60 million
Training hardware	CPU	2 GPUs
Data	MNIST (60K images)	ImageNet (1.2M images)

When to pick which: LeNet-5 is fine for small grayscale classification tasks. AlexNet-style depth and ReLU are needed when the dataset is large, images are color, and classes are many.

11.2.8 Why the Input Size Matters

The professor connects this to transfer learning. The input size is fixed. For AlexNet it is $227 \times 227$ as the professor teaches it. The architecture's layers are designed around this size. When you reuse a pre-trained model, resize your own images to match. Feed in a grayscale image where the model expects RGB, and it will fail. Feed in a $256 \times 256$ image when it expects $227 \times 227$ , and the dimensions will not line up. Knowing these architectural parameters is how you debug transfer learning failures.

11.2.9 Pitfalls

Dropout during testing. If you leave dropout on during evaluation, the network will behave randomly and accuracy will drop. Every framework has a .eval() mode or equivalent — use it.
Augmentation that changes the label. Flipping a "6" to a "9" creates a mislabeled example. Flipping a cat photo horizontally is fine — a flipped cat is still a cat. Always ask: "Does this transformation change what a human would call this object?"
ReLU and dead neurons. ReLU outputs zero for any negative input. If a neuron's weights drift so that it always receives negative input, its gradient is always zero. It never recovers. This is called the "dying ReLU" problem. Leaky ReLU and ELU are variants that fix this.
227 vs 224 input. The professor uses $227 \times 227$ because it produces clean integer dimensions through all pooling layers. The standard AlexNet uses $224 \times 224$ with slightly different layer dimensions. For exam purposes, use whichever your professor uses.

11.2.10 Visual Intuition

Picture the data flow: a $227 \times 227 \times 3$ image enters. After Conv1 ( $11 \times 11$ kernel, stride 4), it becomes $55 \times 55 \times 96$ . After the first max-pool ( $3 \times 3$ , stride 2), it is $27 \times 27 \times 96$ . After the full conv stack, you get $6 \times 6 \times 256$ — a tiny grid. But each cell now summarizes a large patch of the original image with 256 different learned feature detectors. This $6 \times 6 \times 256 = 9, 216$ -value tensor is then flattened. It is fed through the massive fully connected layers that make the final classification decision. The spatial dimensions shrink from 227 to 6; the channel depth grows from 3 to 256. This is the classic CNN pattern: trade spatial resolution for semantic richness.

AlexNet proved that deep CNNs trained on GPUs could crush hand-crafted computer vision pipelines. ReLU, dropout, and data augmentation made it possible. These three innovations are now standard in virtually every deep learning project. Next: VGGNet asked a simple question — what if we just made everything uniform?

Real-world connection: AlexNet's 2012 victory was the "ImageNet moment" — the event that convinced the world deep learning worked at scale. Before 2012, computer vision used hand-designed features (SIFT, HOG, SURF). After 2012, nearly every winning entry used deep CNNs. AlexNet's GPU training approach also launched NVIDIA's pivot from gaming hardware to AI hardware. The ReLU + dropout + data augmentation recipe is still the starting point for most image classification projects today.

11.2.11 Exam Notes

Exam note: Know what each of AlexNet's three innovations solves. ReLU fixes vanishing gradient. Dropout handles overfitting. Data augmentation solves not enough data. Understand the computational cost — 5 conv layers need significant GPU resources. Architecture comparison: AlexNet vs LeNet-5 is a likely question. The professor's dropout analogy ("shut down your brain for a few minutes so you don't over-learn") is a memorable exam cue.

11.3 VGGNet — Depth Through Uniformity

What if the secret to better vision is not fancier filters, but simply stacking more of the same simple ones?

VGGNet answered this question in 2014. It showed that depth and uniformity together beat complexity. Instead of mixing kernel sizes like AlexNet, VGGNet used only 3×3 convolutions everywhere. Every layer looked the same. The only thing that changed was depth.

11.3.1 Intuition Behind Uniform Blocks

Imagine a factory assembly line. Every station does the same small operation — tighten one bolt. The first station tightens a little. The second tightens further. After many stations, the bolt is fully secure. No single station does anything fancy. But together they produce a strong result.

VGGNet works the same way. Each 3×3 convolution is a small step. Stacking many small steps builds a powerful feature-extraction network. Two stacked 3×3 convolutions cover the same area as one 5×5 filter. But two 3×3 layers use fewer parameters and add more non-linearity. More ReLU layers mean more learning capacity.

The key design pattern is the VGG block: several 3×3 convolutions followed by one 2×2 max-pooling layer. Convolutions preserve spatial size. Pooling halves it. This block repeats five times. After each block, the number of channels doubles. The spatial dimensions shrink: 224 → 112 → 56 → 28 → 14 → 7.

VGG Block: A sequence of 3×3 conv layers. Each has stride 1, padding 1, and ReLU. They preserve resolution. After them comes a 2×2 max-pooling layer. It has stride 2. It halves height and width. The block doubles output channels each time. This uniform pattern repeats across the entire network.

The core insight: multiple small convolutions between downsampling steps let the network grow deeper without immediately collapsing spatial dimensions.

11.3.2 Architecture Details

All convolutional layers use:

Kernel size: 3×3
Stride: 1
Padding: 1 (same padding — output size equals input size)

All pooling layers use:

Kernel size: 2×2
Stride: 2
Type: max pooling

The standard VGG block configuration for the family:

Block	Conv Layers	Output Channels	Output Size
1	1–2	64	112 × 112
2	1–2	128	56 × 56
3	2–3	256	28 × 28
4	2–3	512	14 × 14
5	2–3	512	7 × 7

After the convolutional blocks, the network flattens the 7×7×512 feature maps. It passes them through three fully connected layers: FC-4096 → FC-4096 → FC-1000 (with ReLU and dropout).

Two popular variants exist:

VGG-16: 13 convolution layers + 3 FC layers = 16 weight layers
VGG-19: 16 convolution layers + 3 FC layers = 19 weight layers

Worked Example: Receptive Field of Stacked 3×3 Convolutions

Question: Why use two 3×3 convolutions instead of one 5×5?

Consider a single pixel in the output of two consecutive 3×3 layers.

First 3×3 layer: the pixel sees a 3×3 patch in the input.
Second 3×3 layer: each of those 9 pixels sees a 3×3 patch.
Combined effect: the final pixel sees a 5×5 region in the original input.

So two stacked 3×3 convolutions have the same receptive field as one 5×5 convolution.

Now compare parameters (for C input and C output channels):

One 5×5 conv: $5 \times 5 \times C \times C = 25 C^{2}$ parameters
Two 3×3 convs: $2 \times (3 \times 3 \times C \times C) = 18 C^{2}$ parameters

Three stacked 3×3 convs: $3 \times 9 C^{2} = 27 C^{2} \approx 25 C^{2}$ (same receptive field as 7×7).

The stacked 3×3 approach saves parameters AND adds more ReLU non-linearities. More non-linearities = more expressive power.

11.3.3 Assumptions and Scope

When VGGNet assumptions hold:

Input images are reasonably sized (224×224 for ImageNet).
You have enough GPU memory — VGGNet is memory-hungry.
The dataset is large enough to prevent overfitting on ~138M parameters.
You want a simple, uniform design without architectural tricks.

When they break:

Small datasets — the massive parameter count causes severe overfitting.
Resource-constrained environments — VGG-16 needs ~528 MB just for forward pass weights.
Tasks needing very deep networks — vanishing gradients still appear despite ReLU.

11.3.4 Visual Intuition

Picture a stack of identically sized window frames. Each frame looks at the image through the same 3×3 lens. The first few frames detect simple edges and corners. Deeper frames combine edges into textures. Even deeper frames assemble textures into object parts. The final frames recognize whole objects. The pooling layers between blocks act like stepping back — you see a wider view but at lower resolution. This steady, uniform processing is VGGNet's signature.

11.3.5 Comparison: VGGNet vs AlexNet

Feature	AlexNet	VGGNet
Kernel sizes	11×11, 5×5, 3×3	Only 3×3
Stride/Padding	Varied	Always 1/1
Design pattern	Individual layers	Repeated blocks
Depth	8 layers (5 conv + 3 FC)	16–19 layers (13–16 conv + 3 FC)
Parameters	~60 million	~138 million (VGG-16)
GPU memory	~1.2 GB (training)	~8 GB+ (training)
Activation	ReLU	ReLU
Regularization	Dropout + aug	Dropout + aug
Key innovation	Depth works	Uniform depth scales better
Family of models	No	Yes (VGG-11/13/16/19)

11.3.6 Common Pitfalls

Pitfall 1: Ignoring memory cost. VGGNet's three FC layers alone contain ~120M parameters. The FC-4096 layer after flattening connects 25,088 inputs to 4,096 outputs — that is 102M parameters in a single layer. Most of VGG's parameters live in the FC layers, not the convolutions.

Pitfall 2: Training without enough data. With ~138M parameters, VGG-16 needs large datasets. Without enough data, it memorizes instead of generalizes. Always use data augmentation.

Pitfall 3: Expecting depth alone to solve everything. Even with ReLU, gradients can vanish across 16–19 layers during backpropagation. Later architectures (ResNet) explicitly address this with skip connections.

Pitfall 4: Confusing VGG variants. VGG-16 = 13 conv + 3 FC. VGG-19 = 16 conv + 3 FC. The number always refers to total weight layers. Know which variant is being discussed.

Recap: VGGNet proved that uniform 3×3 convolutions, stacked deep in repeating blocks, outperform complex mixed-filter designs. Its simplicity made it a template for modern CNNs. But its massive parameter count (~138M) and depth-induced vanishing gradients motivated the next breakthrough.

Bridge to GoogLeNet: VGGNet showed depth is good but expensive. The next architecture asks: can we go deeper while using fewer parameters? GoogLeNet answers with inception modules — multiple filter sizes running in parallel within the same layer.

Exam note: Be ready to identify VGGNet from description alone. Look for a specific pattern. All 3×3 filters, stride 1, padding 1, 2×2 max pool with stride 2, 13 conv, 3 FC layers. Comparison questions with AlexNet are common. Know why two 3×3 convs replace one 5×5. They have the same receptive field, fewer parameters, and more non-linearity. The block-based design is VGG's lasting contribution to CNN architecture.

11.3.7 Real-World Relevance

VGGNet is rarely trained from scratch today. But its pre-trained weights remain popular for transfer learning. The uniform 3×3 filter pattern became a standard in later networks. VGGNet's block-based design philosophy — compose networks from repeating building blocks — shaped every modern architecture from ResNet to EfficientNet. Its simplicity also makes it a favorite for teaching CNN design. When you first learn to build a deep CNN, you essentially build a VGG-like network.

11.4 Network in Network — 1×1 Convolution and Global Average Pooling

What if a single convolution could learn across channels instead of across space? What if you never needed a fully connected layer again?

VGGNet showed depth works, but its three FC layers consumed ~120M parameters alone. The Network in Network paper (Lin et al., 2013) attacked both problems at once. First, add cheap non-linearity per pixel using 1×1 convolutions. Second, replace all FC layers with a simple averaging operation. Two ideas. One paper. Modern CNNs never looked back.

11.4.1 Intuition — What 1×1 Convolution Actually Does

Imagine you have three paint channels — red, green, and blue — at every pixel of an image. A standard 3×3 convolution looks at neighboring pixels across all three channels. A 1×1 convolution looks at exactly one pixel but across all three channels. It mixes the red, green, and blue values at that pixel into a single new value. No spatial neighbors involved. Just channel mixing.

Think of it like blending three audio tracks (bass, mid, treble) into a single mono track at every time step. Or like compounding spices into one flavor. You combine ingredients from different jars into a single sauce, one spoonful at a time.

The formula is a weighted sum across channels:

output (i, j) = w_{1} \cdot ch_{1} (i, j) + w_{2} \cdot ch_{2} (i, j) + \dots + w_{C} \cdot ch_{C} (i, j)

This is cross-channel linear combination at a single spatial position. It does not slide across space. It goes deep, not wide.

1×1 Convolution: A convolution with kernel size 1×1 that operates only along the channel dimension. It has no spatial extent. It computes a learned linear combination of all input channel values at position $(i, j)$ to produce one output value. When you use $C^{'}$ such 1×1 kernels, you produce $C^{'}$ output channels. This is effectively a small per-pixel fully connected layer applied independently at every spatial location.

output_{c^{'}} (i, j) = k = 1 \sum C w_{c^{'}, k} \cdot ch_{k} (i, j)

where $C$ is the number of input channels, $C^{'}$ is the number of output channels, and $w_{c^{'}, k}$ is the learned weight connecting input channel $k$ to output channel $c^{'}$ .

11.4.2 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$w_{k}$	Weight for input channel $k$	$w_{k}$	scalar, learned
$C$	Number of input channels	—	integer
$C^{'}$	Number of output channels	—	integer
$ch_{k} (i, j)$	Value at spatial position $(i, j)$ in channel $k$	—	scalar
$H \times W$	Height and width of the feature map	—	integer

11.4.3 Worked Example — 1×1 Convolution with 2 Channels

The professor walks through a concrete numerical example during the lecture. The displayed matrices are shown below, followed by the computation. We present the professor's oral walkthrough alongside the corrected matrix-based values, since the live delivery contained two numerical interchanges. What matters is the mechanism — element-wise cross-channel weighting — not the exact numbers.

Input X: Two channels, each 2×2.

Channel 1:

[1021] Channel 2: [0121]

Kernel (depth-wise weights): $w_{1} = 1$ , $w_{2} = 2$

Professor's Oral Walkthrough (as recorded live):

Position	ch₁	ch₂	Computation	Result
(1,1)	1	0	1×1 + 2×0	1
(1,2)	0	2	1×0 + 2×2	4
(2,1)	2	1	1×2 + 2×1	4
(2,2)	1	1	1×1 + 2×1	3

Corrected Version (using displayed matrix values):

The matrix above shows ch₁(1,2) = 2 (not 0) and ch₁(2,1) = 0 (not 2).

Position	ch₁	ch₂	Computation	Result
(1,1)	1	0	1×1 + 2×0	1
(1,2)	2	2	1×2 + 2×2	6
(2,1)	0	1	1×0 + 2×1	2
(2,2)	1	1	1×1 + 2×1	3

Result: A single-channel 2×2 output. The number of channels reduced from 2 to 1. If you need $C^{'}$ output channels, use $C^{'}$ separate 1×1 kernels — each kernel produces one channel. This can also upsample channels back to the original count.

The key insight: after 1×1 convolution, apply ReLU to introduce non-linearity. Then the expensive spatial kernels operate on a reduced-channel, non-linearly transformed representation.

11.4.4 Assumptions and Scope

When 1×1 convolution helps:

You have many input channels (256, 512, 1024) and need to compress them before an expensive spatial convolution.
You want to add non-linearity per pixel without changing spatial dimensions.
You need a bottleneck that reduces parameters while preserving representational capacity.
Your architecture already has spatial feature-extraction layers (3×3 or 5×5 convs) — the 1×1 conv complements, not replaces them.

When 1×1 convolution does not help:

Single-channel inputs — there is nothing to combine across channels. A 1×1 conv on a grayscale image is just scalar multiplication.
Tasks requiring purely spatial transformations — 1×1 has zero spatial extent. Edge detection, for example, still needs 3×3 or larger kernels.
Very shallow networks with few channels — the compression benefit is negligible.

11.4.5 Visual Intuition — The Per-Pixel Micro Network

Picture a grid of 224×224 pixels. At each pixel location, you have a small stack of values — your channel dimension. A 1×1 convolution is like placing a tiny neuron at every pixel. This neuron receives all channel values at that pixel, multiplies each by a learned weight, and outputs one number. Then ReLU fires. Then you move to the next pixel.

Now imagine $C^{'}$ such neurons at every pixel — each with its own set of weights. That is what $C^{'}$ output channels mean. Every pixel gets its own miniature fully connected layer, applied identically across the entire spatial grid. The "network in network" name comes from exactly this: a small neural network (one layer of neurons) embedded at each spatial position.

This per-pixel processing adds non-linearity without touching spatial structure. The spatial convolutions can then work on richer, non-linearly combined features.

11.4.6 Pitfalls — What 1×1 Cannot Do

Pitfall 1: Expecting spatial feature extraction. A 1×1 kernel has zero spatial context. It cannot detect edges, textures, or shapes. It only combines existing channel information at the same pixel.

Pitfall 2: Over-compressing channels. If you reduce 1024 channels to 8 with a 1×1 conv, you lose too much information. The compressed representation cannot carry enough signal for the subsequent spatial convolution to work effectively. There is a real accuracy–efficiency trade-off.

Pitfall 3: Forgetting the ReLU. A 1×1 convolution without ReLU is just a linear projection. With ReLU, it becomes a non-linear feature transformer. The non-linearity is what gives it expressive power. Always apply ReLU after 1×1 conv.

Pitfall 4: Using 1×1 as the only convolutional layer. 1×1 convs supplement spatial convolutions — they do not replace them. A network made entirely of 1×1 convolutions has no spatial reasoning ability.

11.4.7 Parameter Savings — The Bottleneck Pattern

A concrete before-and-after comparison explains why 1×1 convolutions are so effective:

Scenario: Input has 256 channels. You want to apply 5×5 spatial convolutions and produce 256 output channels. No compression.

Parameters without 1\times1 = 5 \times 5 \times 256 \times 256 = 1, 638, 400

With 1×1 bottleneck: First compress 256 → 64 channels using a 1×1 convolution. Then apply 5×5 convolution on 64 channels to produce the desired output.

1\times1 bottleneck (256 \to 64) 5\times5 conv (64 ch \to 256 ch) Total = 1 \times 1 \times 256 \times 64 = 16, 384 = 5 \times 5 \times 64 \times 256 = 409, 600 = 16, 384 + 409, 600 = 425, 984

Reduction: $1, 638, 400/425, 984 \approx 3.85 \times$ fewer parameters.

You can also use a second 1×1 conv to upsample. It restores channels to the desired dimensions. This is a full bottleneck block: 256→64→spatial conv→256. The savings are still substantial. This gives roughly 11× reduction compared to naively increasing channels without any compression.

The pattern: compress → process spatially → expand. Cheap channel operations bookend the expensive spatial operation. This is the bottleneck design used in ResNet-50, ResNet-101, and beyond.

11.4.8 Key Applications of 1×1 Convolution

1×1 convolutions appear in nearly every modern architecture:

GoogLeNet / Inception: Each Inception block uses 1×1 convolutions before 3×3 and 5×5 branches to reduce channels. Without this, the multi-branch design would be computationally prohibitive.
ResNet Bottleneck (ResNet-50/101/152): The bottleneck block follows exactly the compress→process→expand pattern: 1×1 (reduce) → 3×3 (process) → 1×1 (expand). This makes 152-layer networks tractable.
MobileNet: Depthwise separable convolutions use 1×1 (pointwise) convolutions as the channel-mixing step after a depthwise spatial convolution. This is the core operation enabling mobile deployment.
Squeeze-and-Excitation Networks: 1×1 convolutions compress global channel statistics into attention weights that re-weight feature maps.
Semantic Segmentation (FCN, U-Net): 1×1 convolutions map feature channels to class scores at every pixel, replacing FC layers entirely.

11.4.9 Recap + Bridge for 1×1 Convolution

Recap: A 1×1 convolution performs a learned linear combination across channels at each spatial position. It has zero spatial extent — it only mixes channels. When followed by ReLU, it becomes a per-pixel non-linear feature transformer. Its primary role is channel compression before expensive spatial convolutions, reducing parameters by 3–11× with minimal accuracy loss.

Bridge to Global Average Pooling: 1×1 convolutions fix the first problem — adding cheap non-linearity earlier in the network. But VGGNet's second problem remains: those massive fully connected layers at the end. GAP solves that by simply averaging everything away.

11.4.10 Global Average Pooling (GAP) — Intuition

What if, instead of learning millions of weights to connect features to classes, you just took the average of every feature map?

That is Global Average Pooling. After the final convolutional layer, you have $N$ feature maps — each a 2D grid of activations. Instead of flattening them into a long vector and feeding them through expensive FC layers, GAP takes a different approach. It replaces each entire feature map with a single number: its average. An $N$ -channel feature map collapses to a vector of length $N$ .

Each feature map already encodes the presence of a specific learned pattern across the image. Its average tells you how strongly that pattern appears overall. If feature map 23 detects cat ears, a high average means the image likely contains a cat. No FC layer needed to re-learn that association.

11.4.11 GAP — Formal Definition and Worked Example

Global Average Pooling: For a feature map $F_{c}$ of spatial size $H \times W$ in channel $c$ , GAP computes:

GAP (F_{c}) = \frac{1}{H \cdot W} i = 1 \sum H j = 1 \sum W F_{c} (i, j)

The result is a single scalar per channel. For $N$ feature maps, GAP produces a vector of length $N$ . This vector feeds directly into the output layer (softmax for classification, sigmoid for binary). No learnable parameters. No overfitting from giant weight matrices.

Single feature map averaging:

Feature map values (2×2):

[1443]

Average = $(1 + 4 + 4 + 3) /4 = 3$ . The entire 2×2 map becomes one scalar: 3.

Multi-channel GAP:

Feature Map	Values	Average
Map 1	[1, 4, 4, 3]	2
Map 2	[4, 15, 20, 21]	15
Map 3	[3, 5, 6, 6]	5

Output vector: $[2, 15, 5]$ — a 3-element vector passes to the final classification layer. No FC layers between GAP and the output.

11.4.12 GAP Limitations and Pitfalls

Limitation 1: Loss of spatial precision. GAP averages over all spatial positions. It loses information about where a feature appears. For object localization or detection — tasks that need bounding boxes — GAP alone is not enough. The professor notes: "If you have a cluttered environment with very fine grained objects, using GAP would be bad. It just averages out."

Limitation 2: Requires rich pre-GAP features. GAP works only because the preceding convolution layers have already learned to produce meaningful feature maps. Without enough convolutional depth before GAP, the averages carry too little information for classification.

Limitation 3: No learnable parameters means no adaptive weighting. An FC layer can learn which features matter most. But GAP treats every spatial position equally. If some regions are more important than others (e.g., center of the image), GAP cannot learn that.

Limitation 4: Not suitable for variable-sized inputs. Some tasks need fixed-size feature vectors for reasoning. GAP handles variable spatial sizes naturally — the average is always one number. But fine-grained spatial reasoning needs more than averaging.

11.4.13 Student Q&A

Q: How does backpropagation work through GAP if no learning happens there?

A: GAP has no learnable weights, but gradients must still flow through it to earlier layers. The gradient of an average is the average of gradients distributed equally to every position. Think of water flowing through an open pipe — nothing blocks it. GAP does not learn anything. It is just a pooling operation. Gradients pass through freely to the convolution layer before it.

Q: What exactly is a "feature map"?

A: It is the output after convolution and any intermediate operations (ReLU, pooling). We cannot simply call it "features" because the raw convolution output is a compressed transformation of the input. The term "feature map" describes a 2D grid of activations. Each spatial position encodes how strongly a learned pattern is present at that location.

11.4.14 Architectural Shift — Replacing FC Layers

In modern CNN design, the standard pattern has shifted:

Old pattern (AlexNet, VGG):
Conv blocks → Flatten → FC-4096 → FC-4096 → FC-1000 (Softmax)

New pattern (NiN, GoogLeNet, ResNet):
Conv blocks (with 1×1 convolutions) → GAP → FC-1000 (Softmax)

Sometimes a single small FC layer sits between GAP and the output (e.g., a 1024-dim FC for more complex tasks). But the era of multi-layer FC heads with millions of parameters is over. GAP handles the dimensionality collapse. The final classification layer handles the decision.

Parameter comparison for the transition from features to classes:

VGG-16 FC layers: ~120M parameters
NiN with GAP: ~0 parameters (before the final softmax layer)

11.4.15 Exam Guidance

Exam note: 1×1 Convolution

Be able to compute 1×1 convolution by hand on a small multi-channel input (like the 2×2×2 example above).
Know the formula: $output (i, j) = \sum_{k} w_{k} \cdot ch_{k} (i, j)$
Calculate parameter savings: $5 \times 5 \times 256 \times 256$ vs. the bottleneck alternative.
Understand that 1×1 conv has no spatial extent — it only mixes channels.

Exam note: Global Average Pooling

Define GAP: average all spatial values in each feature map to a scalar. No learnable parameters.
Explain why GAP reduces overfitting (no giant FC weight matrices to memorize training data).
Know when GAP is inappropriate (object localization, fine-grained spatial tasks).
Compare parameter counts with and without GAP.

Common exam question: Why does NiN use both 1×1 convolutions and GAP? Answer: 1×1 convs add cheap non-linearity per pixel early in the network. GAP eliminates the giant FC layers at the end. Together they make deep CNNs practical with far fewer parameters.

11.4.16 Real-World and Domain Connection

1×1 convolutions and GAP are not just academic novelties. They power the models running on your phone:

On-device image classification (MobileNet): 1×1 (pointwise) convolutions are the dominant operation. Without them, real-time classification on a smartphone CPU would be impossible.
Self-driving cars (semantic segmentation): The final layer of a segmentation network uses a 1×1 convolution. It maps feature channels to class scores per pixel. GAP is replaced by per-pixel classification — the 1×1 conv handles that mapping.
Cloud-scale image search: Billion-parameter vision models use bottleneck blocks (1×1→3×3→1×1) to keep inference costs manageable at scale.
Medical image analysis: When training data is scarce (common in healthcare), every parameter counts. GAP's zero-parameter design prevents overfitting where FC layers would memorize the small training set.

The principle is universal. Whenever you see a deep CNN, look for 1×1 convolutions in the bottlenecks. Look for GAP in the head. This applies to vision, video, and audio spectrograms. These two ideas from 2013 quietly reshaped how we build neural networks.

11.5 GoogleNet / InceptionNet — Multi-Scale Feature Extraction

What if, instead of guessing the right filter size for each layer, you just used them all at the same time? Then you let the network decide. GoogleNet asked this question and won the 2014 ImageNet competition with 12× fewer parameters than AlexNet.

Analogy: A team of detectives with different magnifying glasses. Imagine three detectives examining a crime scene photo. One uses a wide-angle lens (5×5) to see the big picture — furniture layout, room shape. Another uses a medium lens (3×3) for mid-sized clues — objects on a desk. The third uses a magnifying glass (1×1) for fine detail — a fingerprint. They all look at the same photo simultaneously. Then they pool their findings. The inception module works exactly like this — multiple filter sizes run in parallel on the same input. The results are concatenated. The analogy breaks in that detectives communicate with words; the network just stacks numbers side by side in the channel dimension.

11.5.1 Definition and Explanation

GoogleNet (also called InceptionNet), developed by Google, won the 2014 ILSVRC competition. It introduced the inception module — a block that processes the same input through multiple parallel convolutional paths with different kernel sizes. It then concatenates all results along the channel dimension.

The central insight: an image may contain features at multiple scales. A fine-grained detail (the texture of fur) might need a small receptive field — a 3×3 kernel would detect it. A larger structural pattern needs a bigger receptive field. An example is the overall shape of an animal. A 5×5 kernel would detect that. Rather than guessing which kernel size is right for a given layer, run them all in parallel and combine the outputs. The network learns which path matters most for which features.

GoogleNet learned roughly 5 million parameters — about 12× fewer than AlexNet's 60 million — while achieving better accuracy. The 1×1 convolutions within the inception blocks are responsible for most of this parameter efficiency.

11.5.2 The Inception Module

At any point in the network, the current feature maps are passed through four parallel branches:

A 1×1 convolution — captures fine, per-pixel channel interactions
A 3×3 convolution — captures mid-sized spatial patterns
A 5×5 convolution — captures larger spatial patterns
A 3×3 max pooling followed by a 1×1 convolution — provides a summary-downsample path

The outputs of all four branches are concatenated along the channel dimension. This combined tensor is passed to the next layer.

The key efficiency trick: use 1×1 convolutions as bottlenecks. Place them before the expensive 3×3 and 5×5 convolutions. For example, suppose the input has 256 channels. Applying a 5×5 convolution directly costs $5 \times 5 \times 256 \times C_{out}$ parameters. Instead, a 1×1 convolution reduces channels first. Say to 64. Then the 5×5 convolution operates on just 64 channels. This cuts parameters by roughly a factor of 4.

11.5.3 GoogleNet Architecture Overview

The full architecture:

Stem: 7×7 convolution → 3×3 max pool → 3×3 convolution → 3×3 max pool
Body: 9 inception blocks stacked sequentially
Head: Global Average Pooling → fully connected output layer (1000 classes)

The architecture is deep enough that the full diagram does not fit on a single slide. GoogleNet also replaced the heavy fully connected layers at the end with Global Average Pooling, saving millions of parameters.

11.5.4 Intermediary (Auxiliary) Classifiers

GoogleNet introduced an unusual design: auxiliary classifiers inserted at intermediate depths of the network.

The idea: after a few inception blocks, the network may have already captured features useful for broad classification. For example, it might distinguish animals from plants. A small classifier is plugged in at this intermediate point. It makes a coarse prediction. The main network continues processing toward finer distinctions — say, which specific species of animal.

The professor's description: "Probably they thought that at this point itself, if we can somehow get the data out, plug into a classifier … probably this has an ability to classify it into animals and plants. Then let the upstream tasks proceed further."

Multiple auxiliary classifiers at different depths effectively perform different classification tasks. One identifies broad categories early. Deeper classifiers handle finer distinctions. This is like multi-label classification happening at different depths.

Student Q&A:

Q: After every convolution they get their classifications. How do they consolidate these classifiers in the final state? Does the final classification rely on the final one only?

A: Multiple classifications happen — not just one. One classifier identifies Class A from Class B. Another identifies different species of animals. They are different classification tasks being solved at different depths. It is like multi-label classification — one completes early (broad category), another completes late (fine distinction). The auxiliary classifiers are mainly used during training to inject more gradient signal. At inference time, only the final classifier is typically used.

11.5.5 Comparison: GoogleNet vs AlexNet

Feature	AlexNet (2012)	GoogleNet (2014)
Parameters	~60 million	~5 million (12× fewer)
Convolution layers	5	21 (spread across 9 inception blocks)
FC layers	2 × 4096	None (uses GAP + 1 FC output)
Key innovation	ReLU, Dropout, Augmentation	Inception module, 1×1 bottlenecks
Kernel strategy	Decreasing sizes (11→5→3)	All sizes in parallel at each block
ILSVRC top-5 error	15.5%	6.7%

When to pick which: GoogleNet is more parameter-efficient and accurate. AlexNet is simpler to understand and implement. Both are now mainly of historical and educational interest — modern architectures like ResNet surpass both.

11.5.6 Pitfalls

Extreme computational cost. Each path in an inception block requires its own convolution operations. Running 4 parallel paths per block × 9 blocks = massive compute. Even one iteration through an inception block requires millions of operations per path.
Diminishing returns from 5×5. A 5×5 convolution costs $25/9 \approx 2.8 \times$ more than a 3×3. Later Inception variants (v2, v3) replaced 5×5 with two stacked 3×3 convolutions, which cover the same receptive field at lower cost.
Concatenation grows channels. Every inception block concatenates outputs from multiple paths. The channel count balloons as you go deeper. This is why 1×1 bottlenecks are essential — without them, the parameter explosion would be uncontrollable.
Auxiliary classifiers are mainly for training. Do not assume they are used at test time. Their primary job is to inject more gradient signal during backpropagation, helping earlier layers learn.

11.5.7 Visual Intuition

Imagine the data flow: a feature map enters an inception block. It splits into four separate streams. Stream 1 passes through a 1×1 filter — lightweight, channel-only mixing. Stream 2 goes 1×1 (compress channels) → 3×3 (spatial patterns). Stream 3 goes 1×1 → 5×5 (larger patterns). Stream 4 goes 3×3 max pool (spatial summary) → 1×1 (channel adjust). All four streams produce output maps of the same height and width. These are stacked side by side along the channel axis — like stacking four different-colored transparent sheets together. The next block sees all four perspectives simultaneously.

GoogleNet proved you do not need to choose one filter size per layer. Run them all in parallel, concatenate, and let the network learn which features matter. The 1×1 bottleneck trick made this affordable. Next: ResNet asked an even more fundamental question — what if deeper networks were actually worse than shallower ones?

Real-world connection: The inception module's "run everything in parallel and combine" philosophy influenced modern neural architecture search (NAS). Automated systems try hundreds of parallel branch combinations. GoogleNet itself evolved through Inception v2, v3, and v4, each refining the block design. The 1×1 bottleneck concept is now universal — you will find it in ResNet, MobileNet, EfficientNet, and nearly every efficient CNN.

11.5.8 Exam Notes

Exam note: Understand the purpose of the inception module — parallel kernels of different sizes for multi-scale feature extraction. Know that 1×1 convolutions act as dimension-reducing bottlenecks inside inception blocks. The auxiliary classifier concept may appear as a conceptual question about multi-label classification at different depths. GoogleNet vs AlexNet parameter comparison (5M vs 60M) is a likely exam point.

11.6 Residual Connections — Solving the Deep Network Problem

In 2015, researchers discovered something that made no sense. A 56-layer network was doing worse than a 20-layer network — even on the training data. It was not overfitting. Information was simply getting lost as it flowed through too many layers. The solution — add a shortcut that lets the original input bypass the processing and rejoin later — changed deep learning forever.

Analogy: The professor's refresher-slide metaphor. "By the time you have come to your tenth contact session for deep learning, some of us would have forgotten what a ReLU is." So a refresher slide reminds you. The same thing happens to deep networks — by layer 50, the original pixel information is so heavily transformed. The network has forgotten what it started with. A residual connection is like inserting a photocopy of the input at every block: "Here — in case you forgot, this is what you started with."

Analogy 2: The express highway. Picture a city with only local roads. To cross town, you stop at every intersection. In a deep network, that is every layer. A skip connection is an express highway. You bypass dozens of intersections and go straight from the input to a much deeper layer. During backpropagation, gradients use this highway to flow directly back without fading. This is the gradient highway — the single most important reason residual connections work.

11.6.1 Symbol Registry

Symbol	Meaning	Type / Domain
$x$	Input to a residual block	tensor (any shape)
$F (x)$	Transformation learned by the block's layers	tensor, same shape as $x$
$Output$	Residual block output = $F (x) + x$	tensor, same shape as $x$

11.6.2 Definition and Explanation

A residual connection is also called a skip connection. It takes a copy of the input to a block. It adds that copy directly to the block's output. The block's layers learn only the residual — the difference between the desired output and the input. The formula:

Output = F (x) + x

where $F (x)$ is whatever transformation the block's layers apply (convolutions, batch normalization, ReLU, etc.).

ResNet, introduced by Kaiming He and colleagues at Microsoft Research in 2015, addressed a counterintuitive observation. Making networks deeper was making them perform worse, even on training data. This is not overfitting (where training error stays low but test error rises). This is degradation — training error itself increases with depth.

The problem: in a very deep plain network, information gets distorted as it passes through dozens of weight matrices. By the time it reaches the later layers, the original signal is unrecognizable. Gradients face the same problem in reverse — they vanish before reaching early layers.

11.6.3 How Residual Connections Work

Step-by-step through a residual block:

Input $x$ enters the block.
$x$ is copied (the skip connection or identity shortcut).
$x$ also passes through the block's layers: conv → BN → ReLU → conv → BN.
The block's output is $F (x)$ — the transformed version.
The final output is $F (x) + x$ (element-wise addition of the copy and the transformation).
A ReLU is applied to the sum: $ReLU (F (x) + x)$ .

The professor's plain-language description: "You take a copy of an input and add it here. It is like feature one, feature two, feature 10 … 10 feature maps are getting updated, but along with it, you augment one more information as a feature map — it is basically the original input. So whatever was lost earlier, probably it could be got again."

11.6.4 Why Residual Connections Matter

Solving the vanishing gradient. During backpropagation, gradients flow through two paths: the main path (through $F (x)$ ) and the skip path (through $x$ ). The skip path has derivative 1 — the gradient passes through unchanged. This creates a gradient highway that delivers gradient signal directly from the loss to early layers without attenuation.

Ensemble behavior. A network with skip connections can be viewed as an ensemble of many shallower sub-networks. When you skip from layer 1 directly to layer 50, that path effectively creates a distinct 2-layer sub-network. A ResNet with $n$ blocks implicitly contains $2^{n}$ possible paths. Training it is like training an ensemble of networks of different depths simultaneously.

Feature reuse. The copied features ( $x$ ) are available to later layers even if the intermediate processing lost or distorted them. Low-level information (edges, corners) persists through the entire depth.

11.6.5 Worked Example: Numerical Trace

Consider a residual block where the input $x = [2.0, 0.5, 1.0]$ (a small 3-value feature vector).

The block's layers compute $F (x) = [0.3, 0.1, - 0.2]$ .

The residual output is:

F (x) + x = [0.3 + 2.0, 0.1 + 0.5, - 0.2 + 1.0] = [2.3, 0.6, 0.8]

Key observation: The input values $[2.0, 0.5, 1.0]$ are still visible in the output $[2.3, 0.6, 0.8]$ . The block only had to learn small adjustments $[0.3, 0.1, - 0.2]$ . Without the skip connection, the block would need to learn the full $[2.3, 0.6, 0.8]$ from scratch — a much harder task.

Sense-check: If the block's layers are not helpful, $F (x)$ can be driven to zero by training. Then the output is just $x$ — the block becomes an identity function. This means adding more layers can never hurt. In the worst case, they learn $F (x) = 0$ and behave as if they were not there.

11.6.6 Comparison: Plain Networks vs Residual Networks

Feature	Plain Network	Residual Network
Block output	$F (x)$	$F (x) + x$
Gradient flow	Through every weight matrix	Direct path with derivative 1
Identity mapping	Must be learned	Built in (set $F (x) = 0$ )
Depth limit	Degradation beyond ~20 layers	Can train 152+ layers
What block learns	Full transformation	Only the residual (difference)

When to use: Use residual connections any time your network goes beyond ~20 layers. Below that depth, plain networks work fine. Above it, residuals are essentially mandatory.

11.6.7 ResNet Variants

Multiple versions exist, named by layer count:

ResNet-18 and ResNet-34 — use basic residual blocks (two 3×3 convs per block)
ResNet-50, ResNet-101, ResNet-152 — use bottleneck blocks (1×1 → 3×3 → 1×1) with the 1×1 layers compressing and then expanding channels

Each variant was trained on ImageNet and compared. The deeper variants achieve higher accuracy, confirming that residual connections truly solve the depth problem.

11.6.8 Where Residual Connections Are Used

Residual connections are now a standard component far beyond ResNet:

Transformers — apply residual connections at every attention and feed-forward sub-layer. "Transformers apply the residual connections in multiple levels of their network."
DenseNet — an extension that connects every layer to every subsequent layer (extreme version of residual connections)
U-Net — for biomedical image segmentation, uses skip connections between encoder and decoder
EfficientNet — for efficient model scaling

11.6.9 Visual Intuition

Picture a 50-lane highway. Without skip connections, every car (gradient) must pass through 50 tollbooths (weight matrices). At each booth, part of the car's value is lost. By booth 50, the car is barely a bicycle. With skip connections, every few booths has an express lane that bypasses them entirely. Cars take the express lane and arrive at deep booths intact. They still have enough left to reach the early booths on the way back. The network trains from both ends simultaneously.

11.6.10 Pitfalls

Shape mismatch. The addition $F (x) + x$ requires both tensors to have the same shape. If the block changes channel count or spatial dimensions, you need a projection shortcut. Use a 1×1 convolution on the skip path. It projects $x$ to the right shape. This is called a projection shortcut.
Forgetting batch normalization. The professor's open question hints at this. When you add raw $x$ to processed $F (x)$ , their value scales may differ. Batch normalization inside the block keeps the scales compatible. Apply it before addition.
Skip connection placement. The standard order is: Conv → BN → ReLU → Conv → BN → add skip → ReLU. Putting ReLU before the skip changes what the block learns. The exam may ask you to identify or correct the order.
Not every block benefits equally. Skip connections are most critical in the middle-to-late blocks where gradient vanishing would otherwise dominate. In very early layers, plain convolutions may work fine.

11.6.11 Open Question from the Professor

"When you add the raw input $x$ to the processed output $F (x)$ , do you foresee any challenge? Something related to normalization. Research a little bit and come back."

Answer direction: The two values may have different scales. $F (x)$ has passed through convolution and activation and may have a very different distribution than the raw input $x$ . Adding them directly could destabilize training. Batch normalization — applied inside the block before the addition — normalizes $F (x)$ so its scale matches $x$ . This is why ResNet blocks always use BN before the addition.

Residual connections solved the deepest problem in deep networks: making them deeper made them worse. The fix — add a copy of the input to every block's output — creates a gradient highway and enables ensemble-like behavior. It lets networks reach 152+ layers. Skip connections are now used everywhere: Transformers, DenseNet, U-Net, EfficientNet. If you take one idea from this lecture beyond CNNs, take residual connections.

Real-world connection: Every modern large language model (GPT, Claude, Gemini, Llama) uses residual connections in its Transformer blocks. Without residual connections, Transformers could not be stacked 96+ layers deep. The same skip-connection pattern enables U-Net to segment tumors in medical scans and DenseNet to classify images with extreme parameter efficiency. The residual connection is genuinely one of the top 5 most important ideas in deep learning.

11.6.12 Exam Notes

Exam note: This is a high-value concept. Expect questions on why residual connections are necessary — they solve vanishing gradient in deep networks. Know the formula: $Output = F (x) + x$ . Understand the gradient highway concept (skip path has derivative 1). The ensemble behavior explanation (implicit ensemble of shallower sub-networks). Know where residual connections are used: Transformers, DenseNet, U-Net, EfficientNet. The professor's open question about normalization mismatch is an exam cue — the answer involves batch normalization.

11.7 Transfer Learning — Reusing Pre-Trained Models

Hook: Training a deep CNN from scratch needs millions of labeled images and weeks of GPU time. What if you have only 500 images? Transfer learning lets you start from someone else's finished model and adapt it to your task. This cuts data needs and training time by orders of magnitude.

11.7.1 Purpose

Transfer learning reuses a model trained on a large dataset for a new, related task. You load pre-trained weights that already encode general visual knowledge — edges, textures, shapes. You then adapt only the task-specific layers.

The key insight: low-level features transfer across visual tasks. A model that distinguishes dogs from cats already knows what an edge looks like. That knowledge does not need to be relearned.

11.7.2 Inputs & Outputs

Inputs:

A pre-trained model with saved weights (e.g., ResNet trained on ImageNet)
Your target dataset, typically much smaller than the source dataset
Images whose input dimensions and channel count match the pre-trained model

Outputs:

A model adapted to your task
Trained weights for the new classifier layers
Optionally fine-tuned weights for some or all pre-trained layers

Model weights flow in one direction: source model → loaded → frozen or adapted → target model. Pre-trained weights are always the starting point — you never get a fully from-scratch model as the output.

11.7.3 Decision Framework

Choosing the right strategy depends on two factors. One is how much data you have. The other is how similar your target task is to the source task. The professor's 4-quadrant decision table:

Dataset Size	Similarity	Strategy
Small	Similar	Feature extraction: freeze all conv layers, train only the classifier (FC layers). Reinitialize and train weights between the last hidden layer and the output layer.
Medium	Different	Fine-tuning: freeze early conv layers (low-level features), train the entire FC network. More layers are updated than in feature extraction.
Large	Similar	Full fine-tuning: initialize with pre-trained weights, train all layers with a very small learning rate. Use when accuracy is critical.
Small	Very different	Do NOT use transfer learning. Train from scratch. Selective fine-tuning has no reliable framework — you cannot know which layers to freeze and which to train.
Large	Very different	Train from scratch. Do not even consider selective fine-tuning.

Feature extraction freezes all convolutional layers and retrains only the classifier. The pre-trained conv layers act as fixed feature-extraction components. This works when your small dataset is similar to the source — the pre-trained features are already meaningful.

Fine-tuning freezes early convolutional layers but trains later layers and the entire FC network. Early features (edges) still transfer, but higher-level features need adjustment for the new domain.

Full fine-tuning trains everything with a very small learning rate. All pre-trained weights get a chance to adjust. Use this only with a large dataset — otherwise it overfits.

11.7.4 Trace: Vehicle Number Plate Recognition

The professor walks through a concrete application end to end.

Goal: Read vehicle registration numbers from traffic camera images.

Problem: You have only a handful of images. That is nowhere near enough to train a CNN from scratch. The network would need to learn:

What a vehicle looks like
How to separate foreground from background
How to distinguish different vehicle types
Where the number plate sits on a vehicle
How to read the digits on the plate

Step 1 — Find a pre-trained model. Search for an existing model that can already localize a vehicle in an image. If such a model exists, it already solves roughly 75% of the problem. It knows what a vehicle looks like and where it appears in the frame.

Step 2 — Load and freeze. Load the pre-trained weights. Freeze all layers responsible for detecting vehicle shapes and plate positions. These layers must not change during training.

Step 3 — Add new layers. Append one or two new layers on top of the frozen model. These layers will learn digit recognition from the plate region features.

Step 4 — Train only the new layers. Feed your number-plate images through the frozen base model. The lower layers extract features at the plate location. Train only the new top layers on these features to recognize digit sequences.

Result: The lower layers (vehicle detection, plate localization) remain frozen and unchanged. The new top layers learn to read digits. You built a plate reader with only a handful of labeled images.

Prerequisite: The professor says: "There has to be some similarity between what is existing and what you want." The pre-trained model must be related to your task. You cannot use a dog-breed classifier for vehicles. Unless the model already handles vehicle localization.

11.7.5 Complexity & Cost

Transfer learning is far cheaper than training from scratch — but it is not free.

Storage cost. Pre-trained checkpoints are large. A ResNet-152 file is hundreds of megabytes. Loading it requires disk space and RAM.

Compatibility overhead. Input sizes must match. AlexNet expects 227×227. ResNet expects 224×224. If the model was trained on 3-channel RGB, you cannot feed 1-channel grayscale without conversion. Channel mismatches cause silent failures.

Fine-tuning compute. When you fine-tune many layers, backpropagation flows through the full network. Each epoch is as expensive as training from scratch. The savings come from needing far fewer epochs, not cheaper epochs.

Cost of wrong strategy. If you pick the wrong strategy — fine-tuning with a tiny dissimilar dataset — you waste GPU hours. You get worse results than training a small model from scratch. The 4-quadrant decision table exists to prevent this mistake.

11.7.6 When to Use / Alternatives

Use transfer learning when:

Your target dataset is small (hundreds or a few thousand images)
Your task is visually similar to the source task
You need reasonable accuracy quickly
You lack the compute budget to train a large model from scratch

Do NOT use transfer learning when:

Your target task is fundamentally different from the source (X-rays vs. natural images — low-level features differ)
You have a very large dataset of your own — train from scratch
You need to modify the architecture deeply — the pre-trained structure becomes a constraint

Alternatives:

Train from scratch on your own large dataset. No compatibility constraints.
Data augmentation with a smaller custom model. Sometimes a simple CNN with heavy augmentation beats a misapplied transfer setup.
Self-supervised pretraining on your own domain data, then fine-tuning. Better than transfer from a dissimilar domain when you have unlabeled in-domain data.

11.7.7 Pitfalls

Input size mismatch. AlexNet expects 227×227 images. Feeding different-sized images causes errors. "People typically make this mistake."
Channel mismatch. If the pre-trained model uses 3-channel RGB and you provide 1-channel grayscale, the first convolution layer will fail. Convert grayscale to 3-channel or modify the first layer.
Overwriting pre-trained weights. Some frameworks let you accidentally reinitialize weights after loading. Always verify loaded weights are actually in use.
Learning rate too high. During fine-tuning, use a small learning rate. The pre-trained weights are already close to good values. A large learning rate destroys them.
Wrong strategy choice. Feature extraction on very different data gives poor accuracy. Full fine-tuning with a tiny dataset causes immediate overfitting.

The professor stresses: "It is necessary to understand the architecture. If something goes wrong, you know what is wrong. Perhaps you have not scaled correctly. Or perhaps it expects an RGB channel but you gave a grayscale image instead."

11.7.8 Student Q&A

Q: Is the final classifier just a softmax layer?

A: No. The final classifier refers to the fully connected layers (the FC network), not just the softmax. With a small similar dataset, use feature extraction. Freeze all convolution layers. Freeze the earlier FC layers. Retrain only the last-hidden-to-output connections. The softmax is the final activation — the classifier is the entire FC block that feeds into it.

11.7.9 Real-World Connection

Transfer learning is the default approach in industry:

Medical imaging: Models pre-trained on ImageNet are fine-tuned for tumor detection, retinal analysis, and X-ray classification. Medical datasets rarely grow large enough for from-scratch training.
Autonomous vehicles: Object detection models start from large road-scene pre-trained backbones, then fine-tune for specific sensor setups.
Retail: Product recognition systems use pre-trained backbones fine-tuned on store-specific catalogs.
Satellite imagery: Land-use classifiers start from ImageNet or remote-sensing pre-trained models, then fine-tune on regional data.

Nearly every deployed vision model today starts from a pre-trained checkpoint. Training from scratch is the exception, not the rule.

11.7.10 Recap & Bridge

Recap: Transfer learning loads pre-trained weights and adapts them for a new task. The decision table guides strategy. Use feature extraction for small/similar data. Use fine-tuning for medium/different data. Use full fine-tuning for large/similar data. Train from scratch when domains differ. The vehicle plate example shows how a vehicle-localizing model becomes a digit reader by freezing the base and training new classifier layers.

Bridge: Transfer learning gives you a trained model. But what if the pre-trained architecture itself is limiting? The next topics cover architectural innovations. These include residual connections and inception modules. They make transfer learning even more effective by providing stronger pre-trained backbones.

11.8 CNN Applications — Classification, Detection, and Segmentation

CNNs go beyond predicting a single label. Three core visual tasks form a hierarchy. Classification asks: what is in the image? Detection asks: where is each object? Segmentation asks: which pixels belong to which object? This section compares their output structures, evaluation metrics, and architectural choices.

11.8.1 Symbol Registry — Detection and Segmentation

Symbol	Meaning	Type / Domain
$(x, y)$	Bounding box top-left coordinates	scalar, pixels
$w, h$	Bounding box width and height	scalar, pixels
$K$	Number of classes	integer
$IoU$	Intersection over Union	scalar, $[0, 1]$

11.8.2 Image Classification

Hook: You open a photo app. It sorts your pictures into "beach," "mountain," and "birthday party." That is image classification — one label for the whole image.

Classification Output Structure: A standard CNN ends with a softmax layer that outputs a probability vector over $K$ classes. The predicted class is $ar g max_{k} P (y = k ∣ x)$ .

A standard CNN for classification follows this pattern:

Multiple blocks of convolution + pooling (repeated any number of times)
Option 1: Global Average Pooling → directly to output (softmax for multi-class, sigmoid for binary)
Option 2: Flatten → fully connected layers → output (softmax/sigmoid)

The loss function is cross-entropy. The evaluation metric is often top-K accuracy.

Top-K accuracy: When the model outputs probabilities for, say, 1000 classes, check whether the correct class appears among the top K predictions. For example, top-5 accuracy asks whether the correct answer is in the model's top five guesses. This gives a more nuanced view of model performance. It is especially useful when comparing architectures and hyperparameter choices.

Pitfalls:

Class imbalance: Suppose you want to predict 10 classes. But you have no data for 3 of them. The model cannot learn those classes.
Overfitting: Data augmentation helps; proper sampling ensures all classes are represented.
Top-K vs Top-1: Top-K is a softer metric. It does not tell you whether the model is confidently wrong on the remaining classes.

11.8.3 Object Detection

Hook: Classification tells you a photo contains a cat. Detection draws a box around the cat and says "cat." Localization + classification in one pass.

Detection Output Structure: Each detected object requires:

4 values for the bounding box: $(x, y, width, height)$ — continuous regression outputs
$C$ values for class probabilities — one per class, via softmax
Total output nodes per object: $4 + C$

For an image containing flowers of three types (A, B, C), each detected flower outputs 7 numbers.

YOLO vs R-CNN: A Comparison

Property	YOLO (Single-Shot)	R-CNN (Two-Stage)
Passes through network	One forward pass	Region proposal → classify (two passes)
Speed	Real-time	Slower
Accuracy	Slightly lower	Higher
How it works	Divides image into grid; predicts bbox + class per grid cell	Proposes region candidates, filters redundant ones, classifies remainders
Typical use	Real-time video, dissertations, quick prototyping	High-precision tasks where speed is less critical

Bounding Box Evaluation: IoU and Non-Max Suppression

Intersection over Union (IoU):

IoU = \frac{Area ( bbox _{pred} \cap bbox _{truth} )}{Area ( bbox _{pred} \cup bbox _{truth} )}

IoU ranges from 0 (no overlap) to 1 (perfect match). Higher IoU means better localization. The professor's description: "Come up with an area of intersections, also the area of union. Take the ratio. The more the value, the better."

Non-Max Suppression (NMS): When multiple bounding boxes claim the same object with high IoU, keep only the best one. Rank all predicted boxes by a confidence score:

score = P (object) \times IoU (bbox_{pred}, bbox_{truth})

Select the box with the highest confidence and discard the rest. The professor's description: "Multiply the probability with which an object is present... Multiply it with IoU region and then select the box with the highest confidence score and ditch the other."

Pitfalls:

Small objects: YOLO struggles with tiny objects because grid cells may be too coarse.
Overlapping objects: NMS can accidentally suppress a legitimate adjacent object if IoU threshold is too aggressive.
Speed-accuracy trade-off: Two-stage = slower but better; single-shot = faster but misses fine details.

11.8.4 Semantic Segmentation

Hook: Classification says "this is a road." Detection draws a box around a car. Segmentation colors every pixel — road pixels in gray, car pixels in red, sky pixels in blue. Pixel-level understanding.

Segmentation Output Structure: For an input image of size $H \times W$ , the output is an $H \times W \times K$ tensor. Here $K$ is the number of classes. For each pixel position $(i, j)$ , a softmax over the $K$ channels gives a probability distribution. The predicted class at $(i, j)$ is $ar g max_{k}$ . Every pixel gets a label.

Fully Convolutional Networks (FCN)

Traditional CNNs end with fully connected (FC) layers that destroy spatial structure. FCNs replace all FC layers with $1 \times 1$ convolutions:

After the final conv+pool layers, you have a feature map of size $H^{'} \times W^{'} \times C$ .
Apply $1 \times 1$ convolutions to produce $H^{'} \times W^{'} \times K$ — one channel per class.
Apply softmax pixel-by-pixel across the channel dimension.
This preserves spatial structure end-to-end. Every computation stays convolutional.

The professor's description: "Take the feature map, apply the necessary computations and concatenations, and directly proceed with the output pixel by pixel. One pixel's information is consolidated due to computations. Apply a softmax and come up with the answer."

Transposed Convolution (Upsampling)

Convolutions and pooling shrink spatial dimensions (e.g., $10 \times 10 \to 5 \times 5$ ). For pixel-wise tasks, you must recover the original size. Transposed convolution (sometimes called deconvolution or upsampling) does this:

It "zooms out" the compressed feature maps by inserting zeros (padding) and then convolving.
The kernel learns how to upsample during training.
Result: output map matches original input resolution for per-pixel classification.

Pitfalls:

Resolution loss: Pooling discards fine-grained spatial details. FCNs without skip connections produce coarse segmentation boundaries.
Computational cost: Processing every pixel is more expensive than processing one label per image.

11.8.5 Student Q&A on Transposed Convolution

Q: What is the purpose of transposed convolution?

A: To zoom out (upsample) the image back to its original size. Convolutions and pooling shrink the image to, say, $5 \times 5$ . Transposed convolution zooms it back out. It fills rows symmetrically with padding zeros. These zeros do not impact learned values. This recovers the original $10 \times 10$ resolution needed for pixel-wise classification.

11.8.6 Beyond Computer Vision

CNNs are not restricted to image data. Any data where spatial correlation exists and where locality is the primary characteristic can benefit from convolutional processing. Examples:

OCR: Converting images of text to machine-readable characters.
Audio: 1D convolutions over time-domain waveforms or 2D spectrograms.
Time-series: 1D convolutions for anomaly detection, sensor data.

11.8.7 Quick Reference: Output Structures

Task	Output shape	Loss
Classification	$K$ -vector (class probabilities)	Cross-entropy
Detection	$(x, y, w, h) + K$ -vector per object	Bbox regression + class loss
Segmentation	$H \times W \times K$ per-pixel class map	Per-pixel cross-entropy

11.8.8 Exam Notes

Expect questions distinguishing classification, detection, and segmentation. Know the output structures. Classification outputs a class probability vector. Detection outputs a bounding box with 4 values plus class probabilities. Segmentation outputs a per-pixel class map. The YOLO vs R-CNN distinction is likely to appear. Top-K accuracy as an evaluation metric is also testable.

Exam Guidance Summary

This section consolidates the exam hints scattered across the lecture. The professor flagged specific question types and topics. Use this as your study checklist.

Parameter Counting (Highest-Weight Exam Topic)

You may be given an architecture diagram. You must compute parameters for each layer.
You may need to identify errors in stated parameter counts.
You may need to reverse-engineer kernel sizes from input and output dimensions using: $H_{out} = \frac{H _{in} - K + 2 P}{S} + 1$
Critical trap: In layer 2+ of a CNN, each kernel spans ALL input channels. Suppose Conv2 has 6 input channels. Each 5×5 kernel spans them all. So each kernel has $5 \times 5 \times 6 = 150$ weights. It is NOT $5 \times 5 = 25$ .
Pooling layers have ZERO learnable parameters. Do not count them.

Architecture Comparison Questions

These are likely to appear:

Comparison	Key Answer
AlexNet vs LeNet-5	ReLU (vanishing gradient), Dropout (regularization), Data Augmentation (more data)
VGGNet vs AlexNet	Uniform 3×3 filters everywhere, blocks of repeated layers, deeper
ResNet vs plain deep nets	Residual connections: $F (x) + x$ solves vanishing gradient
GoogleNet vs AlexNet	Multi-scale (parallel kernels), 12× fewer parameters, auxiliary classifiers

Understanding Purpose, Not Just Structure

For every architecture, know three things:

Why the technique was introduced (what problem existed)
What problem it solved
Where it is applied today

Transfer Learning Decision Framework

Exam note: Know the four-quadrant strategy table:

Small dataset + similar task → Feature extraction (freeze all conv, retrain classifier)
Medium dataset + different task → Fine-tuning (freeze early conv, train FC layers)
Large dataset + similar task → Full fine-tuning with small learning rate
Small dataset + very different task → Do NOT use transfer learning
Large dataset + very different task → Train from scratch

Error-Detection Problems

You may get an architecture with stated kernel sizes and output dimensions. You must find inconsistencies. The output size formula is your primary tool.

Key Reminders

Kernel size is a hyperparameter — determined experimentally, just like learning rate. There is no formula for choosing it.
Topics NOT covered in depth (mentioned for awareness): batch normalization (covered in later modules), autoencoders, GANs, encoder-decoder architectures (later semesters).

Key Industry Applications

Every architecture in this lecture started as a research paper and ended up in production. Here is where each one lives today. From the ATM that reads your cheque to the phone that unlocks with your face.

LeNet-5: Handwritten digit and character recognition. The foundation of OCR systems used in banks, postal services, and cheque processing. Some ATMs still run LeNet code from the 1990s.
AlexNet: The 2012 ImageNet winner that launched the deep learning era. Its innovations — ReLU, dropout, and data augmentation — are now standard in virtually every deep learning pipeline. Showed that GPUs could train networks previously thought impossible.
VGGNet: Widely adopted as a backbone for transfer learning. Its uniform, simple structure (all 3×3 filters) makes it easy to modify and reuse. Still used for feature extraction in many computer vision projects.
1×1 Convolution: Core component in InceptionNet, ResNet, and MobileNet. The go-to technique for reducing parameters when deploying models on embedded systems and mobile devices. Enables deep networks to run on hardware with limited computation.
Global Average Pooling (GAP): Replaces heavy FC layers in modern CNNs. Used in GoogleNet, ResNet, and most current architectures. Dramatically reduces parameter counts without losing spatial information.
GoogleNet / InceptionNet: Multi-scale feature extraction. Won ILSVRC 2014. Achieved 12× fewer parameters than AlexNet while being more accurate. The inception module design influenced nearly every subsequent architecture.
ResNet: Residual connections are foundational to modern deep learning. Used in Transformers — the architecture behind GPT, BERT, and all modern LLMs. Also in DenseNet, U-Net (medical imaging), and EfficientNet. The skip connection is one of the most important ideas in neural network design.
Transfer Learning: The industry standard when data is limited. Reuse pre-trained ImageNet models for domain-specific tasks. Powers most real-world computer vision applications where labeled data is scarce.
YOLO (You Only Look Once): Real-time object detection. Used extensively in surveillance systems, autonomous vehicles, retail analytics, and countless student dissertation projects. Processes video frames in milliseconds.
R-CNN / Fast R-CNN: Two-stage object detection offering higher accuracy than YOLO at the cost of speed. Used where precision matters more than real-time performance.
Fully Convolutional Networks (FCN): Pixel-wise classification for semantic segmentation. Used in medical imaging (tumor detection), autonomous driving (road/lane detection), and satellite image analysis.
Transposed Convolution: Upsampling in encoder-decoder architectures. Used for image generation, super-resolution, and segmentation tasks where output must match input dimensions.
CNNs beyond vision: Any data with spatial correlation and locality can use CNNs. Examples: OCR, audio spectrograms, time-series data reformatted as images, and even some NLP tasks before Transformers took over.
ILSVRC (ImageNet Large Scale Visual Recognition Challenge): The competition that drove most of these innovations. The key architectural breakthroughs happened between 2012 and 2015. AlexNet (2012), GoogleNet (2014), and ResNet (2015) were all ILSVRC winners.

DNN Lecture 11 notes · CNN Architectures and Applications

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

1LeNet-5 — The Foundational CNN Architecture

The first CNN to pair convolution and pooling for handwritten digit recognition; layer-by-layer parameter counting and the output-size formula.

2AlexNet — ReLU, Dropout, and Data Augmentation

The 2012 ImageNet winner that introduced ReLU, dropout, and data augmentation, plus GPU training at ~60M parameters.

3VGGNet — Depth Through Uniformity

Uniform 3x3 convolution blocks stacked deep; why two 3x3 convs replace one 5x5 in receptive field and parameters.

4Network in Network — 1x1 Convolution and Global Average Pooling

1x1 convolutions as channel-mixing bottlenecks and GAP as a parameter-free replacement for fully connected heads.

5GoogleNet / InceptionNet — Multi-Scale Feature Extraction

The inception module running parallel kernel sizes with 1x1 bottlenecks and auxiliary classifiers.

6Residual Connections — Solving the Deep Network Problem

Skip connections (Output = F(x) + x) that fix degradation and build a gradient highway for very deep networks.

7Transfer Learning — Reusing Pre-Trained Models

The four-quadrant decision framework for feature extraction, fine-tuning, and full fine-tuning with a worked vehicle plate example.

8CNN Applications — Classification, Detection, and Segmentation

Output structures and metrics for the three core vision tasks, including IoU, NMS, FCNs, and transposed convolution.

9Exam Guidance Summary

Consolidated exam hints: parameter counting, architecture comparisons, transfer learning strategy, and error-detection problems.

10Key Industry Applications

Where each architecture and technique lives in production today, from ATMs and medical imaging to autonomous vehicles and LLMs.

Postgraduate students in Deep Learning and Computer Vision

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

LeNet-5 and the Output Size Formula

Must-know: LeNet-5 (1998) was the first CNN to pair convolution with pooling for digit recognition. The single most examinable skill is counting parameters layer by layer and reversing kernel size from input/output dimensions. Remember: pooling has zero learnable parameters, and in layer 2+ each kernel spans all input channels.

$H_{out} = \frac{H _{in} - K + 2 P}{S} + 1$

⚠️ Top pitfall: Forgetting the input-channel dimension. In Conv2 of LeNet-5, each 5×5 kernel spans all 6 input channels, so its weight count is $5 \times 5 \times 6$ , not $5 \times 5$ . LeNet-5 totals ~61,706 learnable parameters.

Self-check: Given a 14×14 input, stride 1, no padding, and a 10×10 output, what is the kernel size?

Connects to: AlexNet (deeper, ReLU), VGGNet (uniform blocks), parameter counting in any later architecture.

AlexNet — ReLU, Dropout, Data Augmentation

Must-know: AlexNet (2012) won ImageNet with ~60M parameters. Its three innovations each solve a distinct problem: ReLU fixes the vanishing gradient, dropout prevents overfitting during training, and data augmentation synthesizes more training data cheaply.

$ReLU (x) = max (0, x) output = (1 - p) \cdot z at test time$

⚠️ Top pitfall: Leaving dropout ON during testing — the network behaves randomly and accuracy collapses. Dropout is training-only. Also beware label-flipping augmentation (a mirrored "6" becomes "9").

Self-check: Why does a 227×227 input (not 224×224) matter when you reuse AlexNet for transfer learning?

Connects to: LeNet-5 (the blueprint it scaled), VGGNet (depth), transfer learning (input-size matching).

VGGNet — Depth Through Uniformity

Must-know: VGGNet (2014) proved that stacking many uniform 3×3 conv blocks beats mixing kernel sizes. Two stacked 3×3 convolutions match one 5×5 receptive field but use fewer parameters and more ReLU non-linearity. VGG-16 = 13 conv + 3 FC layers.

$params (5 \times 5) = 25 C^{2} vs params (2 \times 3 \times 3) = 18 C^{2}$

⚠️ Top pitfall: Most of VGG's ~138M parameters live in the three FC layers (the FC-4096 alone is ~102M), not the convolutions — so it is memory-hungry and overfits small datasets.

Self-check: How many parameters and how much ReLU does replacing one 7×7 conv with three 3×3 convs change?

Connects to: AlexNet (deeper, uniform), Global Average Pooling (which removes the FC head), ResNet (bottleneck blocks).

1×1 Convolution

Must-know: A 1×1 convolution has no spatial extent — it mixes channels at a single pixel, acting as a tiny per-pixel fully connected layer. Used as a bottleneck, it slashes parameters before expensive spatial convolutions.

$output_{c^{'}} (i, j) = k = 1 \sum C w_{c^{'}, k} \cdot ch_{k} (i, j)$

⚠️ Top pitfall: Forgetting ReLU after a 1×1 conv — without it, the operation is just a linear projection with no expressive power. Also it cannot detect spatial patterns (edges, textures) on its own.

Self-check: With 256 input channels, compare parameters for a direct 5×5 conv to 256 outputs vs. a 1×1(256→64) + 5×5(64→256) bottleneck.

Connects to: Global Average Pooling, GoogLeNet (bottlenecks inside inception), ResNet (bottleneck blocks), MobileNet.

Global Average Pooling (GAP)

Must-know: GAP replaces the giant FC head: it averages each feature map to a single scalar, producing an $N$ -vector that feeds the classifier directly. Zero learnable parameters, so far less overfitting than FC layers.

$GAP (F_{c}) = \frac{1}{H \cdot W} i = 1 \sum H j = 1 \sum W F_{c} (i, j)$

⚠️ Top pitfall: GAP loses where a feature appears — it averages over all positions. For localization or detection (bounding boxes) it is too coarse; use it only when spatial precision is unnecessary.

Self-check: A 7×7×512 feature map passes through GAP. What is the output shape, and how many parameters does GAP learn?

Connects to: 1×1 Convolution (often paired), Network-in-Network, GoogLeNet head, the FC-layer-vs-GAP architectural shift.

GoogLeNet / Inception — Multi-Scale Features

Must-know: GoogLeNet (2014) runs 1×1, 3×3, 5×5 convolutions and a pooling branch in parallel on the same input, then concatenates them. 1×1 bottlenecks inside each branch make this 12× cheaper than AlexNet (~5M vs ~60M params). Auxiliary classifiers inject gradient signal during training only.

$output = [Conv_{1 \times 1} ∥ Conv_{3 \times 3} ∥ Conv_{5 \times 5} ∥ Pool_{3 \times 3}]$

⚠️ Top pitfall: Assuming auxiliary classifiers are used at test time — they are training-only. Also, concatenation balloons the channel count, which is exactly why 1×1 bottlenecks are mandatory.

Self-check: Why does a 5×5 branch get replaced by two stacked 3×3 convs in later Inception versions?

Connects to: 1×1 Convolution (bottlenecks), VGGNet (depth), ResNet (which followed).

Residual Connections (ResNet)

Must-know: ResNet (2015) solves the degradation problem — deeper plain nets perform worse. A skip connection adds the input $x$ back to the block output, so the block learns only the residual difference and gradients flow through an unattenuated "gradient highway."

$Output = F (x) + x$

⚠️ Top pitfall: Shape mismatch — $F (x)$ and $x$ must have the same shape, or you need a 1×1 projection shortcut. Also, batch normalization must sit before the addition so the two terms have compatible scales.

Self-check: If a block's layers are useless, what does $F (x)$ become, and why can adding more residual layers never hurt?

Connects to: Vanishing gradient (Lecture 6/7), Transformers, U-Net, DenseNet, EfficientNet — every modern deep architecture.

Transfer Learning

Must-know: Reuse a pre-trained model's weights and adapt only the task-specific layers. Pick the strategy from a 2-factor table: dataset size × task similarity. Small+similar → freeze conv, retrain classifier; medium+different → fine-tune; large+similar → full fine-tune with tiny LR; very different → train from scratch.

$strategy = f (data size, task similarity)$

⚠️ Top pitfall: Input/architecture mismatch — feeding grayscale to an RGB model, or a 256×256 image to a 227×227 input, fails silently. Always match input size and channel count, and use a small learning rate when fine-tuning.

Self-check: You have 500 X-ray images and a ResNet trained on ImageNet. Which strategy, and why?

Connects to: AlexNet/VGGNet/ResNet backbones, the vehicle plate-recognition worked example, every deployed vision model.

CNN Tasks — Classification, Detection, Segmentation

Must-know: The three tasks differ only in output structure. Classification → a $K$ -vector of class probabilities. Detection → a bounding box $(x, y, w, h)$ plus a $C$ -vector per object. Segmentation → an $H \times W \times K$ per-pixel class map.

$IoU = \frac{Area ( bbox _{pred} \cap bbox _{truth} )}{Area ( bbox _{pred} \cup bbox _{truth} )}$

⚠️ Top pitfall: Confusing the three — especially forgetting that detection adds 4 regression values per object, and that segmentation needs transposed convolution to upsample back to input resolution. YOLO (fast) vs R-CNN (accurate) is a standard compare.

Self-check: For a 224×224 image with 20 classes, what is the output tensor shape of a segmentation FCN?

Connects to: 1×1 Convolution and Global Average Pooling (FCN replaces FC with 1×1 convs), transposed convolution, top-K accuracy.

Previous LectureLecture 10 · Convolutional Neural Networks — From Pixels to Patterns

Next LectureLecture 12 · Recurrent Neural Networks — Foundations and Architecture

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.