Convolutional Neural Networks
Prerequisite Knowledge
This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.
Previously Covered in This Subject
- Introduction and Overview of Deep Neural Networks — covered in Lecture 1
- Deep Neural Network Components and Perceptron — covered in Lecture 2
- Perceptron Learning and Introduction to Regression — covered in Lecture 3
- Linear Neural Networks for Regression — covered in Lecture 4
- Gradient Descent Variants, Classification, and Evaluation — covered in Lecture 5
- Introduction to Convolutional Neural Networks — covered in Lecture 8
Convolutional Neural Networks
9.1 Convolutional Neural Networks — Introduction
9.1.1 Definition and Motivation
Why can't we just use a big fully connected network for images? A one-megapixel photo fed into a hidden layer of 1000 units needs a billion parameters. You would need enormous datasets and GPU farms just to avoid overfitting. Yet a child can spot a cat in a photograph instantly. The answer: images have structure. Nearby pixels are related. Objects look the same no matter where they appear. CNNs bake this structure into the network itself.
Analogy — the magnifying glass inspector. Imagine inspecting a huge factory floor through a small magnifying glass. You slide the glass across the floor one patch at a time. You note what you see — a bolt here, a scratch there. You use the same magnifying glass everywhere. You don't need a different glass for each square inch. That is exactly what a CNN does. It slides a small filter (the kernel) across the input, reusing the same weights at every position. The filter learns to detect a specific pattern — an edge, a corner, or later on, an eye or a wheel. The analogy breaks when the filter grows large enough to cover most of the image. At that point it is just a dense layer again.
A convolutional neural network (CNN) is a neural network that replaces at least one of its general matrix multiplications with a convolution operation. Formally, a convolution between an input and a kernel produces a feature map :
Here is the input. It could be an image, a time series, or any grid-structured data. is the kernel — a small array of learnable weights. is the feature map. It shows where the pattern matched. Each element is the dot product of the kernel and the patch of input centered at .
In a dense layer, every output unit connects to every input unit. In a convolutional layer, each output unit connects only to a small local patch. The patch size is governed by the kernel size. The same kernel weights are reused across all spatial positions.
Symbol registry:
| Symbol | Meaning | Notes |
|---|---|---|
| Input tensor | A grid of data — an image, time series, or 3D volume | |
| Kernel (filter) | A small array of learnable weights, slid across the input | |
| Feature map | The output of convolving with — shows where the pattern matched | |
| Convolution operator | The sliding dot-product operation defined above | |
| Spatial indices | Used to iterate over kernel positions and input patches |
Worked example — parameter count comparison. Suppose you have a grayscale image. That's 1 million pixels. You want to map it to a hidden layer of the same spatial size. The layer has 64 feature maps.
- Dense layer: Each of the output positions connects to each of the input pixels. That's parameters. That is a trillion — impossible to train.
- Convolutional layer with kernels: Each filter has weights (plus 1 bias). With 64 filters, that's parameters. Not 640 million — six hundred and forty. That's a reduction by a factor of over a billion.
Sense-check: A CNN with a few hundred parameters can process a million-pixel image. A dense network would need terabytes of memory just to store the weight matrix. The difference is not incremental — it is categorical.
Scope: when the CNN assumption holds. CNNs assume your data has a grid-like topology: 1D (time series, audio), 2D (images), or 3D (video, CT scans). They also assume that local patterns are meaningful and that the same pattern can appear at different positions. If your data has no such structure, a CNN adds no benefit. Think of a spreadsheet. Column 3 and column 17 have no spatial relationship. Sliding a kernel across them makes no sense. Using a CNN on tabular data is like using a magnifying glass to read a spreadsheet. The sliding makes no sense because nearby cells are not meaningfully related.
Visual intuition. Picture a grayscale image. The x-axis is the column index, the y-axis is the row index. A kernel slides across it. At each position, multiply the 9 kernel values by the 9 pixel values beneath them. Sum the result. The output is a smaller feature map (assuming no padding). Each cell in the feature map fires strongly where the kernel pattern matches. A horizontal-edge kernel lights up on image regions with a strong horizontal brightness change. A blotch-of-color kernel fires on uniform patches. A single image produces multiple feature maps — one per kernel — each highlighting a different aspect of the input. The takeaway: one small kernel, applied everywhere, produces a map of where that pattern lives.
Pitfalls:
- Confusing convolution with cross-correlation. In pure math, convolution flips the kernel before sliding. Most deep learning libraries do not flip — they use cross-correlation but call it convolution. The learning algorithm doesn't care because it just learns the flipped version of the kernel. This matters for reading proofs, not for implementation.
- Thinking CNNs are only for images. CNNs work on any grid-structured data: 1D audio waveforms, 3D medical scans, even graph-structured data with clever adaptations.
- Forgetting about channel depth. A color image is (RGB channels). The kernel also has depth. A kernel over a 3-channel input has weights, not 9. Each filter spans the full input depth.
- Expecting translation invariance from a single conv layer. A convolutional layer is translation equivariant, not invariant. Shift the input, and the output shifts by the same amount. True invariance (same output regardless of position) comes later from pooling layers — which are covered in the full treatment.
A CNN is a neural network that uses convolution (sliding dot products) instead of full matrix multiplication. This gives you three superpowers. First, sparse connections: each output sees only a local patch. Second, parameter sharing: the same kernel is reused everywhere. Third, translation equivariance: shift the input, shift the output. The result is a model that is both dramatically more parameter-efficient and naturally suited to grid-structured data. The full details were deferred due to time. These include strides, padding, pooling, multi-channel kernels, and modern architectures. They will appear in a dedicated session.
Real-world connection. Convolutional networks are the backbone of modern computer vision. Self-driving cars use CNNs to detect lanes, pedestrians, and traffic signs from camera feeds. Medical imaging systems use them to find tumours in MRI and CT scans. Your phone's face unlock runs a CNN. Beyond vision, CNNs process audio (speaker identification, keyword spotting), analyse time-series sensor data, and even power some natural language processing pipelines. The same three ideas — locality, weight sharing, and equivariance — made AlexNet (2012) the breakthrough that launched the deep learning revolution. It cut the ImageNet classification error rate nearly in half compared to the previous state of the art.
9.1.2 Prior Coverage
The motivation behind CNNs was first introduced in earlier sessions — you have already seen why fully connected networks fail on high-dimensional grid data and why a specialized architecture is necessary. The earlier coverage set up the problem. Dense networks treat every input pixel as an independent feature. They ignore the spatial relationships that make images meaningful. This session was intended to deliver the full solution — the convolution operation, pooling, and architectural patterns — but the lecture was cut short. The complete treatment of CNNs, including training mechanics and modern architectures, will be covered in a dedicated upcoming session.
9.2 Midterm Review and Exam Guidance
9.2.1 Midterm Exam Structure
The midterm paper tests your understanding across every topic covered in class. Nothing is excluded. If it was discussed in a lecture, it can appear on the paper. The exam tests three fundamentally different skills: reading code, computing by hand, and reasoning about scenarios. Students who spread their practice across the semester do better. Those who cram everything into the final days struggle.
The midterm examination consists of three question types:
- Code snippet questions. You are shown a short piece of neural network code (Python/TensorFlow/Keras style) and asked to reason about what it does. You may need to trace the forward pass, identify bugs, predict output shapes, or explain what a particular layer contributes.
- Calculation-based questions. You work through numerical computations by hand. This may mean computing a forward pass on paper. Or calculating gradients by backpropagation. Or solving for weight updates from a learning rate and loss number.
- Scenario-based questions. You get a described situation — a dataset, a problem, a proposed architecture. You must reason about what would work, what would fail, and why. These test whether you can apply concepts, not just recall them.
Pitfall: Many students underestimate the length of the paper. The exam is deliberately comprehensive. Without regular practice, you will run out of time. The paper is built for students who have internalized the patterns through repetition. It is not for someone seeing the problems for the first time.
9.2.2 Study Recommendations
The recommended study method is a two-pass approach applied to every exercise:
- First pass (guided). Work through the problem while referring to the lecture PPT and the provided solution slide. Understand every step. Don't just read — actively trace through the calculations.
- Second pass (independent). Attempt the same problem again later, on your own, without looking at the solution. This builds recall and exposes gaps you thought you had filled.
This two-pass method separates two things. Recognition: "I've seen this before and get it when I see the answer." Mastery: "I can produce the answer from scratch under time pressure."
Example of the two-pass method. Suppose the exercise asks you to compute the gradient of cross-entropy loss. You need it with respect to the weights of a single-layer softmax classifier. On pass one, follow the PPT steps: write the loss, expand the softmax, differentiate, simplify. On pass two, a few days later, do the same derivation on a blank sheet. If you get stuck at the softmax derivative, you know exactly which step needs more attention. Review just that piece. This targeted review beats re-reading the entire chapter.
Exam note: Spread your preparation across the semester. One hour per week after each class is enough to stay on top of the material. Trying to prepare everything in a single marathon session is stressful and ineffective. The concepts build on each other. Cramming leaves gaps that compound under exam pressure.
Pitfalls:
- Skipping the second pass. Reading the solution once feels productive. But it does not build recall under time pressure. You must practice retrieval, not just recognition.
- Neglecting code-snippet questions. Code questions test whether you truly understand each line. A student who recites backprop equations but cannot trace
model.fit(x, y, epochs=5)has only surface understanding. - Focusing only on calculation questions. The scenario-based questions carry significant weight. If you cannot reason about when to use batch normalization, or why a certain learning rate causes divergence, you are missing a big piece of the paper.
- Assuming a topic is "unlikely" to appear. The professor was clear: all topics discussed in class appear on the exam. There are no safe assumptions. Nothing is excluded.
Q: How was the midterm experience? Were students able to attempt all the questions?
A: Several students reported that the paper was lengthy. It covered all topics from class — code snippets, calculation problems, and scenario reasoning. Students who practiced regularly finished all questions within the time limit. Students who did not practice found the paper long and could not finish. The pattern was clear. Regular practice was the deciding factor.
The midterm rewards consistent, distributed practice. Do every exercise twice: once guided, once independently. One hour per week is enough. Cramming does not work for this paper. The question types are code snippets, hand calculations, and scenario reasoning — prepare for all three.
Exam Guidance Summary
The midterm examination follows a consistent structure. Review these points before your study sessions and again the night before the paper.
- Coverage. The midterm covers all topics discussed in class. Nothing is excluded. Do not gamble on which topics might or might not appear.
- Question types. Expect three formats. Code snippet questions: read and reason about neural network code. Calculation-based questions: work through numerical computations by hand. Scenario-based questions: apply concepts to a described situation and justify your reasoning.
- Two-pass exercise method. Do every exercise at least twice. First pass: work through the problem while referring to the PPT and solution. Second pass: attempt the same problem independently, without any reference material. The second pass is where real learning happens.
- Study cadence. Spend one hour per week after each class on the exercises. This is enough to stay on top of the material. Do not save all preparation for the days before the exam. The concepts build cumulatively. Gaps discovered late are hard to fill.
- Time management. Regular practice with exercises is the key to completing the paper within the time limit. Students who practice consistently finish. Students who cram do not.
Key Industry Applications
Convolutional neural networks are one of the foundational architectures in deep learning. The landmark AlexNet paper in 2012 showed that CNNs could dramatically outperform traditional computer vision methods. On the ImageNet benchmark, they cut the error rate nearly in half. Since then, CNNs have become the default building block for nearly any task involving grid-structured data.
- Computer vision. Image classification, object detection, semantic segmentation, and instance segmentation are all dominated by CNN-based architectures. Self-driving cars use CNNs (e.g., in Tesla's vision system) for real-time lane detection, pedestrian recognition, and traffic sign reading. Medical imaging systems apply CNNs to detect tumours in MRI and CT scans, sometimes exceeding radiologist-level accuracy for specific tasks.
- Audio and speech. 1D convolutions process raw audio waveforms and spectrograms. Voice assistants (Siri, Alexa, Google Assistant) use convolutional layers for keyword spotting and speaker identification. Music genre classification and environmental sound detection also rely on CNNs.
- Natural language processing. Transformers now dominate NLP. But 1D CNNs were widely used for text classification, sentiment analysis, and sentence modelling. They remain competitive for on-device tasks. Their cost is lower than the quadratic attention cost of transformers.
- Video analysis. 3D convolutions treat time as the third dimension. They process video clips for action recognition, anomaly detection in surveillance, and gesture recognition.
- Computational biology. CNNs predict protein folding patterns, analyse gene expression data arranged in spatial grids, and classify cell types from microscopy images.
- Generative modelling. Convolutional layers form the backbone of generative adversarial networks (GANs) for image synthesis, style transfer, and super-resolution.
The full treatment of CNN applications was deferred from this session due to time constraints. Architectural details and training considerations for each domain will be covered in the dedicated CNN lecture.
DNN Lecture 09 notes · Convolutional Neural Networks
Sections Breakdown
Definition and motivation for CNNs, the convolution operation, parameter sharing, and translation equivariance.
Midterm exam structure, question types, and the two-pass study method.
Consolidated exam preparation checklist and study cadence.
Where CNNs are used across vision, audio, NLP, video, biology, and generative modelling.
Exam Revision Notes
Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.
Convolution Operation
Must-know: A CNN replaces full matrix multiplication with a sliding dot product. The feature map is . The same kernel weights are reused at every position.
⚠️ Top pitfall: Confusing convolution with cross-correlation. Libraries do not flip the kernel; they use cross-correlation but still call it convolution. The learned weights just absorb the flip.
Self-check: For a 6×6 input and a 3×3 kernel with no padding, what is the output feature map size?
Connects to: Parameter Sharing, Translation Equivariance, Feature Map
Parameter Sharing and Sparse Connections
Must-know: A 3×3 kernel over a 1-megapixel image uses 9 weights per filter, not a trillion. This is why CNNs are feasible where dense layers are not. Each output connects only to a local patch.
⚠️ Top pitfall: Forgetting channel depth. A 3×3 kernel over a 3-channel RGB image has 3×3×3 = 27 weights, not 9. The filter spans the full input depth.
Self-check: How many weights does one 3×3 filter have on a 3-channel input, and why is that not 9?
Connects to: Convolution Operation, Feature Map, Translation Equivariance
Translation Equivariance vs Invariance
Must-know: A conv layer is translation equivariant: shift the input and the output shifts by the same amount. True invariance (same output regardless of position) comes later from pooling, not from a single conv layer.
⚠️ Top pitfall: Expecting a single convolutional layer to give translation invariance. It only gives equivariance. Invariance requires pooling or other aggregation.
Self-check: If you slide the input image two pixels right, what happens to the feature map produced by one conv layer?
Connects to: Convolution Operation, Parameter Sharing
Midterm Exam Structure and Strategy
Must-know: The midterm covers every topic discussed in class — nothing is excluded. It tests three skills: reading code, computing by hand, and reasoning about scenarios. Practice every exercise twice: once guided, once independently.
⚠️ Top pitfall: Skipping the second (independent) pass. Reading a solution once builds recognition, not recall under time pressure. Cramming does not work for this paper.
Self-check: Name the three question types on the midterm and what each one tests.
Connects to: Two-Pass Study Method, Scenario Reasoning
Was this lecture useful?
BitsNotes AI Assistant
Subject Notes AssistantConfigure AI Chat
Choose how to access the chatbotSigned in as
Powered by BitsNotes — 20 messages per day. No API key needed. Want unlimited access? Use "Bring Your Own Key" mode.
Sign in to use AI Chat
Get 20 free AI messages per day to ask questions about your lecture notes. Sign in with Google or GitHub — it takes 5 seconds.
Sign In to BitsNotesSwitch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.