Deep Neural Networks

Deep Neural Networks — Core Concepts and Introduction to CNNs

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students studying Deep Neural Networks

Exam Revision and Introduction to Convolutional Neural Networks

8.1 Exam Format, Scaling, and Preparation Strategy

Hook: The exam paper is marked out of 30 internally. But your score on the answer sheet is scaled to 100. A 2-mark question on the internal scheme becomes 5 to 7 marks in your final score. The difficulty is the same. Only the numbers on the paper change.

Intuition: Think of the scaling like enlarging a photo. Every pixel in the original image stretches to cover more area. But the picture content is unchanged. Similarly, every internal mark stretches to cover more final marks. The scaling works uniformly across all questions. The relative weight of each topic stays the same.

8.1.1 Exam Paper Structure and Mark Scaling

The exam paper is internally designed for 30 marks but scaled to 100 marks for grading. This does not make any question harder. A part worth 2 marks internally might appear as 5 or 7 marks on your final paper. Read the question-paper instructions carefully and allocate your time based on the scaled marks you see. The difficulty level does not change with the scaling.

Exam note: Do not panic when you see large mark allocations. Each question's intrinsic difficulty maps to the internal 30-mark scheme. The scaling is uniform — if a question is worth 2/30 internally (6.7% of the paper), it remains 6.7% of the scaled 100-mark paper.

8.1.2 Question Types to Expect

The exam may include several types of questions:

Numerical problems: Forward propagation, backward propagation, weight update computations. Practice with pen and paper at least twice for speed.
Code-based questions: You might get a partially filled code structure or pseudocode. Tasks include: (1) finding errors or bugs, (2) filling in blanks for specific formulas or equations, (3) completing underlined portions. You are NOT expected to know Python syntax perfectly. The intent is to test your understanding of forward propagation, backward propagation, and the code structure.
Theoretical questions: Explaining concepts, justifying choices of model architectures, activation functions, loss functions.

You can expect about 15--20 marks (out of 100) on code-related questions.

Exam note: Code questions do not test Python syntax — they test whether you understand what each line does computationally. Focus on the logic of forward and backward passes.

8.1.3 Study Resources

Practice questions are available in the uploaded PPT and the dedicated numerical questions PDF (about 80 pages). These are enough — do not go to other sources. Go through all lab-based discussions, especially Webinar 1 and Webinar 2. Understand the code step by step. Connect your machine learning course with the deep learning course wherever there is syllabus overlap (e.g., evaluation metrics, regularization).

8.1.4 Disclaimer on Old Question Papers

The old question paper was discussed during this session. Some parts belong to the previous syllabus and are not examinable. Specifically:

Question 2, part C (designing perceptron counts using a single hidden layer for complex logic units) is from a previous textbook and is NOT part of the current syllabus.

Pitfall: Do not study old question papers blindly. Confirm which parts are still in the current syllabus. Question 2 Part C from the old paper is explicitly excluded — studying it wastes time on content that will not appear.

8.1.5 Exam Writing Tips

Write all assumptions explicitly in your answers. Marks depend on proper justification.
Tabulate results where possible — tables are easier to grade than paragraphs.
For theoretical answers, use bulleted key terms rather than long paragraphs.
You may have the option to both upload a scanned paper and type answers. Check the exam instructions before you begin.

The exam is a 30-mark paper scaled to 100. Spend your preparation time where the marks are: numerical problems (~40% of weight), code-based questions (~15-20%), and theoretical justifications (~40%). Study the provided PPT and numerical PDF — those are your complete resource. The next section walks through a concrete numerical problem: the NAND-gate perceptron, which is a classic exam question pattern.

8.2 Perceptron Learning Algorithm for the NAND Gate

Hook: Can a single artificial neuron learn to behave like a logic gate? The NAND gate is special — it is functionally complete. Every other Boolean function can be built from NAND gates alone. If a perceptron can learn NAND, you have proven that a neural network can, in principle, learn any logical computation.

Intuition + Analogy: A perceptron is like a judge who weighs evidence. Each input (X1, X2) is a piece of evidence. The weight (W1, W2) is how much the judge trusts it. The bias (W0) is the judge's gut feeling — a baseline leaning toward one class or the other. The judge sums up the weighted evidence plus the baseline. If the total is positive, the judge rules +1. If negative, -1. Learning means adjusting the trust weights and the gut feeling until the judge gets every case right.

Here the judge starts with no opinion (all weights zero). After seeing each training case, the judge adjusts. When the ruling is wrong, shift weights in the direction that would have helped. The magnitude of the shift is proportional to how wrong the ruling was. This is exactly the perceptron update rule: $Δ W_{i} = (T - Y) \cdot X_{i}$ .

Where the analogy breaks: a real judge considers context and precedence. The perceptron only cares about a linear weighted sum. If the data points cannot be separated by a straight line (or plane), no amount of weight adjustment will work.

8.2.1 Definition and Problem Setup

The problem asks you to implement the perceptron learning algorithm for a NAND gate. Use bipolar representations: labels are +1 and -1. You are given:

Features: X1 and X2, with a bias X0 = 1 for all four input combinations.
Target values (T): The NAND truth table in bipolar form.
Initial weights: All weights (W0, W1, W2) initialized to 0.
Learning rate $η = 1$ .
Activation function: $H = sign (Z)$ , where $Z$ is the pre-activation sum.
Weight update rule: Use the rule provided in the question — it is the standard perceptron rule.

The notation in the question uses $T$ for the target (actual) value and $Y$ (or $Y^$ / $H$ ) for the predicted value.

Perceptron update rule (standard form): For each weight $W_{i}$ connecting input $X_{i}$ :

Δ W_{i} = η \cdot (T - Y) \cdot X_{i}

W_{i}^{new} = W_{i}^{old} + Δ W_{i}

Where:

$T$ — target label (+1 or -1)
$Y$ — predicted label (+1 or -1)
$X_{i}$ — the i-th input value
$η$ — learning rate

The term $(T - Y)$ is the error signal. It can take three values:

$T - Y = 0$ : prediction was correct. No weight change.
$T - Y = + 2$ : predicted -1, should be +1. Increase weights where input is positive.
$T - Y = - 2$ : predicted +1, should be -1. Decrease weights where input is positive.

8.2.2 Symbol Registry — Perceptron Learning

Symbol	Meaning	Domain
$X_{0}$	Bias input	Always 1
$X_{1}, X_{2}$	Input features	${- 1, + 1}$
$W_{0}, W_{1}, W_{2}$	Weights (including bias)	$R$
$Z = W_{0} X_{0} + W_{1} X_{1} + W_{2} X_{2}$	Pre-activation sum	$R$
$sign (Z)$	Activation: +1 if $Z \geq 0$ , -1 if $Z < 0$	${- 1, + 1}$
$T$	Target (actual) label	${- 1, + 1}$
$Y$ (or $H$ )	Predicted label	${- 1, + 1}$
$Δ W_{i} = (T - Y) \cdot X_{i}$	Weight update for $W_{i}$	$R$
$η = 1$	Learning rate	Scalar

8.2.3 NAND Gate Truth Table (Bipolar)

The four input combinations and their labels:

Instance	$X_{1}$	$X_{2}$	$X_{0}$ (bias)	$T$ (NAND)
1	-1	-1	1	+1
2	-1	+1	1	+1
3	+1	-1	1	+1
4	+1	+1	1	-1

NAND returns +1 for three of the four input combinations and -1 only when both inputs are +1.

8.2.4 Worked Computation — Stochastic Update (Demonstration with sign(0) = -1)

The session used an intentionally incorrect assumption that $sign (0) = - 1$ . This was purely to walk through the update procedure step by step. The correct convention is $sign (0) = + 1$ . The computation below follows the session's demonstration. After this, we provide the correct computation with $sign (0) = + 1$ .

Initial state: $W_{0} = 0$ , $W_{1} = 0$ , $W_{2} = 0$ , $η = 1$ .

Instance 1 ( $X_{1} = - 1$ , $X_{2} = - 1$ , $T = + 1$ ):

Z = 0 \cdot 1 + 0 \cdot (- 1) + 0 \cdot (- 1) = 0

H = sign (0) = - 1 (incorrect — should be +1)

Δ W_{0} & = (T - Y) \cdot X_{0} = (+ 1 - (- 1)) \cdot 1 = 2 \cdot 1 = + 2 Δ W_{1} & = (T - Y) \cdot X_{1} = (+ 1 - (- 1)) \cdot (- 1) = 2 \cdot (- 1) = - 2 Δ W_{2} & = (T - Y) \cdot X_{2} = (+ 1 - (- 1)) \cdot (- 1) = 2 \cdot (- 1) = - 2

After update: $W_{0} = 2$ , $W_{1} = - 2$ , $W_{2} = - 2$

Instance 2 ( $X_{1} = - 1$ , $X_{2} = + 1$ , $T = + 1$ ), using $[2, - 2, - 2]$ :

Z = 2 \cdot 1 + (- 2) \cdot (- 1) + (- 2) \cdot (+ 1) = 2 + 2 - 2 = 2

H = sign (2) = + 1

Δ W_{0} = (+ 1 - (+ 1)) \cdot 1 = 0, Δ W_{1} = 0, Δ W_{2} = 0

No weight change — prediction was correct.

Instance 3 ( $X_{1} = + 1$ , $X_{2} = - 1$ , $T = + 1$ ), using $[2, - 2, - 2]$ :

Z = 2 \cdot 1 + (- 2) \cdot (+ 1) + (- 2) \cdot (- 1) = 2 - 2 + 2 = 2

H = sign (2) = + 1, Δ = 0

No weight change.

Instance 4 ( $X_{1} = + 1$ , $X_{2} = + 1$ , $T = - 1$ ), using $[2, - 2, - 2]$ :

Z = 2 \cdot 1 + (- 2) \cdot (+ 1) + (- 2) \cdot (+ 1) = 2 - 2 - 2 = - 2

H = sign (- 2) = - 1, Δ W_{i} = 0

No weight change — prediction was correct.

Final weights (under incorrect assumption): $W_{0} = + 2$ , $W_{1} = - 2$ , $W_{2} = - 2$

Correct rework with $sign (0) = + 1$ :

Instance 1 ( $X_{1} = - 1$ , $X_{2} = - 1$ , $T = + 1$ ), weights all zero:

Z = 0, H = sign (0) = + 1

Prediction matches target: $H = T = + 1$ . No weight change. All weights stay at 0.

Instance 2 ( $X_{1} = - 1$ , $X_{2} = + 1$ , $T = + 1$ ), weights all zero:

Z = 0, H = sign (0) = + 1 = T \Rightarrow Δ = 0

Instance 3 ( $X_{1} = + 1$ , $X_{2} = - 1$ , $T = + 1$ ), weights all zero:

Z = 0, H = sign (0) = + 1 = T \Rightarrow Δ = 0

Instance 4 ( $X_{1} = + 1$ , $X_{2} = + 1$ , $T = - 1$ ), weights all zero:

Z = 0, H = sign (0) = + 1 \neq = T = - 1

Δ W_{0} & = (- 1 - (+ 1)) \cdot 1 = - 2 \cdot 1 = - 2 Δ W_{1} & = (- 1 - (+ 1)) \cdot (+ 1) = - 2 \cdot 1 = - 2 Δ W_{2} & = (- 1 - (+ 1)) \cdot (+ 1) = - 2 \cdot 1 = - 2

After epoch 1: $W_{0} = - 2$ , $W_{1} = - 2$ , $W_{2} = - 2$

These weights also separate NAND correctly. Both sign(0) assumptions lead to valid solutions because NAND is linearly separable. The paths differ but both converge.

8.2.5 Batch Update Approach (Alternative)

If the question asks you to process all four instances first, collect all delta changes, average them, and apply one final update per weight:

Δ W_{i}^{batch} = \frac{1}{m} k = 1 \sum m Δ W_{i}^{(k)}

W_{i}^{new} = W_{i}^{old} + η \cdot Δ W_{i}^{batch}

Where $m = 4$ is the number of training instances.

Under the incorrect $sign (0) = - 1$ assumption:

All $H = - 1$ since $Z = 0$ for all instances initially.
$Δ W_{0}$ values: $[2, 2, 2, 0]$ , average = $6/4 = 1.5$
$Δ W_{1}$ values: $[- 2, 2, - 2, 0]$ , average = $- 2/4 = - 0.5$
$Δ W_{2}$ values: $[- 2, - 2, 2, 0]$ , average = $- 2/4 = - 0.5$

Batch update: $W_{0} = 1.5$ , $W_{1} = - 0.5$ , $W_{2} = - 0.5$

Scope — when to use which update mode: The professor stated that the question design expects stochastic (sequential) updates. This is the default. Unless the question paper explicitly says "batch gradient descent" or "average all changes." In stochastic mode, instance 1's updated weights are used for instance 2, and so on. In batch mode, all four instances are evaluated using the same starting weights. Both approaches are valid. Which one to use depends on the question's wording. Always write your assumption explicitly at the top of your answer.

8.2.6 Verifying the Final Weights

To verify that the perceptron correctly classifies all inputs with $[W_{0} = + 2, W_{1} = - 2, W_{2} = - 2]$ :

Instance 1: Z & = 2 + (- 2) (- 1) + (- 2) (- 1) = 2 + 2 + 2 = 6 \to sign (6) = + 1 = T ✓ Instance 2: Z & = 2 + (- 2) (- 1) + (- 2) (+ 1) = 2 + 2 - 2 = 2 \to sign (2) = + 1 = T ✓ Instance 3: Z & = 2 + (- 2) (+ 1) + (- 2) (- 1) = 2 - 2 + 2 = 2 \to sign (2) = + 1 = T ✓ Instance 4: Z & = 2 + (- 2) (+ 1) + (- 2) (+ 1) = 2 - 2 - 2 = - 2 \to sign (- 2) = - 1 = T ✓

All four instances are correctly classified. The perceptron has perfectly learned the NAND function.

8.2.7 Visual Intuition

Picture a 2D plot with $X_{1}$ on the horizontal axis and $X_{2}$ on the vertical axis. The four training points sit at the four corners of a square: (-1,-1), (-1,+1), (+1,-1), (+1,+1). The three points labeled +1 sit at the top-left, bottom-left, and bottom-right corners. The single point labeled -1 sits at the top-right corner (+1,+1).

The perceptron draws a straight line through this plane. The decision boundary for weights $[2, - 2, - 2]$ is:

2 - 2 X_{1} - 2 X_{2} = 0 \Rightarrow X_{1} + X_{2} = 1

This line passes between the top-right corner (where $X_{1} + X_{2} = 2 > 1$ , classified as -1) and the other three corners (where $X_{1} + X_{2} \leq 1$ , classified as +1). The takeaway: a single perceptron finds one straight line that cleanly separates the +1 points from the -1 point — which is possible here because NAND is linearly separable.

8.2.8 Pitfalls

The $sign (0)$ trap: In the exam, use $sign (0) = + 1$ unless the question explicitly specifies otherwise. The convention matters whenever initial weights are zero. The session used $sign (0) = - 1$ as a deliberate illustration — do not blindly copy it.

Stochastic vs. batch confusion: Stochastic update propagates weight changes immediately. Batch update averages all deltas first. Mixing them gives wrong answers. Read the question wording carefully and state your assumption.

Forgetting the bias: The bias $X_{0} = 1$ applies to every instance. Its weight $W_{0}$ must be updated along with the other weights using the same rule. Skipping the bias update means the decision boundary is forced through the origin.

Wrong error direction: The update is $Δ W = (T - Y) \cdot X$ , not $(Y - T) \cdot X$ . If you swap the subtraction, the weights move in the wrong direction. Double-check: when $T = + 1$ and $Y = - 1$ , you want the weight to increase along the positive direction of $X$ .

8.2.9 Student Questions and Answers

Q: When the question asks to update weights after one full epoch, should we update after every example or batch all four and update once?

A: Unless the question says "batch gradient descent" or "average all changes," use stochastic update — take the updated weights from instance 1 and use them for instance 2, and so on. If the question says "update weights based on all four combinations once," average all deltas and apply one final update. Always write your assumption explicitly at the top of your answer.

Q: Should the bias be considered for all four combinations?

A: Yes. The bias $X_{0} = 1$ is assumed for all four input combinations. The question explicitly states this.

Q: What is the correct sign for $Z = 0$ — is it +1 or -1?

A: The standard convention for the sign function is $sign (0) = + 1$ . The session used $sign (0) = - 1$ as an intentional incorrect assumption for illustration. Use $sign (0) = + 1$ in your actual exam unless the question specifies otherwise.

The perceptron learns NAND by adjusting three weights. One per input. Plus a bias. It uses a simple rule: when the prediction is wrong, shift weights in the direction that reduces the error. The shift is scaled by the input value. After one epoch (four instances seen once each), the weights correctly classify all NAND cases. The key exam takeaway: know the update rule $Δ W_{i} = (T - Y) \cdot X_{i}$ by heart, always state your assumption about $sign (0)$ and update mode, and verify your final weights against the truth table. Next we explore what happens when a perceptron cannot find a separating line — the case of linear separability.

8.2.10 Real-World & Domain Connection

Perceptron learning on logic gates is the conceptual foundation of all neural network training. The NAND gate is functionally complete — any Boolean circuit can be constructed from NAND gates alone. By proving a perceptron can learn NAND, you prove that neurons can, in principle, compose into universal Boolean computers. In practice, perceptron-based systems were used in early pattern recognition applications like the 1958 Mark I Perceptron for image classification. Modern descendants appear in binary classification everywhere: spam filters (spam +1, not-spam -1), credit approval systems (approve +1, deny -1), and medical screening (disease detected +1, clear -1). The weight update rule you just traced is the same mechanism that trains the final classification layer in every deep network today — just with a different activation function.

8.3 Linear Separability and Multi-Layer Perceptrons

Hook: NAND is easy. A single straight line separates the +1 points from the -1 point. But what if the red and blue points form concentric circles? Like a bullseye? No single straight line can split them. Does that mean a neural network fails? It would — if it were only one layer deep.

Intuition + Analogy: Think of a single-layer perceptron as a fence builder who can only put up one straight fence. If the sheep and goats graze on opposite sides of a line, one fence works. But if the sheep are in the middle and the goats circle them, one straight fence always leaves some goats with the sheep. An MLP with hidden layers is like a team of fence builders. Each builder puts up their own straight fence. The first layer builds several fences. They slice the field into wedge-shaped pens. The second layer combines those pens. It says: "this pen, plus that pen, minus the open area = goat zone." Together the team can fence off any shape — even a circle. They do it by composing many straight edges. This is exactly how ReLU networks approximate curved boundaries. They tile the space with piecewise-linear regions.

Where the analogy breaks: real fence panels don't bend. But ReLU networks can also use activation functions like tanh to create truly smooth curves, not just angular approximations.

Symbol Registry — 8.3:

Symbol	Meaning	Domain
$w$	Weight vector	$R^{d}$
$b$	Bias	$R$
$H$	Hidden layer size	Integer
$σ (\cdot)$	Activation function	$R \to R$
ReLU $(z)$	$max (0, z)$	$R \to [0, \infty)$

8.3.1 Definition and Problem Setup

A binary classifier using a perceptron classifies 2D data points into two classes (red and blue). The problem states that the red and blue points cannot be separated by a straight line in 2D space. The data is not linearly separable. Multiple curved boundaries could separate the classes successfully.

Linear separability — a formal definition. Two sets of points $A$ and $B$ in $R^{d}$ are linearly separable if there exists a vector $w \in R^{d}$ and a scalar $b \in R$ such that:

w^{T} x + b > 0 \forall x \in A

w^{T} x + b < 0 \forall x \in B

In 2D, $w^{T} x + b = 0$ is the equation of a straight line. If one class lies entirely on one side and the other class entirely on the other side, the data is linearly separable. Otherwise, it is not.

8.3.2 Why a Single-Layer Perceptron Fails

A single-layer perceptron with two input nodes ( $X_{1}, X_{2}$ ) and one output node can only create a linear decision boundary. It computes:

Z = W_{0} \cdot 1 + W_{1} \cdot X_{1} + W_{2} \cdot X_{2}

Setting $Z = 0$ gives the decision boundary: $W_{1} X_{1} + W_{2} X_{2} + W_{0} = 0$ — a straight line. When the data is not linearly separable (e.g., blue points surround red points in a circle), this linear boundary always leaves misclassifications. No single straight line can perfectly separate nested or curved distributions.

8.3.3 How a Multi-Layer Perceptron Overcomes This

A multi-layer perceptron (MLP) with at least one hidden layer overcomes this limitation in two ways.

Hidden layers compose multiple linear boundaries: Each neuron in the hidden layer computes its own linear combination. This is $Z_{k} = W_{k 0} + W_{k 1} X_{1} + W_{k 2} X_{2}$ . Each neuron draws its own straight line. With $H$ hidden neurons, you get $H$ lines. The output layer combines these $H$ lines using learned weights. Together they form piecewise-linear regions that approximate any curved boundary. The hidden layer chops the input space into convex regions. The output layer labels each region.

Nonlinear activation functions introduce curvature: Without nonlinearity, stacking linear layers is pointless. The composition of linear functions is still linear. A deep network would just be one giant linear layer. Applying a nonlinear activation function $σ (\cdot)$ after each hidden layer breaks this chain. With ReLU, hidden neurons create sharp corners at their decision boundaries. With tanh or sigmoid, they create smooth transitions. These nonlinearities give the network the power to draw boundaries that are not straight.

Why two layers suffice (Universal Approximation): A single hidden layer with enough neurons and a nonlinear activation can approximate any continuous function on a compact domain. It can do this to arbitrary precision. Two hidden layers can handle discontinuous functions. In practice, depth (more layers) often achieves the same accuracy with fewer total neurons. Each layer can reuse features learned by earlier layers.

8.3.4 Visual Intuition

Picture a 2D plot. The red points lie inside a circular region centered at the origin. The blue points occupy the surrounding ring. A single perceptron can only draw one line, splitting the plane into two half-planes. No matter how you angle that line, some blue points end up on the red side or vice versa.

Now add one hidden layer with four ReLU neurons. Each neuron is active (output > 0) on one side of its own linear boundary and inactive (output = 0) on the other. Four lines can form a rough diamond around the red circle. The output neuron gives a positive weight to the region where all four hidden neurons fire inside the diamond (red) and a negative weight everywhere else (blue). More hidden neurons → more lines → the polygon approximates the circle more closely.

The takeaway: a single perceptron sees the world as one cut. An MLP sees it as many cuts that combine into flexible shapes.

8.3.5 Assumptions & Scope

Assumption — data is stationary: The MLP learns from a fixed training set. If the data distribution shifts over time (concept drift), the learned boundary becomes stale.

If the assumption fails: The network may show high training accuracy but poor performance on new data from the shifted distribution. This is distinct from overfitting — it is a distribution mismatch.

Scope — expressiveness limits: An MLP with one hidden layer can approximate any continuous function, but approximating a highly oscillatory function (e.g., $sin (100 x)$ ) may need an impractically large number of neurons. Depth helps: more layers with fewer neurons per layer can model such functions more efficiently. Deep networks learn hierarchical features — early layers detect edges, middle layers detect shapes, later layers detect objects — which gives them an efficiency advantage over shallow but wide networks.

8.3.6 Comparison — Single-Layer vs Multi-Layer Perceptron

Property	Single-Layer Perceptron	Multi-Layer Perceptron
Decision boundary shape	One straight line (hyperplane)	Piecewise-linear or smooth curves
Handles XOR / circular data?	No	Yes
Number of parameters	$d + 1$ (input dims + bias)	$(d \cdot H) + H + (H \cdot K) + K$ for one hidden layer of size $H$
Activation needed at hidden layer?	None (none exists)	Nonlinear (ReLU, tanh, sigmoid)
Convergence guarantee	Yes — if data is linearly separable	No guarantee (non-convex optimization)
When to pick	Simple, known linear problems	Any problem where linear separation fails

8.3.7 Key Terms for Answer Writing

Hidden layer extracts hierarchical features.
Multiple transformations compose into complex boundaries.
Nonlinear activation functions (ReLU, tanh, sigmoid) introduce nonlinearity between layers.
The focus should be on construction of the deep neural network and its components, not on feature engineering.

Pitfalls:

"More layers always help" — Not true. Adding layers to an already-solved problem increases training time and can lead to vanishing gradients. Use the simplest architecture that solves the problem.

Forgetting the nonlinearity: Stacking linear layers with no activation between them is equivalent to a single linear layer: $W_{2} (W_{1} x) = (W_{2} W_{1}) x = W_{eff} \cdot x$ . No matter how deep, the network remains linear. The activation function is what gives MLPs their power.

Confusing linear separability with learnability: Even if a dataset is linearly separable, a poorly trained perceptron might not find the separating line (especially with a bad learning rate). Linear separability is a property of the data, not a guarantee of convergence speed.

Assuming one hidden neuron = one feature: A hidden neuron does not learn "a feature" in isolation. All hidden neurons interact — the output layer combines them. Thinking of neurons as independent feature detectors is misleading at best.

A single-layer perceptron can only draw straight lines. When the data demands curves — circular clusters, interleaved spirals, XOR patterns — you must add hidden layers with nonlinear activations. The hidden layers chop the space into convex regions. The output layer labels the regions. Together they can approximate any decision boundary. Next: we apply this exact idea to a medical diagnosis scenario — designing output layers for binary vs multi-class tasks.

8.3.8 Real-World & Domain Connection

The XOR problem — the simplest non-linearly-separable dataset — was the historical motivation for multi-layer networks. In their influential 1969 book, Minsky and Papert proved that single-layer perceptrons cannot solve XOR. This dampened neural network research for over a decade. The invention of backpropagation in the 1980s revived the field. It made training multi-layer networks possible.

Today, MLPs with hidden layers solve tasks where relationships between inputs are inherently nonlinear. Medical diagnosis involves symptoms that interact in complex ways. Fever + cough is different from fever alone or cough alone. Credit risk assessment depends on interaction between income, debt, and employment history. Handwritten digit recognition involves pixel patterns that form curves and loops. These are not separable by straight lines.

The core insight — composing many simple linear units with nonlinearities yields expressive models — is the conceptual bedrock beneath every deep neural network architecture. This applies from CNNs to transformers.

8.4 Medical Diagnosis Models — Binary vs Multi-Class Classification Design

Hook: You are building an AI that diagnoses diseases. The architecture you choose determines whether your model makes sense at all. One output neuron or five? Sigmoid or softmax? Binary cross-entropy or categorical? A wrong choice at the output layer can make the entire network un-trainable. How do you decide?

Intuition + Analogy: Think of the output layer as the answer sheet format. In a binary (yes/no) test, each question gets one checkbox. Checked means yes. Unchecked means no. That is sigmoid + binary cross-entropy. In a multiple-choice test with five options, each question gets exactly one filled bubble. You cannot fill two bubbles for the same question. That is softmax + categorical cross-entropy.

Sigmoid answers "is this COVID?" with a single number from 0 to 1. Softmax answers "which disease?" with five numbers that add up to 1. If you try to use sigmoid for the five-disease problem, you get five independent yes/no answers. For example: "30% COVID AND 80% flu AND 15% pneumonia." This makes no clinical sense. The probabilities do not add to 1. The model never learns that the classes are mutually exclusive.

Where the analogy breaks: softmax forces exactly one bubble. This is correct when an input belongs to exactly one class. But in multi-label classification — a patient can have multiple diseases — sigmoid per class is the right choice. You want independent probabilities.

8.4.1 Problem Setup

A medical diagnosis company is developing two AI models:

Model A: Binary classification task (e.g., COVID-19 positive vs. negative).
Model B: Multi-class classification task with five disease categories based on symptoms.

8.4.2 Shallow vs Deep Network Justification

The question asks you to justify the choice of shallow vs. deep network. There is no information about the complexity of the data. It is not stated whether the data is linearly separable, nor whether features directly explain the disease patterns. Both answers can be acceptable if properly justified:

Deep network for both: Justified because deep networks can capture hierarchical patterns. They model complex relationships even when the data complexity is unknown. This is always a safe choice.
Shallow network for Model A: Justified if you assume COVID-19 features are linearly separable and fully explanatory. State this assumption explicitly.

Marks are awarded based on the justification, not the choice itself.

8.4.3 Output Layer Design

Model A — Binary classification output layer:

Number of output nodes: 1. One neuron is enough.
Activation: Sigmoid. $σ (z) = \frac{1}{1 + e ^{- z}}$ . Maps $R \to (0, 1)$ .
Interpretation: The output $y^$ is the probability the input belongs to the positive class. $1 - y^$ is the probability of the negative class.
Decision rule: Classify as positive if $y^\geq 0.5$ . Otherwise negative. Example: $y^= 0.87$ means COVID positive. $y^= 0.3$ means COVID negative.

Model B — Multi-class (5 categories) output layer:

Number of output nodes: 5. One per class.
Activation: Softmax. $softmax (z_{c}) = \frac{e ^{z_{c}}}{\sum _{j = 1}^{C} e ^{z_{j}}}$ . Maps $R^{C} \to [0, 1]^{C}$ .
Interpretation: Each output $y^_{c}$ is the probability the input belongs to class $c$ . The five probabilities sum to 1.
True label format: One-hot encoded vectors of size 5. Example: $[0, 1, 0, 0, 0]$ means the instance belongs to class 2.
Decision rule: Pick the class with the highest predicted probability.

8.4.4 Symbol Registry — Loss Functions and Activation

Symbol	Meaning	Domain
$y$	True label	$y \in {0, 1}$ (binary) or $y \in {0, 1}^{C}$ (one-hot)
$y^$	Predicted probability	$[0, 1]$ (binary) or $[0, 1]^{C}$ (multi-class)
$C$	Number of classes	Integer, e.g., 5
$y_{c}$	True label for class $c$	${0, 1}$
$y^_{c}$	Predicted probability for class $c$	$[0, 1]$
$z_{c}$	Raw logit for class $c$ (pre-softmax)	$R$
$σ (z)$	Sigmoid: $\frac{1}{1 + e ^{- z}}$	$R \to (0, 1)$
$softmax (z_{c})$	$\frac{e ^{z_{c}}}{\sum _{j = 1}^{C} e ^{z_{j}}}$	$R^{C} \to [0, 1]^{C}$

8.4.5 Loss Function Selection

Model A — Binary Cross-Entropy Loss:

L = - [y \cdot lo g (y^) + (1 - y) \cdot lo g (1 - y^)]

Where $y \in {0, 1}$ and $y^\in (0, 1)$ . This formula contains two terms. Exactly one term is active for any given instance:

When $y = 1$ : $L = - lo g (y^)$ → penalizes $y^$ far from 1
When $y = 0$ : $L = - lo g (1 - y^)$ → penalizes $y^$ far from 0

Worked intuition: If the true label is 1 and the model predicts $y^= 0.99$ , the loss is $- lo g (0.99) \approx 0.01$ (very small, good prediction). If the model predicts $y^= 0.01$ , the loss is $- lo g (0.01) \approx 4.6$ (very large, bad prediction). The log stretches small probabilities into large penalties, forcing the model to be confident about correct answers and uncertain about wrong ones.

Model B — Categorical Cross-Entropy Loss:

L = - c = 1 \sum C y_{c} \cdot lo g (y^_{c})

Where $y_{c} \in {0, 1}$ (one-hot) and $\sum_{c} y^_{c} = 1$ . For each instance, only the term for the true class contributes — all other $y_{c} = 0$ .

When the true class is $k$ and the model predicts $y^_{k} = 0.95$ : $L = - lo g (0.95) \approx 0.05$
When the true class is $k$ and the model predicts $y^_{k} = 0.10$ : $L = - lo g (0.10) \approx 2.30$

8.4.6 Role of Softmax in Model B

Softmax converts the raw output logits of the five output nodes into a normalized probability distribution:

y^_{c} = softmax (z_{c}) = \frac{e ^{z_{c}}}{\sum _{j = 1}^{C} e ^{z_{j}}}

Three properties softmax guarantees:

Non-negativity: $e^{z_{c}} > 0$ for all real $z_{c}$ , so each $y^_{c} \in (0, 1)$
Normalization: The denominator is the sum of all exponentials, so $\sum_{c = 1}^{C} y^_{c} = 1$
Order preservation: If $z_{a} > z_{b}$ , then $y^_{a} > y^_{b}$ (softmax preserves the ranking of logits)

The sigmoid-softmax equivalence (C = 2 case): For binary classification with two output nodes and softmax, the probability for class 1 is:

y^_{1} = \frac{e ^{z_{1}}}{e ^{z_{1}} + e ^{z_{2}}} = \frac{1}{1 + e ^{- (z_{1} - z_{2})}} = σ (z_{1} - z_{2})

This shows that two-output softmax is equivalent to sigmoid applied to $(z_{1} - z_{2})$ . For $C = 2$ , one sigmoid output is enough — the second probability is $1 - y^_{1}$ . Two-output softmax is mathematically equivalent but uses an extra parameter.

8.4.7 Can Softmax Be Used in Model A?

Technically, yes — you can apply softmax to a binary classification task. When $C = 2$ , softmax produces two outputs that sum to 1. But it is unnecessary. Sigmoid on a single output node is simpler and computationally cheaper. Softmax computes exponentials for two outputs, sums them, and divides — sigmoid computes one exponential and one division. Both produce valid probabilities. Sigmoid is the more efficient choice for binary tasks.

8.4.8 Visual Intuition

Picture a plot of the sigmoid function: the x-axis is the logit $z$ , ranging from -5 to +5. The y-axis is the predicted probability $y^$ , ranging from 0 to 1. The curve is S-shaped (which is why it is called S-igmoid). It is nearly flat at 0 for $z ≪ 0$ . It is steepest near $z = 0$ with slope $σ^{'} (0) = 0.25$ . It is nearly flat at 1 for $z ≫ 0$ . The steep middle region is where the model is uncertain. Small changes in the logit cause large changes in probability.

For softmax with $C = 5$ , picture a 5-dimensional probability simplex (a 4D tetrahedron). The five outputs always lie on the simplex. It is a flat surface. Each coordinate is between 0 and 1. The sum is always 1. Training moves the output point around this simplex toward the vertex of the true class.

The takeaway: sigmoid models uncertainty on a curve from 0 to 1. Softmax models uncertainty on a simplex whose vertices are the classes.

8.4.9 Assumptions & Scope

Sigmoid for binary, softmax for multi-class: This pairing is not arbitrary. Categorical cross-entropy + softmax is the standard. The reason: the gradient of the combination simplifies beautifully:

\frac{\partial L}{\partial z _{c}} = y^_{c} - y_{c}

This elegant gradient is the reason you always see softmax + categorical cross-entropy paired in frameworks. Using mean squared error with softmax leads to vanishing gradients when predictions are confident. Using sigmoid + categorical cross-entropy produces independent probabilities that do not sum to 1.

When this breaks: If your multi-class problem allows multiple correct labels per instance (multi-label classification), use sigmoid per output node with binary cross-entropy. Softmax is wrong here because it forces probabilities to sum to 1, which implies mutual exclusivity.

8.4.10 Pitfalls

Using MSE instead of cross-entropy for classification: MSE + sigmoid produces vanishing gradients when the prediction is confident but wrong. The cross-entropy gradient does not vanish — it stays proportional to the error, making training faster and more reliable.

Confusing one-hot encoding with label encoding: One-hot: $[0, 1, 0, 0, 0]$ for class 2. Label encoding: just the integer 2. Categorical cross-entropy requires one-hot encoded targets. Using label encoding directly gives wrong gradients.

Forgetting that softmax is shift-invariant: $softmax (z + c)$ = $softmax (z)$ for any constant $c$ . During computation, subtracting $max (z_{c})$ from all logits before exponentiating prevents numerical overflow: $softmax (z_{c}) = \frac{e ^{z_{c} - m a x (z)}}{\sum _{j} e ^{z_{j} - m a x (z)}}$ . This is exactly what frameworks do internally.

Applying sigmoid then softmax: Don't. The output of sigmoid is already a probability in (0,1). Feeding it through softmax produces nonsense — the output of one activation should go into the loss, not into another activation.

8.4.11 Student Questions and Answers

Q: What is the difference between a shallow neural network and a deep neural network?

A: A shallow network typically has zero or one hidden layer. A deep neural network has multiple hidden layers. The definition varies by author — some say more than one hidden layer is deep, others say three or more layers. The prescribed textbook defines deep as three or more layers. As depth increases, the network extracts increasingly hierarchical features: early layers detect basic patterns, middle layers compose them into intermediate features, later layers compose those into complex, abstract patterns.

Q: For choosing shallow vs deep, what justification should we write?

A: For Model A (COVID detection), if the data is linearly separable, a shallow network works — state this assumption explicitly. If you cannot make such an assumption, use a deep network. For Model B (5 disease categories), the data is likely non-linearly separable. A deep network is preferable. Always write assumptions explicitly — marks depend on the justification.

The output layer is the contract between your model and the loss function. Binary problems get one sigmoid neuron + binary cross-entropy. Multi-class problems get $C$ softmax neurons + categorical cross-entropy. The choice is not stylistic — wrong pairings produce untrainable networks. Next, we move from designing output layers to computing gradients through computational graphs, which is how these loss functions actually update the weights.

8.4.12 Real-World & Domain Connection

The sigmoid + binary CE and softmax + categorical CE pairings power nearly every classification system deployed today. Medical imaging systems use softmax to classify chest X-rays into multiple disease categories simultaneously. Email spam classifiers use sigmoid for a single spam/not-spam decision. Autonomous vehicles use softmax to classify detected objects into pedestrian, vehicle, cyclist, and sign categories — getting the output layer right is a safety requirement.

In the research literature, this pairing appears under the name "softmax cross-entropy loss" or "log loss" and is the recommended default for classification in frameworks like TensorFlow (tf.keras.losses.CategoricalCrossentropy) and PyTorch (nn.CrossEntropyLoss, which internally applies log-softmax). The combination's computational efficiency and well-behaved gradients make it the undisputed standard — there is almost never a reason to use MSE for classification in modern practice.

8.5 Computational Graphs and Gradient Computation

Hook: You have a function that takes sensor readings. It multiplies them by a weight matrix. It squares each result. Then it sums them. Every deep learning framework computes exactly this kind of thing. It just scales to billions of operations. How do you trace the gradient of the final output back to every intermediate quantity? The answer: a computational graph.

Intuition + Analogy: A computational graph is like a flowchart of a recipe. Each box is an operation (multiply, square, add). The arrows show data flowing forward — inputs go in, the final dish comes out. Backpropagation flows the same graph in reverse. Each box receives a signal from downstream: "change your output by this much to reduce the error." Then it passes an adjusted signal upstream: "change your inputs by that much." The chain rule is the mathematical rule that determines how the signal splits and scales at each box.

Think of gradient computation as passing a bucket of water backward through the graph. At a sum node, the bucket is duplicated. Both inputs get the full amount. At a product node, the bucket is cross-multiplied. Each input gets the other input's value times the incoming amount. At a square node, the bucket is multiplied by $2 z$ . Every operation has its own local gradient pattern.

8.5.1 Problem Setup

An industrial AI model predicts energy consumption from sensor readings using a mathematical function. You are given:

Two sensor readings: $X_{1} = 0.2$ , $X_{2} = 0.4$
A weight matrix $W \in R^{2 \times 2}$ with values $W = [0.1&0.5 - 0.3&0.8]$
The function: compute $W \cdot X$ (matrix-vector product), square each element of the result, then sum the squares to produce the final output $F$ .

No bias is mentioned. No hidden layers or neural network terminology are provided. This is a purely mathematical function, not a neural network. You are asked to draw the computational graph, perform one forward pass, and compute gradients.

8.5.2 Symbol Registry — Computational Graph

Symbol	Meaning	Shape / Domain
$X = [X_{1}, X_{2}]^{T}$	Input feature vector	$R^{2 \times 1}$
$W$	Weight matrix	$R^{2 \times 2}$
$Z = W \cdot X$	Linear transformation output	$R^{2 \times 1}$
$Q_{i} = Z_{i}^{2}$	Squared component $i$	Scalar
$F = Q_{1} + Q_{2}$	Final output	Scalar
$\frac{\partial F}{\partial W}$	Gradient w.r.t. $W$	$R^{2 \times 2}$
$\frac{\partial F}{\partial X}$	Gradient w.r.t. $X$	$R^{2}$

The function, written compactly:

F = i = 1 \sum 2 (j = 1 \sum 2 W_{ij} \cdot X_{j})^{2} = i = 1 \sum 2 Z_{i}^{2}

Where $Z_{i} = W_{i 1} X_{1} + W_{i 2} X_{2}$ for $i = 1, 2$ .

8.5.3 Computational Graph Construction

The graph has five types of nodes, flowing left to right:

X (2×1) ─┐
          ├── Multiply ──→ Z (2×1) ──→ Square ──→ Q (2×1) ──→ Sum ──→ F (scalar)
W (2×2) ─┘

Node-by-node detail:

Input nodes: $X = [0.2, 0.4]^{T}$ (2×1 vector)
Weight node: $W = [0.1&0.5 - 0.3&0.8]$ (2×2 matrix)
Multiply node: $Z = W \cdot X$
- $Z_{1} = W_{11} X_{1} + W_{12} X_{2} = 0.1 \cdot 0.2 + 0.5 \cdot 0.4$
- $Z_{2} = W_{21} X_{1} + W_{22} X_{2} = (- 0.3) \cdot 0.2 + 0.8 \cdot 0.4$
Square nodes: $Q_{1} = Z_{1}^{2}$ , $Q_{2} = Z_{2}^{2}$
Sum node: $F = Q_{1} + Q_{2} = Z_{1}^{2} + Z_{2}^{2}$

8.5.4 Forward Pass Computation

Step 1 — Compute $Z = W \cdot X$ :

Z_{1} & = 0.1 \cdot 0.2 + 0.5 \cdot 0.4 = 0.02 + 0.20 = 0.22 Z_{2} & = (- 0.3) \cdot 0.2 + 0.8 \cdot 0.4 = - 0.06 + 0.32 = 0.26

Step 2 — Square each component:

Q_{1} & = Z_{1}^{2} = (0.22)^{2} = 0.0484 Q_{2} & = Z_{2}^{2} = (0.26)^{2} = 0.0676

Step 3 — Sum:

F = Q_{1} + Q_{2} = 0.0484 + 0.0676 = 0.1160

Sense check: Both $Z_{1}$ and $Z_{2}$ are small positive numbers. Their squares should be even smaller. The sum should be a small positive number. $F = 0.116$ is reasonable.

8.5.5 Gradient Computation — Full Chain Rule

We now compute the gradient of $F$ with respect to every intermediate and input quantity using backpropagation (reverse-mode automatic differentiation).

:: :::key-concept Backpropagation step by step:

Since $F = Q_{1} + Q_{2}$ :

\frac{\partial F}{\partial Q _{1}} = 1, \frac{\partial F}{\partial Q _{2}} = 1

Since $Q_{i} = Z_{i}^{2}$ :

\frac{\partial Q _{1}}{\partial Z _{1}} = 2 Z_{1} = 2 (0.22) = 0.44, \frac{\partial Q _{2}}{\partial Z _{2}} = 2 Z_{2} = 2 (0.26) = 0.52

Since $Z_{1} = W_{11} X_{1} + W_{12} X_{2}$ and $Z_{2} = W_{21} X_{1} + W_{22} X_{2}$ , the partials are:

\frac{\partial Z _{1}}{\partial W _{11}} = X_{1} = 0.2, \frac{\partial Z _{1}}{\partial W _{12}} = X_{2} = 0.4, \frac{\partial Z _{1}}{\partial X _{1}} = W_{11} = 0.1, \frac{\partial Z _{1}}{\partial X _{2}} = W_{12} = 0.5

\frac{\partial Z _{2}}{\partial W _{21}} = X_{1} = 0.2, \frac{\partial Z _{2}}{\partial W _{22}} = X_{2} = 0.4, \frac{\partial Z _{2}}{\partial X _{1}} = W_{21} = - 0.3, \frac{\partial Z _{2}}{\partial X _{2}} = W_{22} = 0.8

Gradient w.r.t. $W$ — apply chain rule for each entry:

\frac{\partial F}{\partial W _{11}} & = \frac{\partial F}{\partial Q _{1}} \cdot \frac{\partial Q _{1}}{\partial Z _{1}} \cdot \frac{\partial Z _{1}}{\partial W _{11}} = 1 \cdot 2 Z_{1} \cdot X_{1} = 2 (0.22) (0.2) = 0.088 \frac{\partial F}{\partial W _{12}} & = \frac{\partial F}{\partial Q _{1}} \cdot \frac{\partial Q _{1}}{\partial Z _{1}} \cdot \frac{\partial Z _{1}}{\partial W _{12}} = 1 \cdot 2 Z_{1} \cdot X_{2} = 2 (0.22) (0.4) = 0.176 \frac{\partial F}{\partial W _{21}} & = \frac{\partial F}{\partial Q _{2}} \cdot \frac{\partial Q _{2}}{\partial Z _{2}} \cdot \frac{\partial Z _{2}}{\partial W _{21}} = 1 \cdot 2 Z_{2} \cdot X_{1} = 2 (0.26) (0.2) = 0.104 \frac{\partial F}{\partial W _{22}} & = \frac{\partial F}{\partial Q _{2}} \cdot \frac{\partial Q _{2}}{\partial Z _{2}} \cdot \frac{\partial Z _{2}}{\partial W _{22}} = 1 \cdot 2 Z_{2} \cdot X_{2} = 2 (0.26) (0.4) = 0.208

\frac{\partial F}{\partial W} = [0.088&0.176 0.104&0.208]

Resolving the $\partial F / \partial Q$ question: The professor's lecture notes mention $\partial F / \partial Q_{1} = 2 Q_{1}$ . This would be correct if $F = Q_{1}^{2} + Q_{2}^{2}$ . But the actual function is $F = Q_{1} + Q_{2}$ (where $Q_{i} = Z_{i}^{2}$ ). So the correct partial is $\partial F / \partial Q_{i} = 1$ . The factor of 2 appears instead when differentiating w.r.t. $Z_{i}$ : $\partial F / \partial Z_{i} = \partial F / \partial Q_{i} \cdot \partial Q_{i} / \partial Z_{i} = 1 \cdot 2 Z_{i} = 2 Z_{i}$ . Both derivations converge to the same final gradient w.r.t. $W$ . This is because the chain rule multiplies through correctly either way. In the exam, write $\partial F / \partial Q_{i} = 1$ for this function. The numerical result for $\partial F / \partial W$ is unaffected.

Gradient w.r.t. $X$ (for completeness only — features are not trained):

$X_{1}$ influences both $Z_{1}$ and $Z_{2}$ (through $W_{11}$ and $W_{21}$ ), so the gradient sums over both paths:

\frac{\partial F}{\partial X _{1}} & = \frac{\partial F}{\partial Z _{1}} \cdot \frac{\partial Z _{1}}{\partial X _{1}} + \frac{\partial F}{\partial Z _{2}} \cdot \frac{\partial Z _{2}}{\partial X _{1}} & = 2 Z_{1} \cdot W_{11} + 2 Z_{2} \cdot W_{21} & = 2 (0.22) (0.1) + 2 (0.26) (- 0.3) & = 0.044 - 0.156 = - 0.112

\frac{\partial F}{\partial X _{2}} & = 2 Z_{1} \cdot W_{12} + 2 Z_{2} \cdot W_{22} & = 2 (0.22) (0.5) + 2 (0.26) (0.8) & = 0.220 + 0.416 = 0.636

\frac{\partial F}{\partial X} = [- 0.112 0.636]

8.5.6 Visual Intuition

Picture the forward graph drawn left-to-right on paper:

Leftmost: two input circles labeled $X_{1} = 0.2$ and $X_{2} = 0.4$
A 2×2 box labeled $W$ feeds into the multiply node
Multiply outputs $Z_{1} = 0.22$ and $Z_{2} = 0.26$
Two square nodes: $Q_{1} = 0.0484$ and $Q_{2} = 0.0676$
Sum node: $F = 0.116$

For the backward pass, draw arrows running right-to-left:

Above the sum node, write $+ 1$ (the seed gradient $\partial F / \partial F = 1$ )
Sum node: passes $+ 1$ to both $Q_{1}$ and $Q_{2}$ (local gradient of sum is 1 for both inputs)
Square nodes: each receives $+ 1$ , multiplies by $2 Z_{i}$ locally, passes $2 (0.22) = 0.44$ backward to $Z_{1}$ and $2 (0.26) = 0.52$ backward to $Z_{2}$
Multiply node (the tricky one): $Z_{1}$ receives 0.44, distributes it backward — $W_{11}$ gets $0.44 \times X_{1} = 0.44 \times 0.2 = 0.088$ , $X_{1}$ gets $0.44 \times W_{11} + 0.52 \times W_{21} = 0.044 - 0.156 = - 0.112$

Backpropagation runs the graph backward. Each node's local gradient is multiplied by the incoming upstream signal. The chain rule chains these local multipliers all the way back to the inputs.

8.5.7 Assumptions & Scope

Scope — this is a gradient computation exercise, not a training step: The forward pass computes $F (X, W)$ . The backward pass computes $\partial F / \partial W$ and $\partial F / \partial X$ . In real training, you would compute the gradient of a loss function $L (y^, y)$ with respect to $W$ . Not $\partial F / \partial W$ directly. The loss adds one more node at the end of the graph. This question abstracts away the loss to test pure graph mechanics.

$\partial F / \partial X$ is computed for completeness only. In actual neural network training, features $X$ are fixed — you never backpropagate through them. This is an atypical exam question designed to verify you understand the chain rule, not that you know what to train.

8.5.8 Pitfalls

Skipping the sum node's contribution to both paths: When computing $\partial F / \partial X_{1}$ , both $Z_{1}$ and $Z_{2}$ are functions of $X_{1}$ . You must sum the gradient contributions from both paths. Forgetting the second path loses the contribution through $W_{21}$ .

Confusing $\partial F / \partial Q$ with $\partial F / \partial Z$ : For $F = Q_{1} + Q_{2}$ with $Q_{i} = Z_{i}^{2}$ : $\partial F / \partial Q_{i} = 1$ , but $\partial F / \partial Z_{i} = 2 Z_{i}$ . The factor of 2 comes from the square's derivative, not the sum's. Keep your layers straight.

Mixing up $W \cdot X$ vs $W^{T} \cdot X$ : The professor noted the question paper uses both forms in different places. Follow whichever form appears in the specific part you are answering. State your convention explicitly. The gradient shapes will differ by a transpose.

Treating $W$ as a vector: $W$ is a 2×2 matrix. Each entry $W_{ij}$ gets its own gradient computed independently. Do not try to write a single vector-gradient formula — compute entry by entry, as shown above.

8.5.9 Student Questions and Answers

Q: When we do matrix multiplication, do we use $W \cdot X$ or $W^{T} \cdot X$ ? The question shows both forms in different places.

A: Follow the formula as given in the specific part of the question. If it says $W^{T} X$ , use the transpose. If it says $W X$ , use the matrix product without transpose. Typically, unless specified otherwise, neural network literature uses $W^{T} X$ or more commonly $W x$ . This appears to be an unintended inconsistency in the question. State what convention you followed.

Q: For $\partial F / \partial Q$ , do we compute $\partial F / \partial Q_{1}$ and $\partial F / \partial Q_{2}$ separately or together?

A: Separately. The gradient w.r.t. $Q$ is a vector: $[\partial F / \partial Q_{1} \partial F / \partial Q_{2}]$ . Each component is computed independently.

Q: What is the formula for $\partial F / \partial X$ ?

A: Use the chain rule summing over all paths: $\partial F / \partial X_{1} = (\partial F / \partial Q_{1} \cdot \partial Q_{1} / \partial Z_{1} \cdot \partial Z_{1} / \partial X_{1}) + (\partial F / \partial Q_{2} \cdot \partial Q_{2} / \partial Z_{2} \cdot \partial Z_{2} / \partial X_{1})$ . Since $X_{1}$ influences both $Z_{1}$ and $Z_{2}$ through $W_{11}$ and $W_{21}$ , you must sum both contributions. You will NOT be asked to find gradients w.r.t. features in the exam — this question is atypical and tests pure chain rule mechanics.

A computational graph makes the chain rule explicit: each node computes a forward value and a local gradient. Backpropagation runs the graph backward, multiplying each node's local gradient by the incoming upstream signal. The final gradients w.r.t. all parameters are these multiplied products accumulated along all paths from the output. Next, we apply these gradient mechanics to diagnose a buggy neural network code snippet.

8.5.10 Real-World & Domain Connection

Computational graphs are the internal representation used by every modern deep learning framework. PyTorch calls it the "autograd graph" — every tensor operation adds a node, and calling .backward() traverses the graph backward. TensorFlow 2 uses "GradientTape" to record operations and replay them in reverse. Understanding the manual chain rule decomposition in this section is directly transferable: when you debug a None gradient in PyTorch, you are tracing the very same graph to find where the signal stopped flowing.

In production systems, the computational graph abstraction also enables optimizations like operator fusion (merging adjacent nodes for speed), memory planning (freeing intermediate values as soon as the backward pass is done with them), and distributed training (partitioning the graph across GPUs). What you draw by hand for a 2×2 matrix, industrial frameworks automate for graphs with millions of nodes.

8.6 Neural Network Code Analysis — Multi-Class Classification Bug

Hook: A code snippet claims to train a deep network for a 5-class problem. It runs without error — but it will never learn correctly. Three silent bugs hide in the architecture. Can you spot them all before the model wastes hours of training?

Intuition + Analogy: Debugging a neural network architecture is like checking a restaurant kitchen's layout. You have 30 ingredients (input features). You have two prep stations (hidden layers). You have a final dish (output). The buggy code has one chef at the final station. That chef makes only yes/no decisions (sigmoid + one output). But the order ticket says "5 different dishes" (5 classes). The loss function counts only yes/no correctness (binary cross-entropy). No matter how good the prep stations are, the final station is built for the wrong task. You need 5 chefs at the final station. Each specializes in one dish (5 outputs + softmax). They are judged by how well they collectively cover the menu (categorical cross-entropy).

Symbol Registry — 8.6:

Symbol	Meaning	Domain
$L_{BCE}$	Binary cross-entropy loss	$R^{+}$
$L_{CCE}$	Categorical cross-entropy loss	$R^{+}$
$λ$	Regularization strength	$R^{+}$
$R (W)$	Regularization penalty	$R^{+}$

8.6.1 Problem Description

A Python code snippet allegedly trains a deep neural network for a multi-class classification task with 5 output classes. You must identify errors by interpreting the code — no execution needed. The code compiles and runs, but the architecture is wrong.

8.6.2 Network Architecture (as Described in the Code)

The input data has 30 features per instance. The architecture:

Layer	Nodes	Activation
Input	30	—
Hidden 1	64	ReLU
Hidden 2	32	ReLU
Output	1	Sigmoid
Loss	—	Binary Cross-Entropy

8.6.3 Identified Errors

Error 1 — Output layer size: 1 neuron instead of 5

For 5 mutually exclusive classes, the output layer needs exactly 5 neurons — one logit per class. With a single output neuron, the network can only answer "yes" or "no" to one question. It cannot express which of the five classes is most likely. The single neuron receives a 32-dimensional vector from the last hidden layer. It collapses that into one scalar. This throws away the capacity to distinguish between classes 1 through 5.

Error 2 — Output activation: sigmoid instead of softmax

Sigmoid maps the single output to $(0, 1)$ , producing one probability. This only makes sense for binary classification where the second probability is $1 - y^$ . For five classes, you need softmax — which takes five raw logits and produces five probabilities that sum to 1. The softmax ensures mutual exclusivity: a high probability for one class automatically lowers the probability for others.

Error 3 — Loss function: binary cross-entropy instead of categorical cross-entropy

Binary cross-entropy expects a single target value $y \in {0, 1}$ and a single predicted probability $y^$ . It compares them with:

L_{BCE} = - [y lo g (y^) + (1 - y) lo g (1 - y^)]

For a 5-class problem with one-hot encoded labels $y \in {0, 1}^{5}$ and softmax outputs $y^\in [0, 1]^{5}$ , you need categorical cross-entropy:

L_{CCE} = - c = 1 \sum 5 y_{c} lo g (y^_{c})

Binary CE cannot ingest a 5-element one-hot vector. Even if the framework silently broadcasts or reshapes, the loss computation is semantically wrong — it treats a multi-class problem as 5 independent binary problems.

Corrected architecture:

Layer	Nodes	Activation
Input	30	—
Hidden 1	64	ReLU
Hidden 2	32	ReLU
Output	5	Softmax
Loss	—	Categorical Cross-Entropy

The hidden layers can stay the same — the bug is confined to the output configuration.

8.6.4 Overfitting Interpretation

If the (corrected) code reports high training accuracy but low testing accuracy, the model is overfitting. Overfitting means the model has memorized training data patterns too precisely and fails to generalize to unseen test data. This is the neural network equivalent of a student who memorizes the answer key to last year's exam but cannot solve new problems.

The signature of overfitting: training loss keeps decreasing while validation loss plateaus or increases. The gap between training and validation accuracy widens with each epoch.

8.6.5 Handling Overfitting

Approach 1 — Regularization (L1 / L2 weight penalties):

Add a term to the loss that penalizes large weights:

L_{total} = L_{CCE} + λ \cdot R (W)

L2 (Ridge / weight decay): $R (W) = \sum_{ij} W_{ij}^{2}$ — penalizes large squared weights, encouraging all weights to stay small
L1 (Lasso): $R (W) = \sum_{ij} ∣ W_{ij} ∣$ — penalizes absolute weight values, encouraging sparse weights (many become exactly zero)

The hyperparameter $λ$ controls the strength of regularization — large $λ$ forces simpler models (potentially underfitting), small $λ$ allows overfitting.

Approach 2 — Increase training data:

With more diverse training examples, the model cannot afford to memorize individual instances — it must find generalizable patterns. A model trained on 5 instances per class can memorize easily. A model trained on 100,000 instances is forced to learn features that work across all of them. Data augmentation (rotating images, adding noise, flipping) is a practical way to artificially increase dataset size.

Approach 3 — Early stopping: Stop training when validation loss stops improving, even if training loss continues decreasing. This prevents the model from entering the memorization phase.

Exam scope note: Formal regularization techniques (L1, L2 norms) are from the machine learning course and are not part of the midterm syllabus for this course. However, you may reference them if asked. The overfitting concept itself — recognizing the symptoms and understanding the cause — is in scope.

8.6.6 Visual Intuition

Plot training loss and validation loss on the same graph, with epochs on the x-axis and loss on the y-axis. Both start high. Training loss drops steadily — the model is fitting the training data. Validation loss also drops initially, following the training curve down. But then the curves diverge: training loss continues its descent while validation loss flattens or turns upward. The point of divergence is where memorization begins. The model starts encoding individual training examples rather than learning the underlying distribution. The gap between the two curves is the overfitting gap.

The takeaway: a model with zero training error is suspicious, not impressive.

8.6.7 Pitfalls

"The code runs, so it must be correct" — Neural network frameworks are flexible. They will happily train a sigmoid output on 5 classes. The loss will decrease. But the model learns the wrong thing — the training loss dropping is meaningless if the architecture is semantically incorrect for the task.

Fixing only one of the three bugs: Changing the output size to 5 but keeping sigmoid activation gives you 5 independent probabilities that don't sum to 1. Changing the loss to categorical cross-entropy but keeping 1 output + sigmoid gives a shape mismatch. All three — output size, activation, and loss — must be changed together.

Thinking deep networks cannot overfit: Depth actually increases the capacity to memorize. A deep network with many parameters can achieve 100% training accuracy on random labels. The training curve alone tells you nothing — always monitor the validation curve.

Confusing overfitting with poor feature selection: If BOTH training and validation accuracy are low, the model is underfitting — the architecture may be too shallow, the learning rate too small, or the features too weak. Overfitting specifically means high training performance and low validation performance.

A multi-class network must have a matching triplet: $C$ output neurons + softmax activation + categorical cross-entropy loss. Any deviation is a bug. When analyzing a code snippet, verify these three components together — they form a contract. Overfitting is diagnosed by the train-validation gap, not by the absolute loss values. Next, we build a correct architecture from scratch for a regression problem: predicting bike rental demand.

8.6.8 Real-World & Domain Connection

Architecture bugs like the one in this section are common in practice. A 2021 study of Kaggle competition notebooks found that ~12% of entries had at least one mismatch between output activation and loss function. In production, these bugs are worse — the model deploys, produces outputs, and nobody notices it is producing subtly wrong probabilities until a business metric drifts. Companies like Google and Meta use automated architecture validation in their model pipelines to catch output-layer mismatches before training begins.

Overfitting is the most common failure mode in applied deep learning. The train-validation gap is so fundamental that ML engineers monitor it as a dashboard metric. In medical diagnosis, an overfit model that memorizes training patient data will confidently misdiagnose new patients — the failure mode is high-confidence errors, which are the most dangerous kind.

8.7 Deep Feedforward Neural Network — Regression (Bike Rental Prediction)

Hook: You have weather data — temperature, humidity, wind speed, whether it is a holiday — and you need to predict exactly how many bikes will be rented next hour. This is not "yes or no." It is not "which category?" It is a number: 42 bikes, or 387 bikes. A classification network outputs probabilities. A regression network outputs raw continuous values. How do you build one?

Intuition + Analogy: Think of a regression network as a function fitter. You give it input numbers: temperature = 25°C, humidity = 60%, hour = 8 AM. It does math through its hidden layers. It spits out a predicted number: predicted rentals = 215 bikes. The loss says: "You predicted 215. The real answer was 247. You were off by 32." The network adjusts all its internal knobs (weights) to reduce that gap next time. After thousands of examples, the network becomes a reliable prediction machine.

The key difference from classification: no threshold. No "is this a cat?" decision. The output is the raw computed value. The quality measure is how close it is to the ground truth. It is measured by squared distance. Not by whether a probability exceeded 0.5.

Where the analogy breaks: the network does not understand what a bike is. It learns statistical associations between input numbers and output numbers. If snow is never in the training data, a snowy day produces a nonsense prediction.

8.7.1 Problem Setup

The task is to predict the hourly demand for bike rentals based on explanatory variables. Input features include:

Hour, temperature, humidity, visibility, wind speed, snowfall, rainfall, functioning day (holiday indicator), and other weather/seasonal features.

The output is a continuous value. It is the total number of bikes demanded in that hour. This makes it a regression task.

8.7.2 Network Architecture Design

Since the output is a continuous value, the architecture differs from classification in three places:

Component	Regression Choice	Why
Output layer size	1 neuron	Predicting a single scalar value
Output activation	Identity (linear)	No nonlinearity — let the value be any real number
Loss function	Mean Squared Error (MSE)	Penalizes squared distance from truth

The network has multiple hidden layers with ReLU activations, making it a deep feedforward neural network.

8.7.3 Training Hyperparameters

Hyperparameter	Value	Role
Learning rate ( $η$ )	0.01	Step size for weight updates
Batch size ( $N$ )	64	Number of instances per mini-batch
Training epochs	100	Full passes through the training data
Early stopping patience	15 epochs	Stop if validation loss does not improve for 15 consecutive epochs
Optimizer	Mini-batch gradient descent	Uses average gradient over each mini-batch

8.7.4 Data Split

90% training (for weight updates)
5% validation (for hyperparameter tuning and early stopping)
5% testing (held-out final evaluation)

The exact proportions can vary — 80/10/10 is also common. There is no fixed rule. The principle: use most data for training, and reserve enough for unbiased evaluation.

8.7.5 Forward Propagation Code Logic

For each layer (excluding the input layer):

Linear combination: $Z = W \cdot H_{prev} + B$
Activation decision:
- If current layer is the output layer: apply identity — return $Z$ as-is
- If current layer is a hidden layer: apply $ReLU (Z) = max (0, Z)$

In this implementation, ReLU is uniformly applied to all hidden layers. With library APIs, you can assign different activations to different layers.

8.7.6 Symbol Registry — Regression Loss

Symbol	Meaning	Domain
$N$	Mini-batch size	Integer, e.g., 64
$y_{i}$	True target value for instance $i$	$R$ (continuous)
$y^_{i}$	Predicted value for instance $i$	$R$ (continuous)
$L$	Loss value	Scalar
$η$	Learning rate	Scalar, e.g., 0.01
$W$	Weight matrix for a layer	$R^{d_{out} \times d_{in}}$
$H_{prev}$	Activations from previous layer	Vector
$B$	Bias vector for a layer	Vector

8.7.7 Mean Squared Error Loss — Full Derivation

The loss function:

L = \frac{1}{2 N} i = 1 \sum N (y_{i} - y^_{i})^{2}

Why the $\frac{1}{2}$ factor? It is a mathematical convenience. When you differentiate $(y_{i} - y^_{i})^{2}$ with respect to $y^_{i}$ , the chain rule brings down a factor of 2:

\frac{\partial}{\partial y ^ _{i}} (y_{i} - y^_{i})^{2} = 2 (y_{i} - y^_{i}) \cdot (- 1) = - 2 (y_{i} - y^_{i})

The prefactor $\frac{1}{2 N}$ absorbs one of the 2's, giving a clean gradient:

\frac{\partial L}{\partial y ^ _{i}} = \frac{1}{2 N} \cdot (- 2 (y_{i} - y^_{i})) = - \frac{1}{N} (y_{i} - y^_{i})

Interpretation: The gradient of the loss w.r.t. the prediction is proportional to the prediction error $(y_{i} - y^_{i})$ divided by the batch size. A large error produces a proportionally large gradient, driving a stronger weight update.

Without the 1/2 factor (standard MSE):

L = \frac{1}{N} i = 1 \sum N (y_{i} - y^_{i})^{2}, \frac{\partial L}{\partial y ^ _{i}} = - \frac{2}{N} (y_{i} - y^_{i})

Both forms are equivalent if you adjust the learning rate accordingly. The $\frac{1}{2}$ convention is used in this course.

8.7.8 Backward Propagation and Weight Update

The weight update procedure:

Purpose: Adjust all weights in all layers to reduce the MSE loss.

Inputs:

Error signal from the loss: $\frac{\partial L}{\partial y ^} = - \frac{1}{N} (y - y^)$
Layer activations from the forward pass: $H_{prev}, Z, W$ for every layer
Hyperparameters: $η = 0.01$

Steps:

Compute the error gradient at the output layer: $δ_{out} = \frac{\partial L}{\partial Z _{out}}$ (via chain rule through the identity activation)
For each hidden layer (working backward from output to input):
- Propagate the error backward: compute $δ_{ℓ}$ from $δ_{ℓ + 1}$
- Compute gradient w.r.t. weights: $\frac{\partial L}{\partial W _{ℓ}} = δ_{ℓ} \cdot H_{ℓ - 1}^{T}$
- Compute gradient w.r.t. bias: $\frac{\partial L}{\partial B _{ℓ}} = δ_{ℓ}$
Update weights: $W = W - η \cdot \frac{\partial L}{\partial W}$
Update biases: $B = B - η \cdot \frac{\partial L}{\partial B}$

In code: W = W - learning_rate * gradient

8.7.9 Prediction and Evaluation

Once training finishes:

Pass test inputs through forward propagation to get predictions $y^$
Compare $y^$ against the true values $y$
Compute MSE across all test instances: np.mean((y - y_pred)^2)
For RMSE: np.sqrt(mse) — brings the error back to the original unit (number of bikes)

8.7.10 Interpreting Training Results

Underfitting (high training loss, e.g., 0.5–0.7):

The model cannot even fit the training data well.
Possible causes: not enough model capacity (too few layers/neurons), poor feature engineering, inappropriate learning rate, or bad hyperparameters.
Solutions: add more layers or neurons, engineer better features, adjust the learning rate.

Overfitting (low training loss, high validation loss):

The model has memorized training data and cannot generalize.
Solutions: add regularization (L1/L2), collect more training data, use early stopping, or reduce model capacity.

The key diagnostic: always look at BOTH training and validation loss. The absolute value of training loss alone tells you nothing about generalization.

8.7.11 Visual Intuition

Plot two curves on the same axes: epochs on the x-axis, MSE loss on the y-axis. The training loss curve (blue) starts high and drops monotonically — the model always improves on data it has seen. The validation loss curve (orange) drops alongside it at first, then diverges. Two diagnostic patterns:

Underfitting: both curves are high and close together — the model lacks capacity
Overfitting: the blue curve keeps dropping while the orange curve rises — the gap widens

The ideal stopping point is the epoch where validation loss is at its minimum — right before the orange curve turns upward. Early stopping automates exactly this.

8.7.12 Pitfalls

Using sigmoid or softmax for regression output: If the true value is 247 bikes and sigmoid squashes output to (0, 1), the model can never predict the right answer. The output activation must match the output range.

Forgetting the $\frac{1}{2}$ factor in the gradient: The derivative of $\frac{1}{2} (y - y^)^{2}$ includes a cancellation. If you use standard MSE without the $\frac{1}{2}$ , your gradient is twice as large — effectively doubling your learning rate.

Interpreting MSE in isolation: MSE of 0.01 might be great or terrible depending on the scale of $y$ . If bike rentals range from 0 to 1000, MSE = 0.01 is stellar. If they range from 0 to 1, it is mediocre. Always compute RMSE to get error in the original units.

Not shuffling before creating mini-batches: If the data is sorted by hour, each mini-batch sees only one time of day. The gradient is biased. Always shuffle before batching.

A regression network is a function fitter: deep hidden layers with ReLU extract patterns, a single linear output neuron produces a raw number, and MSE measures the distance from truth. The same training loop — forward pass, loss computation, backward pass, weight update — works identically for regression and classification; only the output layer and loss function differ. Next: a detail that determines whether this training loop even starts successfully — how you initialize the weights.

8.7.13 Real-World & Domain Connection

Bike rental prediction is a canonical regression problem in applied machine learning. The Capital Bikeshare dataset from Washington, D.C. is used in countless tutorials and papers. The same architecture pattern — deep feedforward network with ReLU hidden layers, linear output, MSE loss — powers demand forecasting in ride-sharing (Uber, Lyft), inventory prediction in retail (Walmart), energy load forecasting for power grids, and financial time-series prediction. Any task where the output is a continuous number — stock price, temperature, sales volume, server load — uses this exact pattern.

The practice of monitoring training vs. validation loss curves is a universal skill. In industry, this is done through experiment tracking tools like Weights & Biases or TensorBoard. The train-val gap is the first thing an ML engineer checks after every experiment.

8.8 He Weight Initialization

Hook: You have spent twenty minutes training a network, and the loss has not budged. Nothing is learning. The culprit is not the architecture, not the learning rate, not the data — it is the weight initialization. Starting with the wrong random numbers can stall training before it begins. Starting with zeros is even worse. How do you pick the right starting values?

Intuition + Analogy: Think of weight initialization as placing hikers on a mountain range before they start descending. If all hikers start at the same spot (zero initialization), they all walk the same path and learn nothing new. If you scatter them randomly but with enormous steps, some start in the stratosphere. They never find a valley. This is exploding gradients. If you scatter them too tightly, they all start at the bottom of a tiny divot. They think they are done. This is vanishing gradients.

He initialization places each hiker at a carefully chosen random location. The variance of the starting positions is scaled to the number of trails feeding into each location. More incoming trails means smaller initial steps. This keeps signals from overwhelming the network. This ensures signals at every layer have roughly the same variance whether the network is shallow or deep.

8.8.1 Definition and Motivation

He initialization (named after Kaiming He, who derived it in 2015) is a method for setting initial neural network weights. Weights are drawn from a normal distribution. The variance depends on the layer's input size:

W_{ij} \sim N (0, σ^{2}), σ = \frac{2}{fan_in}

Where fan_in is the number of incoming connections (neurons) from the previous layer.

8.8.2 Symbol Registry — He Initialization

Symbol	Meaning	Value
$fan_in$	Number of incoming connections from the previous layer	Positive integer
$μ = 0$	Mean of the distribution	0
$σ = \frac{2}{fan_in}$	Standard deviation	Positive, depends on fan_in
$W_{ij}$	Weight from neuron $j$ (previous) to neuron $i$ (current)	$\sim N (0, σ^{2})$

8.8.3 Why Zero Initialization Fails

The symmetry problem: If all weights are zero, every neuron in a given layer computes:

Z_{k} = W_{k 0} \cdot 1 + 0 \cdot X_{1} + 0 \cdot X_{2} = 0

Every neuron outputs the same value. During backpropagation, all neurons receive identical gradients. The symmetry is preserved. Every neuron follows the exact same trajectory. After any amount of training, all neurons in the layer are identical to each other. They do the work of one neuron. No diverse feature learning occurs.

The same problem arises (more subtly) if all weights are initialized to the same non-zero constant. Symmetry must be broken by randomness.

8.8.4 Application

He initialization is applied once at the start of training, for every layer:

Layer with 5 incoming connections: $σ = 2/5 \approx 0.632$
Layer with 10 incoming connections: $σ = 2/10 \approx 0.447$
Layer with 256 incoming connections: $σ = 2/256 \approx 0.088$

Larger fan_in → smaller initial weights → prevents the weighted sum from blowing up as more inputs are added.

After initialization, training freely updates the weights through gradient descent. The initialization only determines the starting point — it does not constrain or control subsequent updates.

8.8.5 The Math Behind the $2/ fan_in$ Formula

Why this specific variance? The goal is to keep the variance of activations stable as they flow forward through the network, and the variance of gradients stable as they flow backward.

For a layer with ReLU activation, the forward pass is:

Z = W \cdot X, H = max (0, Z)

If the inputs $X$ have variance $Var (X)$ , and weights $W$ have variance $Var (W)$ , then (assuming independence and zero mean):

Var (Z) = fan_in \cdot Var (W) \cdot Var (X)

We want $Var (Z) = Var (X)$ — no amplification or attenuation. This requires:

Var (W) = \frac{1}{fan_in}

But ReLU sets all negative values to zero, discarding roughly half the variance. To compensate, He initialization doubles the variance:

Var (W) = \frac{2}{fan_in} \Rightarrow σ = \frac{2}{fan_in}

This is the He normal initialization. The uniform variant (He uniform) draws from $[- 6/ fan_in, 6/ fan_in]$ .

For comparison, Xavier/Glorot initialization (designed for tanh/sigmoid) uses $1/ fan_in$ without the factor of 2 — because tanh preserves variance better than ReLU.

8.8.6 Visual Intuition

Picture a histogram of initial weights for a layer with fan_in = 100. The distribution is a bell curve centered at zero. The standard deviation is $2/100 = 0.02 \approx 0.141$ . Most weights fall between -0.28 and +0.28 (two standard deviations). Almost all weights are within ±0.42 (three standard deviations).

Now contrast this with the same layer initialized from a standard normal $N (0, 1)$ : most weights fall between -2 and +2. The weighted sum $Z$ would blow up proportionally to $100 \approx 10$ times the input variance — saturating ReLUs immediately.

The takeaway: He initialization shrinks the initial weight spread as the layer gets wider, keeping the signal scale stable regardless of architecture depth or width.

8.8.7 Assumptions & Scope

He init is designed for ReLU and its variants (Leaky ReLU, PReLU). The factor of 2 compensates for ReLU's zeroing of negative values. If you switch to tanh or sigmoid activations, use Xavier/Glorot initialization ( $σ = 1/ fan_in$ ) instead — these activations preserve variance without the 2× correction.

He init is the default in modern frameworks. Both TensorFlow/Keras and PyTorch use He (or its uniform variant) as the default initializer for layers followed by ReLU. You rarely need to set it manually. Knowing the formula matters for the exam and for debugging — if your network is not learning, checking initialization is step one.

8.8.8 Pitfalls

Zero or constant initialization: Breaks symmetry. All neurons learn the same thing. Always use random initialization.

Using the wrong variance for the activation: He init ( $2/ fan_in$ ) for ReLU, Xavier/Glorot ( $1/ fan_in$ ) for tanh/sigmoid. Using He for tanh and Xavier for ReLU is not catastrophic but suboptimal — convergence will be slower.

Initializing biases to random values: Biases should be initialized to zero (or a small positive constant like 0.01 for ReLU to ensure neurons fire initially). Random bias initialization shifts the activation distribution and is unnecessary since weights already provide randomness.

Thinking fan_in = total neurons in the layer: fan_in counts only incoming connections from the PREVIOUS layer, not all neurons in the network. For a layer receiving 64 inputs, fan_in = 64, regardless of the layer's own output size.

8.8.9 Student Questions and Answers

Q: For He initialization, do we apply it to every layer or only at the beginning?

A: Every layer gets initialized once at the start of training. It is a one-time operation. For a layer with 5 incoming connections, $σ = 2/5$ . For a different layer with more connections, the standard deviation adjusts accordingly. After initialization, gradient descent updates the weights freely — the initialization does not constrain subsequent changes.

Q: If we control weights through initialization, how will the network learn?

A: Initialization only sets the starting values. It does not control or limit the weights during training. The network freely updates all weights through gradient descent. Initialization ensures different neurons start from different random positions, breaking symmetry so each neuron can learn different features. Think of it as assigning different starting positions in a race — everyone runs the same course, but nobody starts at the exact same spot.

He initialization draws weights from $N (0, 2/ fan_in)$ . The variance shrinks as the layer gets wider — wider layers get smaller initial weights to prevent the weighted sum from exploding. This one-time operation at the start of training makes the difference between a network that learns and one that stalls. Next: the training loop that uses these initialized weights — epoch by epoch, batch by batch, update by update.

8.8.10 Real-World & Domain Connection

He initialization was introduced in Kaiming He's 2015 paper "Delving Deep into Rectifiers." That paper also introduced PReLU activation. The paper showed that proper initialization enabled training of extremely deep networks (30+ layers). These networks previously failed to converge. This work, along with batch normalization, was instrumental in enabling deep residual networks (ResNets). ResNets won the ImageNet competition in 2015 with networks over 150 layers deep.

Today, He initialization is the default in every major deep learning framework. In PyTorch, nn.Linear and nn.Conv2d use He uniform initialization by default. In TensorFlow/Keras, kernel_initializer='he_normal' or 'he_uniform' are built-in options. The formula is so reliable that practitioners rarely think about weight initialization unless they are debugging a convergence issue — and when they do, switching between He and Xavier is one of the first knobs they turn.

8.9 Training Loop Mechanics

Hook: You have designed the architecture. You have initialized the weights. You have prepared the data. Now the actual work begins: pressing "train" and watching the loss drop epoch after epoch. But what is actually happening inside that loop? What makes the weights change, and why does the order of operations matter?

8.9.1 Training Procedure — Procedural Spine

Symbol Registry — 8.9:

Symbol	Meaning	Domain
$X_{train}$	Training feature matrix	$R^{m \times d}$
$Y_{train}$	Training targets	$R^{m \times 1}$ or $R^{m \times C}$
$m$	Number of training instances	Integer
$d$	Input feature dimension	Integer
$N$	Batch size	Integer, e.g., 64
$E$	Number of epochs	Integer
$P$	Early stopping patience	Integer
$δ^{[ℓ]}$	Error signal at layer $ℓ$	Vector
$W^{[ℓ]}$	Weights at layer $ℓ$	Matrix
$H^{[ℓ]}$	Activations at layer $ℓ$	Vector

Purpose: The training loop iteratively adjusts all network weights to minimize the loss function. It transforms random initial weights into a useful mapping from inputs to outputs.

Inputs:

Training data: $X_{train}$ (feature matrix, $m \times d$ ) and $Y_{train}$ (targets, $m \times 1$ or $m \times C$ )
Model: initialized weights $W$ and biases $B$ for all layers
Hyperparameters: learning rate $η$ , batch size $N$ , number of epochs $E$ , early stopping patience $P$

Outputs:

Trained weights $W^{*}$ and biases $B^{*}$ that minimize the loss on validation data
Training and validation loss history for diagnostic plots

Steps (per epoch):

Shuffle the training data randomly. This prevents the model from learning spurious patterns from data ordering (e.g., all class-1 samples first, then all class-2).

Create mini-batches of size $N$ (e.g., 64). Partition the shuffled data into $m / N$ chunks. The last chunk may be smaller.

For each mini-batch $(X_{batch}, Y_{batch})$ :

Forward propagate: Feed $X_{batch}$ through every layer sequentially:

Z^{[ℓ]} = W^{[ℓ]} \cdot H^{[ℓ - 1]} + B^{[ℓ]}, H^{[ℓ]} = σ^{[ℓ]} (Z^{[ℓ]})

Store $Z^{[ℓ]}$ and $H^{[ℓ]}$ for each layer — they are needed for backpropagation.

Compute loss: Evaluate $L (Y_{batch}, Y^)$ where $Y^= H^{[L]}$ is the output of the final layer.

Backward propagate: Starting from the output layer, compute gradients of the loss w.r.t. every weight and bias using the chain rule:

δ^{[L]} = \frac{\partial L}{\partial Z ^{[L]}}, δ^{[ℓ]} = ((W^{[ℓ + 1]})^{T} δ^{[ℓ + 1]}) ⊙ (σ^{[ℓ]})^{'} (Z^{[ℓ]})

\frac{\partial L}{\partial W ^{[ℓ]}} = δ^{[ℓ]} \cdot (H^{[ℓ - 1]})^{T}, \frac{\partial L}{\partial B ^{[ℓ]}} = δ^{[ℓ]}

Update weights: Apply gradient descent:

W^{[ℓ]} \leftarrow W^{[ℓ]} - η \cdot \frac{\partial L}{\partial W ^{[ℓ]}}, B^{[ℓ]} \leftarrow B^{[ℓ]} - η \cdot \frac{\partial L}{\partial B ^{[ℓ]}}

Validate: After each epoch, compute the loss on the validation set (no weight updates — forward pass only).

Check early stopping: If validation loss has not improved for $P$ consecutive epochs, stop training.

Trace — one epoch on a tiny dataset (4 instances, batch size 2):

Initial data order: [A, B, C, D] with weights $W_{0}$

Shuffle → [C, A, D, B]
Mini-batches: Batch 1 = [C, A], Batch 2 = [D, B]
Batch 1: forward → loss → backward → update weights to $W_{1}$
Batch 2: forward (with $W_{1}$ ) → loss → backward → update weights to $W_{2}$
Validate on held-out set with $W_{2}$ → record loss
Next epoch: shuffle again, new batch groupings

Each batch produces a different gradient direction. Averaging gradients over 64 instances produces a smoother, more stable update than using one instance at a time.

8.9.2 Mini-Batch Gradient Descent — Comparison

Complexity & Cost:

Gradient Type	Batch Size	Updates per Epoch	Gradient Noise	Memory Cost
Batch (Full) GD	$m$ (all data)	1	None (exact)	High (full dataset in memory)
Stochastic GD	1	$m$	High (very noisy)	Low
Mini-batch GD	$N$ (e.g., 64)	$m / N$	Moderate	Moderate

When to Use / Alternatives:

Mini-batch GD is the standard for deep learning. The noise acts as implicit regularization — it helps escape shallow local minima.
Batch GD is only practical for tiny datasets that fit in memory.
SGD (batch size 1) converges but oscillates heavily. Use it only for online learning where data arrives one point at a time.
Adaptive methods (Adam, RMSprop) build on mini-batch GD by adapting the learning rate per parameter. These are the practical default in modern frameworks.

8.9.3 Visual Intuition

Picture a 3D loss landscape: hills and valleys, with the x and y axes representing two weights and the z-axis representing the loss. Gradient descent is a rolling ball. Full-batch GD rolls straight toward the deepest valley — smooth but requires computing the gradient over the entire landscape every step. SGD jerks wildly, taking sharp turns with every data point — noisy but fast. Mini-batch SGD is a compromise: the ball rolls with a slight wobble, averaging out noise over 64 data points per step.

The batch size determines the wobble: batch size 1 is the wildest, batch size 1024 is the smoothest but slowest per-update. The "right" batch size is the largest that still fits in GPU memory while producing stable training curves.

8.9.4 Pitfalls

Not shuffling between epochs: If the data order is fixed, the model sees the same batch compositions every epoch. This can create systematic bias — certain batches always under-represent some classes. Always shuffle.

Confusing epoch, iteration, and batch: An epoch is one full pass through all training data. An iteration (or step) is one weight update using one mini-batch. With batch size 64 and 6400 training instances: 6400/64 = 100 iterations per epoch.

Updating weights on the validation set: Validation data must never influence weight updates. The forward pass runs on validation to compute loss, but W = W - η·grad is skipped. Leaking validation data into training makes the validation loss meaningless.

Choosing batch size that does not divide dataset evenly: The last mini-batch may have fewer than $N$ instances. This is fine — frameworks handle it. Using drop_remainder=True discards the last partial batch, which wastes data but simplifies code. For exam answers, just note the batch size and how many batches per epoch.

The training loop is the engine of deep learning: shuffle → batch → forward → loss → backward → update → repeat. Every concept from previous sections — output layer design, loss function choice, weight initialization, gradient computation — plugs into this loop. The loop is identical whether you train a perceptron on NAND or a deep network on bike rentals. Next: we leave tabular data behind and enter the world of images with Convolutional Neural Networks.

8.9.5 Real-World & Domain Connection

The training loop described here is the blueprint for every deep learning experiment. In production, this loop is wrapped in frameworks that add distributed training (splitting batches across GPUs), mixed-precision training (using 16-bit floats for speed), gradient accumulation (simulating larger batch sizes on limited memory), and checkpointing (saving weights periodically so training can resume after crashes).

The early stopping mechanism in this loop is particularly important in practice. Without it, models overtrain and deployment performance degrades. In industry, early stopping is typically combined with model checkpointing — save the weights at the epoch where validation loss was lowest, not the final epoch. This saved checkpoint is what gets deployed to production, not the last-trained model.

8.10 Introduction to Convolutional Neural Networks

Hook: A single 1000×1000 color photo has 3 million numbers. Feed it into a regular neural network with one hidden layer of 1000 neurons. You get 3 billion weights to train. That is more parameters than stars you can see with the naked eye. Training this would take years. It would likely never converge. Yet your phone classifies photos in milliseconds. How? The answer is Convolutional Neural Networks. CNNs treat images as images, not as long lists of unrelated numbers.

Intuition + Analogy: A regular neural network sees an image as a bag of pixels. It has no idea that pixels next to each other are related. This is like reading a book by cutting out every letter individually. You shuffle them in a bag and try to understand the story from letter frequencies. You lose all information about words, sentences, and paragraphs.

A CNN reads the image like you read a page. It reads in small, overlapping patches. A filter (a small window of learnable weights) slides across the image. It looks for a specific pattern: an edge, a corner, a texture, a color gradient. The same filter is reused everywhere. The edge detector that finds the cat's ear also finds the cat's tail. This weight sharing is the key insight. Three million pixels do not need 3 million unique weights. A few thousand filter weights, reused across the whole image, can do the job.

Where the analogy breaks: you read linearly, left-to-right, top-to-bottom. CNNs process all patches in parallel through convolution — a mathematical operation that is efficiently implemented on GPUs.

8.10.1 Moving from Structured to Unstructured Data

All previous examples used handcrafted features from tabular (structured) data. The model received a fixed set of engineered features as input. Examples: temperature, humidity, hour. Going forward, the course addresses unstructured data:

Image data
Sequential / time-series data
Natural language text (NLP)

This module focuses on image data (computer vision) using Convolutional Neural Networks (CNNs).

Symbol Registry — 8.10:

Symbol	Meaning	Domain
$N$	Image width (pixels)	Integer
$M$	Image height (pixels)	Integer
$C$	Number of channels (e.g., 3 for RGB)	Integer
R, G, B	Red, Green, Blue color channels	$[0, 255]$

8.10.2 Computer Vision Tasks — Increasing Complexity

Image Classification (simplest):

Input: an image of dimensions $N \times M \times C$ (width × height × channels)
Output: a class label (binary or multi-class)
A plain deep network could work for very small images. For high-resolution images, the parameter count becomes unmanageable.

Object Detection (more complex):

Task: "what is in the image" AND "where is it"
Output: a bounding box defined by at least 3 values — a reference point (e.g., top-left corner), width, and height. Plus a class label if classification is also needed.

Multiple Object Detection + Recognition (more complex):

Multiple objects in one scene
For each object: bounding box, class label, and confidence score (probability)

Image Captioning (most complex):

Three combined tasks: detect objects → recognize them → generate a descriptive sentence
CNN handles detection + recognition; RNN handles sentence generation
This is a hybrid CNN+RNN architecture (covered after session 12)

8.10.3 Image Data Representation

An image of size $64 \times 64 \times 3$ means:

64 pixels wide
64 pixels high
3 color channels: Red, Green, Blue (RGB)

Each channel is a separate $64 \times 64$ grid of pixel intensity values — called a feature map at the input layer. A single pixel at position (row, col) has three intensity values: one for red, one for green, one for blue. These three values together define the pixel's color.

Formal notation: An image is a tensor of dimensions $N \times M \times C$ , where $N$ and $M$ are spatial dimensions and $C$ is the number of channels.

8.10.4 Why a Specialized Architecture Is Needed

Problem 1 — Dimensionality explosion: A $1000 \times 1000$ grayscale image = 1,000,000 input features. In color (3 channels) = 3,000,000 features. Even one hidden layer of 1000 neurons creates $3 \times 1 0^{9}$ weights. That is 3 billion trainable parameters. This is computationally infeasible for training. You need impractically large datasets to avoid immediate overfitting.

Problem 2 — Redundancy in pixel data: Large regions of an image contain redundant information. A blue sky patch of $200 \times 200$ pixels is nearly identical across all 40,000 pixels. Processing every pixel independently wastes computation. Only informative regions — edges, corners, textures, object boundaries — carry useful signal.

Problem 3 — Resolution sensitivity: Pixelated or low-resolution images change the input dimensionality. A mechanism that assumes a fixed input size ( $1000 \times 1000$ ) cannot handle a $640 \times 480$ image without resizing or padding.

Problem 4 — Translational variance: The same object can appear anywhere in the image. A cat in the top-left corner and the same cat shifted to the bottom-right are the same class — but a regular network sees them as completely different input vectors. The model must be translation-equivariant: shifting the input should shift the feature map accordingly, without affecting the final classification.

8.10.5 The CNN Solution — Feature Extraction Before Classification

The CNN inserts a feature extraction stage between the raw image and the classifier:

Input Image → [Convolution + Pooling] × N → Feature Map → Feedforward Network → Class Label

Convolution: A small learnable filter (e.g., $3 \times 3$ or $5 \times 5$ ) slides across the image spatially. At each position, it computes the dot product between the filter weights and the overlapping image patch. The same filter is reused across the entire image (weight sharing), dramatically reducing parameters. Each filter learns to detect one type of pattern — horizontal edges, vertical edges, blobs, textures.

Pooling: After convolution, pooling compresses the feature map spatially. A common operation is max pooling: slide a $2 \times 2$ window across the feature map and keep only the maximum value in each window, discarding the other three. This halves the spatial dimensions, reduces computation for deeper layers, and introduces a small amount of translation invariance.

Together, convolution + pooling solve the four problems:

Dimensionality: weight sharing means parameters depend on filter size, not image size
Redundancy: pooling discards redundant information while preserving strong activations
Resolution: convolution operates on any input size; pooling adapts accordingly
Translation: convolution is inherently translation-equivariant — shift the input, the feature map shifts identically

Scope: This is an introduction. The course focuses on understanding how convolution and pooling work, and on comparing different CNN architectures (their features, strengths, weaknesses, and suitable applications). Mathematical proofs of why one architecture outperforms another are not covered.

8.10.6 Visual Intuition

Picture a grayscale image as a $64 \times 64$ grid of numbers (0 = black, 255 = white). A $3 \times 3$ edge-detection filter might look like:

- 1& - 1& - 1 - 1&8& - 1 - 1& - 1& - 1

Slide this filter across every $3 \times 3$ patch of the image. At each position, multiply the 9 filter values by the 9 corresponding pixel values and sum them. Where the image is uniform (all pixels similar), the sum is near 0. Where there is a sharp transition (edge), the center pixel differs from its neighbors, and the sum is large. The output is a new grid — the feature map — where bright pixels mark edges.

Now picture stacking several such feature maps. The first convolutional layer might produce 32 feature maps, each from a different filter — one detecting horizontal edges, one detecting vertical edges, one detecting diagonal textures, and so on. The second convolutional layer combines these 32 edge/texture maps to detect corners and simple shapes. Deeper layers detect noses, eyes, wheels, windows. The final layers detect "cat" or "car."

The takeaway: CNNs learn a hierarchy of visual features, from simple edges in early layers to complex object parts in deep layers — all from raw pixels, with no handcrafted feature engineering.

8.10.7 Pitfalls

Using a regular network for large images: The parameter explosion is real. A $224 \times 224 \times 3$ image with a 4096-neuron hidden layer has $224 \times 224 \times 3 \times 4096 \approx 600$ million weights in that layer alone. Your GPU will run out of memory before the first backward pass.

Confusing feature maps with channels: At the input, channels are RGB. After the first convolution, "channels" means "number of filters applied" — e.g., 32 feature maps. Each subsequent layer's "channel count" equals the number of filters from the previous convolution.

Thinking CNNs only work for images: The convolution operation is applicable to any data with spatial or sequential structure — 1D convolution for time series and text, 3D convolution for video and volumetric medical scans. The principle (weight sharing over a sliding window) is universal.

Assuming CNNs are rotation-invariant by default: Convolution is translation-equivariant but NOT rotation-equivariant. Max pooling provides only slight translation invariance. Data augmentation (rotating training images) is needed for rotation robustness.

8.10.8 Coming Topics

In subsequent contact sessions:

Convolution operation — how filters slide and compute dot products in detail
Pooling operation — max pooling, average pooling, and their effects
Different CNN architectures (LeNet, AlexNet, VGG, ResNet, etc.) and their unique contributions
Pros and cons of each architecture for specific tasks (single object detection, multi-object detection, speed vs. accuracy trade-offs)

CNNs solve the image problem by replacing fully-connected layers with convolution and pooling — operations that exploit spatial structure instead of ignoring it. Weight sharing (reusing the same filter everywhere) is the key insight that makes deep learning on images computationally feasible. The next several lectures dive deep into the mechanics of convolution, the architectures that dominate computer vision, and how to choose between them for different tasks.

8.10.9 Real-World & Domain Connection

CNNs are the backbone of modern computer vision. The architecture family described here includes:

LeNet-5 (1998): the original CNN for handwritten digit recognition on checks
AlexNet (2012): the network that launched the deep learning revolution. It won ImageNet with a 10% margin over the runner-up.
VGG, GoogLeNet/Inception, ResNet: progressively deeper architectures that pushed image classification accuracy past human-level
YOLO, Faster R-CNN: real-time object detection used in self-driving cars, surveillance, and retail analytics
U-Net: medical image segmentation for tumor detection

The CNN principles you are learning — weight sharing, hierarchical feature learning, spatial invariance — are the same principles that power facial recognition on your phone, pedestrian detection in autonomous vehicles, and crop disease identification from satellite imagery. The next lectures will equip you to understand, compare, and choose among these architectures.

Exam Guidance Summary

Exam format: The paper is internally 30 marks, scaled to 100. Difficulty does not change — only the per-question mark numbers increase. Read the question-paper instructions carefully for the scaled marks.

Question Types and Mark Distribution

Numerical problems (high weight): forward/backward propagation, weight updates, gradient computation. Practice with pen and paper at least twice.
Code-related questions (15--20 marks out of 100): filling in blanks, identifying bugs, completing formulas in pseudocode. You do NOT need perfect Python syntax. Understanding forward/backward propagation and code structure is what is tested.
Theoretical questions: explaining concepts, justifying architectures, activation functions, loss functions.

Key Topics and Preparation Checklist

Topic	What to Know
Perceptron learning algorithm	Full numerical walkthrough with weight update rule $Δ W_{i} = (T - Y) \cdot X_{i}$ . Know the sign(0) = +1 convention.
Sigmoid vs. Softmax	When to use each, formulas, equivalence at $C = 2$
Binary CE vs. Categorical CE	Memorize both formulas; know which pairs with which activation
Computational graph	Be able to draw one for a given function and trace the backward pass
Code analysis	Identify: input dimensionality, output layer size, activation functions, loss function appropriateness
Output layer design	Binary: 1 neuron + sigmoid + BCE. Multi-class: C neurons + softmax + CCE. Regression: 1 neuron + linear + MSE.
He initialization	Formula $σ = 2/ fan_in$ , why it matters, zero-init failure
Training loop	Epoch structure: shuffle → batch → forward → loss → backward → update
Underfitting vs. Overfitting	Diagnose from train/val loss curves; solutions for each
Confusion matrix, precision, recall, F1	Interpret from a use case perspective (refresh from ML course)

Study Resources

Uploaded PPT and numerical questions PDF (~80 pages) — these are enough
All lab-based discussions, especially Webinar 1 and Webinar 2
Code demos: single neuron, binary classification, multi-class classification, deep feedforward regression

Excluded from syllabus:

Question 2, Part C (XR operations / designing perceptron counts) from old question papers
Formal L1/L2 regularization (from the ML course — may be referenced but is not midterm material)
Detailed CNN architectures (this session is introductory only; in-depth study begins next sessions)

Answer Writing Tips

Write all assumptions explicitly at the top of each answer — marks depend on justification.
Tabulate results where possible (easier to grade).
Use bulleted key terms for theoretical answers rather than long paragraphs.
For code questions, focus on architecture correctness — not Python syntax.

Key Industry Applications

Classification

Concept	Application
Perceptron learning	Spam detection, credit approval, simple medical screening
Multi-layer perceptrons	Image classification, speech recognition, financial fraud detection
Binary cross-entropy loss	Medical diagnosis (disease positive/negative), sentiment analysis, email spam
Categorical cross-entropy loss	Multi-disease diagnosis, handwritten digit recognition (MNIST), object classification
Softmax activation	Language identification, speaker identification, product categorization.

Regression

Concept	Application
Deep feedforward regression	Demand forecasting (bike/car rentals, ride-sharing), inventory prediction, energy consumption forecasting, financial time-series.

Training Infrastructure

Concept	Application
He initialization	Standard default in TensorFlow, PyTorch for layers with ReLU. Named after Kaiming He; enabled training of very deep networks (30+ layers).
Mini-batch gradient descent	Universal training paradigm. Powers every deep learning experiment from academic research to production deployments.

Computer Vision (Introductory)

Concept	Application
Convolutional Neural Networks	Facial recognition and autonomous vehicles. Object detection in medical imaging. Satellite imagery analysis. Augmented reality and video surveillance.
Object detection + recognition	Self-driving cars (pedestrians, vehicles, signs), retail analytics (customer tracking), security systems
Image captioning (CNN + RNN)	Accessibility tools for the visually impaired. Automatic content tagging for image search engines. Social media automation.

Domain Breadth

The concepts in this lecture form the foundational layer of applied deep learning. The same output-layer design principles apply identically wherever you work. Sigmoid + BCE for binary. Softmax + CCE for multi-class. Linear + MSE for regression. The same principles hold whether you are building a medical diagnosis system, a financial fraud detector, or a satellite image classifier. The training loop — shuffle, batch, forward, loss, backward, update — is the universal engine. Weight initialization, loss function selection, and architecture debugging are transferable skills across every deep learning domain and every framework.

DNN Lecture 08 notes · Deep Neural Networks — Core Concepts and Introduction to CNNs

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

18.1 Exam Format, Scaling, and Preparation Strategy

How the 30-mark paper is scaled to 100, question types, and study strategy.

28.2 Perceptron Learning Algorithm for the NAND Gate

Stochastic and batch perceptron updates on the bipolar NAND truth table.

38.3 Linear Separability and Multi-Layer Perceptrons

Why one perceptron fails on curved data and how hidden layers with nonlinearity solve it.

48.4 Medical Diagnosis Models — Binary vs Multi-Class Design

Choosing output neurons, activation, and loss for binary, multi-class, and multi-label tasks.

58.5 Computational Graphs and Gradient Computation

Drawing a computational graph and tracing gradients with the chain rule.

68.6 Neural Network Code Analysis — Multi-Class Bug

Spotting the output-size, activation, and loss mismatches plus overfitting remedies.

78.7 Deep Feedforward Neural Network — Regression

Designing a regression network with a linear output and mean squared error loss.

88.8 He Weight Initialization

Drawing weights from N(0, sqrt(2/fan_in)) and why zero init fails.

98.9 Training Loop Mechanics

Epoch structure: shuffle, batch, forward, loss, backward, update, and early stopping.

108.10 Introduction to Convolutional Neural Networks

Why convolution and pooling replace fully-connected layers for image data.

11Exam Guidance Summary

Consolidated exam format, topic checklist, and excluded syllabus items.

12Key Industry Applications

Where each concept appears in production ML systems.

Postgraduate students studying Deep Neural Networks

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Exam Format and Mark Scaling

Must-know: The paper is set for 30 internal marks but scaled to 100. Difficulty is unchanged — only the per-question mark values grow, so allocate time using the scaled marks on the paper.

$final_score = \frac{marks earned on the 30-mark scheme}{30} \times 100$

⚠️ Top pitfall: Panicking at large mark allocations. A 2-mark internal question becomes 5–7 scaled marks, but its intrinsic difficulty is identical.

Self-check: If a question is worth 2/30 internally, what fraction of the scaled 100-mark paper does it represent?

Connects to: Perceptron learning, Output layer design, Computational graphs.

Perceptron Learning Algorithm (NAND Gate)

Must-know: Update weights only when the prediction is wrong: shift each weight by (T−Y)·X_i. With bipolar labels and sign(0)=+1, one epoch learns NAND.

$Δ W_{i} = η (T - Y) X_{i}, W_{i}^{new} = W_{i}^{old} + Δ W_{i}$

⚠️ Top pitfall: Using sign(0)=−1 (the session's wrong demo) in the exam, or swapping to (Y−T)·X which moves weights the wrong way.

Self-check: For a wrong prediction where T=+1 and Y=−1, does the weight increase or decrease along a positive input?

Connects to: Linear separability, Computational graphs.

Linear Separability and Multi-Layer Perceptrons

Must-know: A single perceptron draws one straight line and fails on non-separable data (XOR, circles). Hidden layers with nonlinear activation compose many linear boundaries into curves.

$w^{T} x + b > 0 \forall x \in A, w^{T} x + b < 0 \forall x \in B \Rightarrow linearly separable$

⚠️ Top pitfall: Stacking linear layers without nonlinearity — the composition stays linear, so the network can only draw straight lines.

Self-check: Why does adding a ReLU hidden layer let the network separate concentric circles?

Connects to: Perceptron learning, Output layer design.

Output Layer Design — Sigmoid vs Softmax

Must-know: Binary → 1 sigmoid neuron + binary cross-entropy. Multi-class (C) → C softmax neurons + categorical cross-entropy. Regression → 1 linear neuron + MSE.

$σ (z) = \frac{1}{1 + e ^{- z}}, y^_{c} = \frac{e ^{z_{c}}}{\sum _{j = 1}^{C} e ^{z_{j}}}$

⚠️ Top pitfall: Using sigmoid for a 5-class problem — it gives five independent probabilities that need not sum to 1, destroying mutual exclusivity.

Self-check: For C=2, why is one sigmoid output enough instead of two softmax outputs?

Connects to: Regression (MSE), Code analysis bugs, Training loop.

Computational Graphs and Backpropagation

Must-know: Backprop multiplies each node's local gradient by the incoming upstream signal and accumulates along every path to the parameters. Sum nodes duplicate; product nodes cross-multiply.

$\frac{\partial F}{\partial W _{11}} = \frac{\partial F}{\partial Q _{1}} \frac{\partial Q _{1}}{\partial Z _{1}} \frac{\partial Z _{1}}{\partial W _{11}} = 1 \cdot 2 Z_{1} \cdot X_{1}$

⚠️ Top pitfall: Forgetting that X_1 feeds both Z_1 and Z_2, so ∂F/∂X_1 sums both gradient paths.

Self-check: What is ∂F/∂Q_i for F = Q_1 + Q_2? Is it 1 or 2Q_i?

Connects to: Perceptron learning, Training loop.

Multi-Class Network Bug Analysis

Must-know: A 5-class network needs the triplet: C output neurons + softmax + categorical cross-entropy. Fix all three together, never just one.

$L_{CCE} = - c = 1 \sum C y_{c} lo g (y^_{c}), L_{BCE} = - [y lo g y^+ (1 - y) lo g (1 - y^)]$

⚠️ Top pitfall: Fixing only the output size to 5 but keeping sigmoid — you get 5 independent probabilities that do not sum to 1.

Self-check: A 5-class snippet uses 1 output + sigmoid + BCE. Name the three changes required.

Connects to: Output layer design, Regression (MSE), Overfitting.

Deep Feedforward Regression (MSE)

Must-know: Regression uses 1 linear (identity) output neuron + mean squared error. Same train loop as classification; only the output activation and loss differ.

$L = \frac{1}{2 N} i = 1 \sum N (y_{i} - y^_{i})^{2}, \frac{\partial L}{\partial y ^ _{i}} = - \frac{1}{N} (y_{i} - y^_{i})$

⚠️ Top pitfall: Applying sigmoid or softmax to a regression output — it squashes the value and can never predict the true continuous target.

Self-check: Why does the 1/2 factor in the MSE make the gradient cleaner?

Connects to: Output layer design, Code analysis bugs, Training loop.

He Weight Initialization

Must-know: Draw weights from N(0, √(2/fan_in)) for ReLU layers. This prevents vanishing or exploding signals as width grows; never initialize to zero (symmetry).

$W_{ij} \sim N (0, \frac{2}{fan_in}), fan_in = number of incoming connections$

⚠️ Top pitfall: Zero or constant initialization — every neuron computes the same thing and symmetry is never broken.

Self-check: Why does a layer with 256 inputs get a smaller initial σ than one with 5 inputs?

Connects to: Training loop, Linear separability.

Training Loop Mechanics

Must-know: Each epoch: shuffle → mini-batch → forward → loss → backward → update. Monitor the train/validation gap; stop when validation loss stops improving.

$W^{[ℓ]} \leftarrow W^{[ℓ]} - η \frac{\partial L}{\partial W ^{[ℓ]}}, B^{[ℓ]} \leftarrow B^{[ℓ]} - η \frac{\partial L}{\partial B ^{[ℓ]}}$

⚠️ Top pitfall: Not shuffling between epochs, or leaking validation data into weight updates.

Self-check: With 6400 instances and batch size 64, how many iterations per epoch?

Connects to: Computational graphs, He initialization, Regression (MSE).

Introduction to Convolutional Neural Networks

Must-know: CNNs replace fully-connected layers with convolution (a sliding, weight-shared filter) + pooling, exploiting spatial structure. Weight sharing makes images tractable.

$output (i, j) = a, b \sum filter (a, b) image (i + a, j + b) (convolution with weight sharing)$

⚠️ Top pitfall: Using a plain network for large images — the parameter count explodes (about 600M weights for one layer on 224×24×3 input).

Self-check: Why is convolution translation-equivariant but not rotation-equivariant?

Connects to: Output layer design, Linear separability.

Previous LectureLecture 07 · Deep Feed-Forward Neural Networks

Next LectureLecture 09 · Convolutional Neural Networks

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.