Deep Neural Network

Deep Feed-Forward Neural Networks

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students studying deep learning and neural network fundamentals.

Deep Feed-Forward Neural Networks

7.1 Review of Deep Neural Network Fundamentals

Why can't a single neuron solve every problem? Because real-world data is rarely a straight line. A deep network solves this by stacking layers — each layer learns to see a richer picture than the one before it.

7.1.1 Quick Recap of Module 5

A deep neural network extends a single-layer perceptron by introducing multiple hidden layers. Think of a single perceptron as a person who can only make yes/no decisions by drawing one straight line. A deep network is a team of such people working in stages. The early team members spot basic shapes. They pass their notes to the next team, who combine those shapes into objects. The final team says "it's a cat."

A multilayer perceptron has one or more hidden layers between input and output. Each hidden layer is a group of neurons. Every neuron computes a weighted sum of its inputs, adds a bias, then passes the result through an activation function.

Nonlinearity is the key ingredient. You introduce it through activation functions applied at each neuron. Without nonlinearity, stacking a hundred layers would be no better than one layer — The whole network would collapse to a single straight-line decision boundary.

Backpropagation works by propagating adjustment factors backwards from the output. The gradient of the loss with respect to each weight tells you how much to nudge that weight. The chain rule of calculus does the heavy lifting: it breaks the blame for. The final error into tiny pieces and assigns each piece to the right weight.

When adjustment factors propagate across many layers, two problems can arise:

Vanishing gradient problem: Gradients become so small that earlier layers stop learning.

The weight update signal fades away before it reaches the early layers.

Exploding gradient problem: Gradients become so large that weights diverge. The updates swing wildly and the network becomes unstable.

7.1.2 Visualizing What a Deep Network Does

To understand why depth matters, consider progressively harder classification problems. Picture a 2D scatter plot. Blue dots and orange dots. You need to draw a boundary between them.

Case 1 — Nearly linearly separable data: A single straight line can almost separate blue dots from maroon dots. A curve would do even better, but a linear boundary nearly works. One perceptron is almost enough.

Case 2 — Moderately complex data: The decision boundary between red and green points becomes wavy. You need a polynomial boundary — curves that bend multiple times. A single line cannot capture this. You need hidden layers to learn the bends.

Case 3 — Multi-region data: The space divides into four regions — A, B, C, D. Regions A and B belong to one class; C and D to another. A single line cannot distinguish four regions. You need at least two decision boundaries working together, which means multiple neurons in one layer combining their signals.

Case 4 — Arbitrarily complex yellow-region classification: Label a point as class. 1 if it falls inside a yellow polygon bounded by five straight lines. Label it as class 0 otherwise. Each line is one decision boundary. A single perceptron with a threshold unit can model one such line. So five perceptrons in one hidden layer each learn one of the five boundaries. But that is not enough — you need to combine them. A second hidden layer takes the outputs from the five boundary-detectors and makes the final yes/no call. This two-layer hierarchy is what depth gives you.

Case 5 — Two disjoint yellow regions: Now class 1 covers two separate polygons. Each polygon needs its own set of boundary detectors. The second layer detects "region 1" or "region 2." A simple OR logic across those. Two detectors gives the final answer: label 1 if the point falls in either region.

Think of a car assembly line. The first station attaches the chassis. The next adds the wheels. Then the engine, the doors, the paint. No single worker builds a car alone. Each station builds on the work of the station before it. A deep network works the same way: early layers detect edges. Middle layers group edges into shapes. Late layers combine shapes into objects. The analogy breaks where the network can learn in parallel across neurons, unlike a strictly sequential assembly line.

The key insight: as patterns become more complex, you build hierarchical features. Lower layers detect simple edges or boundaries. Upper layers combine those into regions. Even higher layers combine regions into concepts.

This is the intuition behind depth and width.

Depth (number of hidden layers) lets you build hierarchy. Simple features become complex patterns. Those patterns become concepts.
Width (number of neurons per layer) lets you capture more features at a given abstraction level.

Scope: A shallow network with enormous width can represent any function — this is the Universal Approximation Theorem. But the number of neurons required becomes impractical. A deeper network distributes that capacity across layers, making learning more efficient. The catch: depth brings vanishing-gradient risks (explored in Section 7.8).

Visual Intuition: Imagine a 2D plot where the x-axis is "input feature 1" and the. Y-axis is "input feature 2." Now draw the decision boundaries from each case on top. Case 1: one straight line. Case 2: one wavy curve. Case 3: two intersecting lines creating four regions. Case 4: five lines forming a polygon. Case 5: two separate polygons. The progression shows that deeper networks can carve the input space into more intricate regions. The number of linear regions a network can create grows with both depth and width.

Common pitfalls: (1) Thinking "more layers always better" — they are, but only up to a point. Beyond that, vanishing gradients and overfitting hurt you. (2) Confusing depth with number of parameters — a wide shallow network can have more parameters than a deep narrow one. (3) Assuming one hidden layer is always enough — yes, for representing any function in theory. But learning that function with one layer needs exponentially many neurons.

Depth builds a ladder of abstraction. Each rung transforms raw data into something a little more meaningful. Width gives you more hands on each rung. Together they let the network learn complex patterns without needing an impossibly large single layer.

Real-world & domain connection: The hierarchical feature learning of deep networks is why they dominate computer vision. A face-recognition system's early layers might detect edges and color blobs, middle layers detect eyes and noses, And late layers detect entire faces. This mirrors what neuroscientists have observed in the visual cortex — The brain itself processes vision in a hierarchy of increasingly abstract representations.

7.2 Designing a Deep Feed-Forward Neural Network Architecture

You have a pile of wood, nails, and tools. How many rooms should your house have? How wide should each room be? That is architecture design. There is no universal blueprint — but there are patterns that work.

7.2.1 The Four Components

Every deep feed-forward network has four design components:

Data — what you feed in (tabular, image, text, etc.)
Architecture — number of layers, number of units per layer
Loss function — what you optimize
Learning process — how weights get updated (SGD, mini-batch, batch)

The application determines what you can fix upfront. The number of nodes in the input layer comes from your feature count. The number of nodes in the output layer and its activation function come from the task type. But everything between — how many hidden layers, how many neurons in each — must be determined through experimentation. No universal rule exists.

A deep feed-forward network is fully defined by four choices: what data enters. How the layers are arranged. What the network is minimizing. And how it updates its weights. Two of these (input size, output size) are fixed by your data and task. The rest — the hidden architecture — is what you design.

7.2.2 Three Common Architecture Patterns

Equal-width architecture: Every hidden layer has the same number of neurons. If layer 1 has 4 neurons, layer 2 also has 4. This pattern is often preferred for time-series data where you are predicting trends. The consistency helps the network maintain a steady representational capacity through all layers.

Pyramid architecture: The number of neurons decreases as you go deeper. The first hidden layer (closest to input) has the most neurons; later layers have progressively fewer. This is common for classification tasks. Early layers extract many distinctive features from raw inputs; later layers combine those features into fewer, more abstract representations. Visualized from input to output, the layer widths form a pyramid shape.

Hourglass architecture (encoder-decoder): The width first decreases, then increases. The narrowest point in the middle represents a compressed representation of the input. You use this when the goal is to understand and compress data, then reconstruct or generate something from that compressed form. Autoencoders use this pattern. The first half (encoder) compresses; the second half (decoder) reconstructs. You will encounter hourglass architectures in more advanced courses on autoencoders and generative models.

When to pick which:

Pattern	Shape	Best for
Equal-width	Same neurons every layer	Time-series, trend prediction
Pyramid	Widest near input, narrows toward output	Classification with hierarchical features
Hourglass	Wide → narrow → wide	Compression, reconstruction, generation

Think of these patterns as three different kitchen layouts. Equal-width: every counter is the same length — steady, predictable, good for repetitive cooking tasks. Pyramid: you start with a wide chopping board, move to a medium mixing bowl, then to a small serving plate — refining as you go. Hourglass: you pour everything into a blender (compress), then pour it back out into individual glasses (reconstruct).

7.2.3 Worked Example: Parameter Counting in an Equal-Width Network

Consider a network for house price prediction:

Input: 3 features (house size in sq m, number of bedrooms, location parameters)
Hidden layer 1: 4 neurons
Hidden layer 2: 4 neurons
Output: 1 neuron (predicted price)

Activation at output: linear (or ReLU to keep prices non-negative, since price cannot be negative).

Now count the total parameters the network learns.

Between input (3 nodes) and hidden layer 1 (4 nodes):

weights = 3 \times 4 = 12

Plus 4 biases (one per neuron in layer 1). Total = $12 + 4 = 16$ .

Between hidden layer 1 (4 nodes) and hidden layer 2 (4 nodes):

weights = 4 \times 4 = 16

Plus 4 biases. Total = $16 + 4 = 20$ .

Between hidden layer 2 (4 nodes) and output (1 node):

weights = 4 \times 1 = 4

Plus 1 bias. Total = $4 + 1 = 5$ .

Total parameters = 16 + 20 + 5 = 41

Sense-check: With 3 inputs, 4+4 hidden neurons, and 1 output, 41 parameters is reasonable. Each parameter is a number the network must learn. 41 numbers from data — this is a small, fast network.

General formula for parameters between layer $i$ (with $n_{i}$ nodes) and layer $i + 1$ (with $n_{i + 1}$ nodes):

parameters = n_{i} \times n_{i + 1} + n_{i + 1}

The $n_{i + 1}$ accounts for the bias terms — one per neuron in the receiving layer.

7.2.4 A Note on Layers, Depth, and Notation

When counting layers, exclude the input layer. A network with one hidden layer and one output layer has two layers total (layer 1 = hidden, layer 2 = output).

When measuring depth, count the flow of information from output to input. In the example above, from the output layer, the first hidden layer is one level deep. The input is two levels deep. So the depth is two.

Notation conventions (used throughout):

$Z^{l}$ — the pre-activation value at layer $l$ (weighted sum plus bias, before applying activation).
$A^{l}$ — the post-activation output at layer $l$ (after applying activation to $Z^{l}$ ).
$W^{l}$ — the weight matrix between layer $l - 1$ and layer $l$ .
$b^{l}$ — the bias vector for layer $l$ .

So $Z^{3}$ means the pre-activation output at the third layer (after input). $A^{2}$ means the activated output at the second layer.

Visual Intuition: Draw a vertical stack of rectangles — input at top, then hidden layer 1, hidden layer 2, output at bottom. Between each pair, draw arrows for weights. Label the arrows with the shape of the weight matrix. For 3→4→4→1, the arrow labels are $W^{1} \in R^{3 \times 4}$ , $W^{2} \in R^{4 \times 4}$ , $W^{3} \in R^{4 \times 1}$ . Count the arrows plus the small bias nodes on each layer — that gives 41 total learnable numbers.

Common pitfalls: (1) Forgetting to count biases — each neuron in a layer adds exactly one bias parameter. (2) Mixing up the order when multiplying weight shapes: between layer with $n_{i}$ nodes and layer with $n_{i + 1}$ nodes, $W$ has shape $n_{i} \times n_{i + 1}$ — rows match sending layer. Columns match receiving layer. (3) Counting the input layer as a "layer" — it provides data but has no weights or biases of its own.

Architecture design fixes the skeleton of your network. Input size and output size come from your data and task. Everything in the hidden middle — depth, width, and pattern — is chosen through experimentation. The parameter-count formula lets you check how big your skeleton really is.

Real-world & domain connection: The equal-width architecture appears in transformer models for language processing. — each encoder and decoder layer maintains the same dimension (e.g., 512 or 768). Pyramid architectures power image classifiers like VGGNet, where deep layers have fewer channels but more abstract features. Hourglass architectures drive image segmentation (U-Net) and generative models (VAEs), where compressing and then expanding the representation is the core idea.

7.3 Forward Propagation: A Full Numerical Example

You put ingredients into a machine. Numbers flow through a pipeline of weighted sums and activation switches. At the far end, a prediction pops out. Forward propagation is just that pipeline — running your input through the network to get an answer.

7.3.1 Network Setup

Consider a network with:

Input layer: 2 features ( $x_{1} = 1$ , $x_{2} = 0.5$ )
Hidden layer 1: 3 neurons
Output layer: 2 neurons

This example uses a single training instance for illustration. In practice, you would batch multiple instances.

7.3.2 Symbol Registry — Forward Propagation Example

$x$ — input feature vector — $x \in R^{2}$
$W^{1}$ — weight matrix between input and hidden layer 1 — $W^{1} \in R^{3 \times 2}$
$b^{1}$ — bias vector for hidden layer 1 — $b^{1} \in R^{3}$
$Z^{1}$ — pre-activation at hidden layer 1 — $Z^{1} \in R^{3}$
$A^{1}$ — post-activation at hidden layer 1 (after ReLU) — $A^{1} \in R^{3}$
$W^{2}$ — weight matrix between hidden layer 1 and output — $W^{2} \in R^{2 \times 3}$
$b^{2}$ — bias vector for output layer — $b^{2} \in R^{2}$
$Z^{2}$ — pre-activation at output layer — $Z^{2} \in R^{2}$
$A^{2}$ — post-activation at output layer (after Sigmoid) — $A^{2} \in R^{2}$
$y$ — true label — $y \in {0, 1}^{2}$ (one-hot encoded for two classes)

Think of forward propagation like a postal sorting system. A letter (input x) arrives at the first sorting station. Workers there apply routing rules (W¹ weights and b¹ bias) to assign it to a bin (Z¹). Then a supervisor (ReLU activation) discards any negative routing scores and sends only positive ones forward (A¹). The next station repeats the process with its own rules (W², b²) and a final stamp (Sigmoid). This converts the score into a delivery probability (A² = ŷ).

7.3.3 Step-by-Step Forward Pass

Step 1 — From input to hidden layer 1:

Compute the pre-activation by multiplying the weight matrix with the input and adding the bias:

Z^{1} = W^{1} \cdot x + b^{1}

The weight matrix $W^{1}$ has shape $3 \times 2$ . Each row corresponds to one neuron in the hidden layer; each column corresponds to one input feature. For instance, $w_{21}^{1} = 0.2$ is the weight from the second input feature to the first hidden neuron. The bias $b^{1}$ is a separate vector of 3 values — one per hidden neuron.

Parameter count between input and hidden layer 1: $3 \times 2 = 6$ weights plus 3 biases = 9 parameters.

Step 2 — Apply ReLU activation:

ReLU (Rectified Linear Unit) is defined as:

ReLU (z) = max (0, z)

If the input is positive, it passes through unchanged. If the input is negative, it outputs zero.

Applying ReLU to $Z^{1}$ :

A^{1} = ReLU (Z^{1})

After applying ReLU, any negative values in $Z^{1}$ become zero. This introduces nonlinearity — without it, stacking linear layers would just be one big linear transformation.

Step 3 — From hidden layer 1 to output layer:

Z^{2} = W^{2} \cdot A^{1} + b^{2}

The weight matrix $W^{2}$ has shape $2 \times 3$ . There are 2 output neurons, each receiving signals from all 3 hidden neurons. Parameter count: $3 \times 2 = 6$ weights plus 2 biases = 8 parameters.

Step 4 — Apply Sigmoid activation at the output layer:

A^{2} = σ (Z^{2}) = \frac{1}{1 + e ^{- Z^{2}}}

Sigmoid squashes each output value into the range $(0, 1)$ . This produces a probability-like output. It is the right choice for the output layer when each output must be a probability of belonging to a category.

Total parameters learned by this network: $9 + 8 = 17$ .

Full numerical trace — forward propagation with concrete values:

Let the weights and biases be:

W^{1} = 0.2&0.8 0.5&0.3 0.9&0.1, b^{1} = - 0.1 0.4 - 0.3, x = [1 0.5]

Step 1 — Compute $Z^{1}$ :

Z^{1} = W^{1} x + b^{1} = 0.2 (1) + 0.8 (0.5) - 0.1 0.5 (1) + 0.3 (0.5) + 0.4 0.9 (1) + 0.1 (0.5) - 0.3 = 0.2 + 0.4 - 0.1 0.5 + 0.15 + 0.4 0.9 + 0.05 - 0.3 = 0.5 1.05 0.65

Step 2 — Apply ReLU:

A^{1} = ReLU (Z^{1}) = max (0, 0.5) max (0, 1.05) max (0, 0.65) = 0.5 1.05 0.65

All three values are positive, so ReLU passes them through unchanged.

Step 3 — Compute $Z^{2}$ :

W^{2} = [0.3&0.2&0.5 0.4&0.6&0.1], b^{2} = [- 0.2 0.1]

Z^{2} = W^{2} A^{1} + b^{2} = [0.3 (0.5) + 0.2 (1.05) + 0.5 (0.65) - 0.2 0.4 (0.5) + 0.6 (1.05) + 0.1 (0.65) + 0.1]

Z^{2} = [0.15 + 0.21 + 0.325 - 0.2 0.20 + 0.63 + 0.065 + 0.1] = [0.485 0.995]

Step 4 — Apply Sigmoid:

A^{2} = [σ (0.485) σ (0.995)] = [\frac{1}{1 + e ^{- 0.485}} \frac{1}{1 + e ^{- 0.995}}] \approx [0.619 0.730]

The network predicts: class A with 61.9% confidence, class B with 73.0% confidence. The true label is $y = [1, 0]$ , meaning it is class A. The network leans slightly toward class A. But it is not very confident — a loss value will quantify exactly how wrong it is.

7.3.4 Output Layer Design by Task Type

The output layer activation depends on the problem:

Regression (e.g., house price): 1 output node with linear activation (identity). If you need only positive outputs, ReLU is an option.
Binary classification: 1 output node with Sigmoid.

The output gives the probability of the positive class. If the output $\geq 0.5$ , predict class 1; otherwise, predict class 2. You do not need two output nodes for binary classification.

Multi-class classification ( $K$ classes): $K$ output nodes with Softmax.

Each node gives the probability of one class, and all probabilities sum to 1. The classes are mutually exclusive.

Multi-label classification: Multiple output nodes, each with Sigmoid.

Each node independently predicts whether the instance belongs to that label. Labels are not mutually exclusive.

7.3.5 Computational Graph — Forward Propagation

The forward pass can be represented as a computational graph showing the flow from input to output:

x \to Z^{1} = W^{1} x + b^{1} \to A^{1} = ReLU (Z^{1}) . \to Z^{2} = W^{2} A^{1} + b^{2} \to A^{2} = σ (Z^{2}) = y^

This graph will be reversed during backpropagation. Every node in the forward graph participates in the backward computation of gradients.

Visual Intuition: Draw a flowchart. Start with a box labeled x (2 values). Draw an arrow to the next box: Z¹ = W¹x + b¹ (3 values). Arrow to A¹ = ReLU(Z¹) (3 values). Arrow to Z² = W²A¹ + b² (2 values). Arrow to A² = σ(Z²) = ŷ (2 probability values). This is your forward pass. Each box has a shape — the number of values it holds. Arrows between boxes have weight matrices. The backward pass will trace this exact path in reverse.

Common pitfalls: (1) Applying Sigmoid to hidden layers instead of ReLU — Sigmoid saturates and kills gradients in deep networks. Reserve it for the output layer. (2) Using Softmax for binary classification — one Sigmoid node is simpler and equivalent for two classes. (3) Forgetting that the output activation choice changes the loss function you must use — Sigmoid output pairs with binary cross-entropy, Softmax output pairs with categorical cross-entropy. Linear output pairs with MSE.

Forward propagation is a chain of two repeating operations: multiply by weights and add bias, then apply activation. The activation introduces nonlinearity. The output activation converts raw scores into the right format for your task — probabilities for classification, real numbers for regression.

Real-world & domain connection: Forward propagation in a neural network is essentially the same computation as evaluating a complex mathematical model. In production systems like real-time object detection (YOLO) or speech recognition (Whisper), forward propagation runs hundreds of times per second on GPUs. The computational graph abstraction used here is also what frameworks like TensorFlow and PyTorch build internally — They construct a graph of operations, then optimize its execution and compute gradients automatically (autograd).

7.4 Computing the Loss

You guessed a number. The true answer is something else. How wrong were you? The loss function is the scoreboard — it turns "kinda wrong" into a single number that the network can learn from.

7.4.1 Binary Cross-Entropy Loss

For classification tasks where the output layer uses Sigmoid, use binary cross-entropy loss:

L = - [y \cdot lo g (y^) + (1 - y) \cdot lo g (1 - y^)]

where:

$y$ is the true label (0 or 1)
$y^$ is the predicted probability (output of Sigmoid)

The loss function must be differentiable so gradients can flow backwards.

Binary cross-entropy measures the distance between two probability distributions — the true distribution (y is 0 or 1 with certainty). And the predicted distribution (ŷ is between 0 and 1). It is the negative log-likelihood for a Bernoulli distribution. The closer ŷ is to y, the smaller the loss. When ŷ exactly matches y, loss hits zero.

Why the log? The log turns products into sums (easier to differentiate) and heavily penalizes confident wrong answers. If y=1 and the network predicts ŷ=0.001, the term $- lo g (0.001) \approx 6.9$ is huge — the network gets a strong correction signal.

Think of binary cross-entropy as a teacher grading a yes/no quiz. If the correct answer is "yes" (y=1) and the student says "probably yes" (ŷ=0.9), the penalty is small. If the student says "definitely no" (ŷ=0.01), the penalty is huge. The log amplifies the punishment for confident mistakes — being confidently wrong is much worse than being uncertain.

7.4.2 Worked Example: Loss Calculation

Given:

True label: $y = [1, 0]$ (the instance belongs to class A, not class B)
Predicted output (after Sigmoid): $y^= [0.646, 0.494]$

For the first output node (class A, $y_{1} = 1$ , $y^_{1} = 0.646$ ):

L_{1} = - [1 \cdot lo g (0.646) + (1 - 1) \cdot lo g (1 - 0.646)] = - lo g (0.646)

The second term vanishes because $(1 - 1) = 0$ .

For the second output node (class B, $y_{2} = 0$ , $y^_{2} = 0.494$ ):

L_{2} = - [0 \cdot lo g (0.494) + (1 - 0) \cdot lo g (1 - 0.494)] = - lo g (0.506)

The first term vanishes because $0 \cdot lo g (0.494) = 0$ .

The total loss is the sum: $L = - lo g (0.646) - lo g (0.506)$ .

- lo g (0.646) & \approx 0.437 - lo g (0.506) & \approx 0.681 L & = 0.437 + 0.681 = 1.118

Wait — the professor's example earlier said the loss is about 0.560. Let me reconcile this. The professor likely summed the two binary cross-entropy terms and then averaged across the two output nodes. If we average:

L_{avg} = \frac{1.118}{2} \approx 0.559

This matches the professor's 0.560 (rounding). The convention varies: some sum, some average. In most frameworks (PyTorch, TensorFlow), the default is to average over output nodes and batch. Follow whichever convention your course materials specify.

Sense-check: The prediction for class A was 0.646 when the true label was 1 — off by about 0.35. This gives a moderate loss contribution. For class B, prediction was 0.494 when true was 0 — off by about 0.49, giving a similar contribution. The combined loss of ~0.56 is moderate. The network is in the right ballpark but could be much better.

Common pitfalls: (1) Using MSE loss for classification — MSE works for regression but gives weak gradients for classification. Cross-entropy is the right tool for probability outputs. (2) Confusing binary cross-entropy (one Sigmoid output for 2 classes) with categorical cross-entropy (K Softmax outputs for K classes). (3) Applying the loss to raw logits (Z²) instead of the activated outputs (A²=σ(Z²)) — the values will be negative. They produce NaNs from log of negatives.

The loss is the network's report card. Binary cross-entropy converts a probability prediction and a true label into a single number. That number controls how much every weight in the network gets adjusted during backpropagation.

Real-world & domain connection: Cross-entropy loss is not specific to neural networks. It comes from information theory, Where it measures the average number of bits needed to encode events from one distribution using a code optimized for another. In practice, binary cross-entropy is used in spam detection, medical diagnosis (disease present? yes/no), and any binary decision system. Multi-class cross-entropy powers every image classifier that distinguishes between dog, cat, bird, etc.

7.5 Backward Propagation and Gradient Computation

Forward propagation gave you a prediction. Now you need to figure out: which weights are to blame for the error, and by how much? Backpropagation answers that — it traces the error backwards through the network and assigns credit (or blame) to every weight.

7.5.1 The Delta (Error) at the Output Layer

For binary cross-entropy loss with Sigmoid output, The derivative of the loss with respect to the output simplifies to a beautifully clean form:

\frac{\partial L}{\partial y ^} = y^- y

In words: the error at a given output node is simply the predicted value minus the true value. This is the delta — the adjustment signal that will flow backward through the network.

The delta $δ = y^- y$ is not pulled from thin air. Here is the derivation. For binary cross-entropy:

L = - [y lo g (y^) + (1 - y) lo g (1 - y^)]

Differentiate with respect to $y^$ :

\frac{\partial L}{\partial y ^} = - [y \cdot \frac{1}{y ^} + (1 - y) \cdot \frac{- 1}{1 - y ^}] = - \frac{y}{y ^} + \frac{1 - y}{1 - y ^}

Now multiply by the Sigmoid derivative $\frac{\partial y ^}{\partial z} = y^(1 - y^)$ (where $z = Z^{2}$ ):

\frac{\partial L}{\partial z} = \frac{\partial L}{\partial y ^} \cdot \frac{\partial y ^}{\partial z} = (- \frac{y}{y ^} + \frac{1 - y}{1 - y ^}) \cdot y^(1 - y^)

= - y (1 - y^) + (1 - y) y^= - y + y y^+ y^- y y^= y^- y

The Sigmoid derivative cancels the complex fraction and leaves the simple difference. This is why the binary cross-entropy + Sigmoid pair is so elegant — the math simplifies to "prediction minus truth."

For the example above:

Node A: $δ^{A} = 0.646 - 1 = - 0.354$
Node B: $δ^{B} = 0.494 - 0 = 0.494$

These two delta values are the error components at the output layer. Node A was too low — negative delta means increase the prediction. Node B was too high — positive delta means decrease the prediction.

7.5.2 Propagating Errors Backward

Each delta propagates backward through the links that fed into that output node. The delta from node A adjusts the weights on all links entering. Node A — three links from hidden layer 1, plus the bias. Similarly, the delta from node B adjusts the links entering node B.

Now you must propagate errors one more level back. Go from the output layer to the hidden layer. Then the weights between input and hidden layer 1 also get adjusted. This needs the chain rule.

7.5.3 The Chain Rule for Multiple Layers

For the weights in the first layer ( $W^{1}$ ), the gradient involves three nested levels:

\frac{\partial L}{\partial W ^{1}} = \frac{\partial L}{\partial A ^{2}} \cdot \frac{\partial A ^{2}}{\partial Z ^{2}} . \cdot \frac{\partial Z ^{2}}{\partial A ^{1}} \cdot \frac{\partial A ^{1}}{\partial Z ^{1}} \cdot \frac{\partial Z ^{1}}{\partial W ^{1}}

$\frac{\partial L}{\partial A ^{2}}$ — derivative of loss with respect to the output activation (the delta at the output layer)
$\frac{\partial A ^{2}}{\partial Z ^{2}}$ — derivative of Sigmoid activation: $σ^{'} (Z^{2}) = A^{2} (1 - A^{2})$
$\frac{\partial Z ^{2}}{\partial A ^{1}}$ — the weight matrix $W^{2}$
$\frac{\partial A ^{1}}{\partial Z ^{1}}$ — derivative of ReLU activation: 1 if $Z^{1} > 0$ , 0 otherwise
$\frac{\partial Z ^{1}}{\partial W ^{1}}$ — the input $x$

The chain rule multiplies these together. The error component from layer 2 gets broken down and distributed to layer 1.

Picture a river splitting into tributaries. Water flows downhill (forward pass). Now imagine tracing each drop of water back to its source (backward pass). The chain rule does exactly that — it breaks the total error into contributions from each upstream weight. Each weight gets blamed in proportion to how much it contributed to the wrong answer.

7.5.4 Weight Updates

Once you have the gradient for each weight, you update it:

W_{new} = W_{old} - η \cdot \frac{\partial L}{\partial W}

where $η$ is the learning rate.

For example, suppose a particular weight was $W = 0.4$ . The computed gradient says adjust by $- 0.248$ . The new weight becomes $0.4 - (- 0.248) = 0.648$ . This assumes $η = 1$ for the illustration.

Every single weight in the network — all 17 of them in this example — gets its own adjustment equation. At each epoch, every weight is nudged in the direction that reduces the loss.

Complete one-epoch weight update trace:

Continuing from Section 7.3's forward pass where:

$x = [1, 0.5]^{T}$
$A^{1} = [0.5, 1.05, 0.65]^{T}$
$A^{2} = [0.619, 0.730]^{T}$
True $y = [1, 0]^{T}$

Step 1 — Output delta: $δ^{2} = A^{2} - y = [0.619 - 1, 0.730 - 0] = [- 0.381, 0.730]^{T}$

Step 2 — Gradient for $W^{2}$ : $\frac{\partial L}{\partial W ^{2}} = δ^{2} \cdot (A^{1})^{T}$

\frac{\partial L}{\partial W ^{2}} = [- 0.381 0.730] [0.5&1.05&0.65] = [- 0.1905& - 0.400& - 0.2476 0.365&0.7665&0.4745]

Each entry is the gradient for the corresponding weight in $W^{2}$ .

Step 3 — Propate delta to hidden layer: $δ^{1} = ((W^{2})^{T} \cdot δ^{2}) ⊙ ReLU^{'} (Z^{1})$ Since all $Z^{1}$ values were positive, $ReLU^{'} (Z^{1}) = [1, 1, 1]^{T}$ .

δ^{1} = 0.3&0.4 0.2&0.6 0.5&0.1 [- 0.381 0.730] = 0.3 (- 0.381) + 0.4 (0.730) 0.2 (- 0.381) + 0.6 (0.730) 0.5 (- 0.381) + 0.1 (0.730) = 0.1777 0.3618 - 0.1175

Step 4 — Update weights (with $η = 0.1$ ): Each weight $w_{ij}$ is updated: $w_{ij}^{new} = w_{ij}^{old} - η \cdot \frac{\partial L}{\partial w _{ij}}$

For the first entry of $W^{2}$ (was 0.3, gradient = -0.1905):

w_{11}^{new} = 0.3 - 0.1 \cdot (- 0.1905) = 0.3 + 0.01905 = 0.319

All 17 parameters get similar proportional nudges. The network moves a tiny step in the right direction.

Sense-check: The output delta had mixed signs (node A negative, node B positive), which makes sense — node A needs to increase toward 1, node B needs to decrease toward 0. The gradient magnitudes are small, so with a learning rate of 0.1, the updates are gentle — the network learns steadily without overshooting.

7.5.5 Computational Graph — Backward Propagation

Just as forward propagation has a computational graph flowing forward, backward propagation flows in reverse:

L \to \frac{\partial L}{\partial A ^{2}} \to \frac{\partial L}{\partial Z ^{2}} \to \frac{\partial L}{\partial W ^{2}}, \frac{\partial L}{\partial . B ^{2}} \to \frac{\partial L}{\partial A ^{1}} \to \frac{\partial L}{\partial Z ^{1}} \to \frac{\partial L}{\partial W ^{1}}, \frac{\partial L}{\partial b ^{1}}

The error component at the output splits into adjustments for $W^{2}$ and $b^{2}$ , And simultaneously propagates further back to produce adjustments for $W^{1}$ and $b^{1}$ . The depth of this particular network is 2, so the error propagates across 2 levels.

Visual Intuition: Draw the forward graph from Section 7.3.5 again. Now draw a second graph underneath, flowing right to left. Start with a box labeled $L$ (loss). Arrows flow backward: from $L$ to $δ^{2} = y^- y$ , then branching to $\frac{\partial L}{\partial W ^{2}}$ (updates. For the second weight matrix) and simultaneously to $\frac{\partial L}{\partial A ^{1}}$ (error for. The hidden layer), then to $\frac{\partial L}{\partial Z ^{1}}$ (through ReLU derivative), then to $\frac{\partial L}{\partial W ^{1}}$ (updates for the first weight matrix). The backward graph mirrors the forward graph exactly, just in reverse, with each forward operation replaced by its derivative.

Q: With 2 output nodes but multiple training examples, how does backpropagation work?

A: When you process data in batches (mini-batch or batch gradient descent), you compute the loss across all $N$ instances in the batch, average it. Then propagate that averaged loss backward. If using stochastic gradient descent, you take one instance at a time — compute its loss, backpropagate, and update weights immediately. The mechanism is the same regardless of batch size: the computed (or averaged) loss flows backward through all weights.

Q: Should we use the full loss value (0.56) directly in backpropagation?

A: The loss value 0.56 is the consolidated scalar — it summarizes how wrong the network was. But to assign blame — how much of that loss came from each link — you compute gradients. The loss itself is not directly plugged in; rather, its derivatives with respect to each weight are computed. The delta values are the decomposed error signals. For example, $- 0.354$ and $0.494$ at the output layer tell you how much to adjust each specific weight. The 0.56 is like a team's final score. The deltas are the individual player statistics that explain the score.

Common pitfalls: (1) Forgetting the activation derivative when backpropagating through a layer — the chain rule requires multiplying by $σ^{'} (z)$ for Sigmoid or $ReLU^{'} (z)$ for ReLU. (2) Mixing up the order of multiplication in the chain rule — the gradient at layer l depends on the transpose of W^{l+1}. It is not W^{l+1} itself. (3) Using $y^$ instead of the delta $(y^- y)$ as the error signal — the raw prediction is not the error. The difference from truth is.

Backpropagation is the chain rule applied to the entire network. The gradient at any weight is the product of all downstream derivatives along the path from that weight to the loss. The delta at the output ( $y^- y$ for BCE+Sigmoid) is the clean starting signal that then ripples backward through every layer.

Real-world & domain connection: Backpropagation was popularized by Rumelhart, Hinton, and Williams in 1986, though the idea traces back to the 1960s. Every deep learning framework today (PyTorch, TensorFlow, JAX) implements automatic differentiation — a generalization. Of backpropagation that can compute gradients for arbitrary computational graphs, not just feed-forward networks. When you call loss.backward() in PyTorch, you are triggering the same chain-rule machinery described here, applied automatically to your specific network architecture.

7.6 The Effect of Nonlinearity — Interactive Demonstration

What happens if you build a deep network but skip the activation functions? Nothing. Literally nothing. Stacking linear layers is just one big linear layer. Nonlinearity is the whole reason depth works.

7.6.1 Setup

A neural network playground visualization shows how nonlinearity affects learning. The setup uses:

A two-dimensional dataset (two input features)
Varying complexity of data patterns
Configurable network: number of hidden layers, neurons per layer, activation functions
Configurable hyperparameters: learning rate, noise, train/test split ratio, batch size

Three key metrics to watch during training:

Training loss — how well the network fits the training data
Test loss — how well it generalizes to unseen data
Weight thickness — the visualized link weights. Thicker lines mean larger absolute weight values.

7.6.2 Experiment 1 — Linearly Separable Data, No Hidden Layers

With a simple linearly separable dataset (blue vs orange points), the network needs no hidden layers at all. Two input nodes feed directly to one output node with Sigmoid.

Noise = 0: Within fewer than 10 epochs, a clean linear decision boundary perfectly separates blue from orange.
Noise = max: Some blue points appear in the orange region and vice versa.

Still, within 10-15 epochs, the network finds a reasonable boundary. Those misclassified points are outliers — noise in the data.

With linear data and no hidden units, training converges quickly. The decision boundary is a single straight line.

Key insight: For linearly separable data, a single perceptron (logistic regression) is all you need. Adding hidden layers would be overkill.

7.6.3 Experiment 2 — Nonlinear Data, No Hidden Layers

Now switch to a complex dataset where blue and orange points form swirly, nonlinear patterns — think concentric circles or spirals. With no hidden units, the network is just a linear classifier.

Results:

The training loss stays stuck at about 0.5.
Epochs increase, but loss does not decrease.
No decision boundary forms. No weights change.
The network cannot learn nonlinear patterns with only a linear boundary.

7.6.4 Experiment 3 — Nonlinear Data, With Hidden Layers but No Activation

Add hidden layers (say, two hidden layers with a pyramid structure: 4 neurons then 2 neurons). But set activation to linear in all hidden layers.

Results:

Training and test loss saturate very quickly — well below 100 epochs.
No decision boundary emerges.
No weights get updated. The loss stays flat.

Stacking multiple linear layers is equivalent to a single linear layer. Without nonlinear activation functions between layers, no matter how deep the network, it cannot learn nonlinear decision boundaries. The entire network collapses to $y = W_{effective} \cdot x + b_{effective}$ .

Here is the proof. With two linear layers:

A^{1} = W^{1} x + b^{1}, A^{2} = W^{2} A^{1} + b^{2}

Substitute:

A^{2} = W^{2} (W^{1} x + b^{1}) + b^{2} = (W^{2} W^{1}) x + (W^{2} b^{1} + b^{2}) = W_{eff} x + b_{eff}

A composition of linear functions is just another linear function. You can multiply all the $W$ matrices into one and add all the transformed biases. The extra layers add nothing — they just waste computation. You need nonlinearity to break this collapse.

7.6.5 Experiment 4 — Nonlinear Data, Sigmoid Activation

Now add Sigmoid activation to the hidden layers (same 4→2 pyramid architecture).

Results:

Starting from 0.5, the loss slowly decreases: 0.499 after a few hundred epochs, then 0.3, then lower.
With a larger learning rate (e.g., 0.3 instead of 0.03), the descent speeds up.
Around 800 epochs, a clear decision boundary forms. The blue and orange regions are separated.
Around 0.02 training loss and 0.008 or 0.009 test loss, the separation is stable.

Key observation: Sigmoid works, but it learns slowly. Between 3-4 successive trainings where loss barely changes, you can apply early stopping — stop training when improvements become negligible. This saves computation time.

7.6.6 Experiment 5 — Even More Complex Data, Tanh Activation with More Layers

With a dataset of even higher complexity (wider swirls, tighter spirals), Sigmoid with the same 4→2 architecture struggles — Even at 1500 epochs, it cannot cleanly separate all points.

Now add more layers (more depth) and use Tanh activation.

Within 300 epochs, a much better boundary forms than the previous setup even at 1500 epochs.
Within 1000 epochs, a highly complex boundary successfully separates the data.

Takeaway: As data complexity increases, you need both more depth (to build hierarchical features) and a good activation function. More layers help the network construct progressively abstract representations faster than simply widening a single layer would.

7.6.7 How Capacity Affects Overfitting

If you have only 2 simple features but you use 1000 neurons in the first hidden layer, the network will overfit. It has way too much capacity for the problem — it essentially memorizes the training data instead of learning general patterns. It is like "copy-paste" — the machine just copies and mimics the input features rather than understanding them.

The capacity of your model should match the difficulty of the pattern you are trying to learn.

Experiments summary — side-by-side comparison:

Experiment	Data	Hidden Layers	Activation	Result
1	Linearly separable	None	—	Perfect boundary, <10 epochs
2	Nonlinear (spirals)	None	—	Stuck at loss 0.5. Cannot learn
3	Nonlinear	2 (4→2)	Linear (none)	Collapses to single line. No learning
4	Nonlinear	2 (4→2)	Sigmoid	Works slowly. ~800 epochs to converge
5	Very complex	More layers	Tanh	Works much faster. Better boundaries

Key pattern: The jump from Experiment 2 to Experiment 3 shows that layers alone (without activation) don't help. The jump from Experiment 3 to Experiment 4 shows that adding Sigmoid unlocks learning — but slowly. The jump from Experiment 4 to Experiment 5 shows that Tanh + more depth handles harder problems faster.

Visual Intuition: Imagine five plots side by side. Each is a 2D scatter of blue and orange points with the network's decision boundary overlaid. Plot 1: a clean straight line splitting the space. Plot 2: a straight line that fails — half the points on the wrong side. Plot 3: also a straight line — no better than Plot 2 despite having hidden layers. Plot 4: a wavy, curved boundary that slowly wraps around the blue cluster. Plot 5: a tighter, more complex boundary that cleanly traces the spirals. The progression from straight line to tight spiral is the progression from no depth to deep with nonlinearity.

Q: While you were adding layers and changing neuron counts, it looked random. Is there actual logic for choosing the architecture?

A: There is no hard rule or formula that says "for application X with complexity Y, use N layers with M neurons." It must be experimentally determined. There is no closed-form solution. However, practitioners have suggested starting points to avoid random guessing. These are heuristics to get you started. From there, you observe training behavior and tweak.

Q: Could we use polynomial features (e.g., $x_{1}^{2}$ , $x_{2}^{2}$ , $x_{1} x_{2}$ ) alongside a deep network?

A: Yes, you can engineer polynomial features in addition to using hidden layers. This is part of feature engineering from machine learning. It may help the network detect patterns faster with less network complexity. This is especially useful when computational resources are limited.

Common pitfalls: (1) Assuming more neurons always helps — extra capacity without enough data leads to overfitting (memorization, not learning). (2) Using Sigmoid in deep hidden layers without expecting slow training — ReLU or Tanh will almost always be faster. (3) Judging convergence by training loss alone — the gap between training and test loss tells you about overfitting.

Nonlinearity is what makes depth useful. Without it, a 100-layer network is just a 1-layer network in disguise. The choice of activation function determines how fast the network learns and how well it handles complexity. ReLU is fast but can kill neurons; Tanh handles complex boundaries well; Sigmoid works but is slow.

Real-world & domain connection: The interactive demonstration described here is similar to the TensorFlow Playground (playground.tensorflow.org), a widely used educational tool. In production systems, The same principles guide architecture selection: understanding your data's complexity determines whether. You need a simple linear model, a shallow network, or a deep architecture. The experiment progression — Linear data → nonlinear → more complex → overfitted — Mirrors the debugging workflow of a machine learning engineer when a model performs poorly.

7.7 Architecture Design Heuristics — Getting a Starting Point

You stand before a blank canvas. How many hidden layers? How many neurons in each? There is no magic formula — but there are sensible starting lines. Think of these heuristics as the first brushstroke, not the finished painting.

7.7.1 Heuristic for Number of Hidden Layers

Purpose: The hidden-layer heuristic gives you a reasonable upper bound for the number of hidden layers when working with tabular data. It prevents you from starting with a ridiculously deep network for a small dataset.

Inputs: Number of training instances, number of features, number of output classes.

Output: A suggested maximum number of hidden layers to start experimenting with.

max_layers \approx ⌊ lo g_{2} (\frac{instances}{features \times classes}) ⌋ + 1

Steps:

Divide the number of training instances by (features × classes).
Take $lo g_{2}$ of that ratio.
Take the floor (round down to the nearest integer).
Add 1. This is your suggested maximum number of hidden layers.
Start experimenting with architectures between 1 and this maximum.

Worked example — Tabular customer segmentation:

800 instances, 20 features, 3 classes
Input layer: 20 neurons
Output layer: 3 neurons (with Softmax for multi-class classification)

Apply the formula:

lo g_{2} (\frac{800}{20 \times 3}) = lo g_{2} (\frac{800}{60}) = lo g_{2} (13.33) \approx 3.74

Taking the floor: 3. Then +1: maximum suggested hidden layers = 4.

So start experimenting with 1 to 4 hidden layers. This is a starting point, not a rule.

7.7.2 Heuristic for Number of Neurons per Layer with Parameter Constraints

Purpose: Once you pick a number of layers, you need neuron counts per layer. The parameter constraint prevents the model from having more learnable parameters than the data can reasonably support.

Constraint:

total_parameters \leq \frac{training_instances}{c}

where $c$ is a system-dependent constraint factor (e.g., 5). The number of learnable parameters should not exceed, say, one-fifth of the training instances.

Steps:

Pick a candidate number of neurons per hidden layer.
Count total parameters using $n_{i} \times n_{i + 1} + n_{i + 1}$ for each adjacent pair.
Check against the constraint $\frac{instances}{c}$ .
If the count exceeds the limit, reduce neurons. If it is far below, you can increase them.
Repeat until you find a count that satisfies the constraint.

Worked example (continued):

Start with 2 hidden layers: input(20) → hidden(?) → output(3).

Try 40 neurons in the hidden layer:

Input (20) → hidden (40): $20 \times 40 + 40 = 840$
Hidden (40) → output (3): $40 \times 3 + 3 = 123$
Total = 963.

Check against constraint with $c = 5$ : $\frac{800}{5} = 160$ . Limit = 160. 963 >> 160 — way too many.

Try 20 neurons: $20 \times 20 + 20 = 420$ and $20 \times 3 + 3 = 63$ . Total = 483. Still exceeds.

Try 10 neurons: $20 \times 10 + 10 = 210$ and $10 \times 3 + 3 = 33$ . Total = 243. Still exceeds.

Try 6 neurons: $20 \times 6 + 6 = 126$ and $6 \times 3 + 3 = 21$ . Total = 147. Below 160 — acceptable.

So start training with 6 neurons per hidden layer.

Sense-check: With 800 instances and 147 parameters, the data-to-parameter ratio is about 5.4:1. This is reasonable — the model has enough data to learn without overfitting immediately.

7.7.3 Scaling to Image Data: The Design Loop

Consider a digit classification task using raw pixels:

28 × 28 grayscale images → each image flattened to 784 features
5,000 training images
10 output classes (digits 0 through 9)

Start with the same heuristic: $lo g_{2} (5000/ (784 \times 10)) = lo g_{2} (0.637) \approx - 0.65$ . Floor: -1. Then +1 = 0. This heuristic breaks down for high-dimensional data — the formula suggests 0 hidden layers, which is unreasonable for image classification. So you fall back to experimentation.

Try 2 hidden layers with a pyramid structure. Since GPU computation is optimized for powers of 2, choose neuron counts accordingly. For example:

Hidden layer 1: 128 neurons
Hidden layer 2: 64 neurons

Between input (784) and hidden 1 (128): $784 \times 128 + 128 = 100, 480$ . Between hidden 1 (128) and hidden 2 (64): $128 \times 64 + 64 = 8, 256$ . Between hidden 2 (64) and output (10): $64 \times 10 + 10 = 650$ . Total = 109,386 parameters.

If you try larger values, the total soars. For hidden 1 = 512 and hidden 2 = 256, it reaches about 3.7 million. That is far more than reasonable for 5,000 training examples. So you iteratively reduce: 256→128, 128→64, 64→32, 32→16.

If further reduction does not help and loss won't decrease, consider other options. You can reduce the number of layers — go from 2 hidden layers to 1. You can use regularization techniques, covered after midterm. These let large architectures work by preventing all weights from being learned in every iteration. You can also use feature engineering like polynomial features. This helps detect patterns with less network complexity.

7.7.4 Scaling to Text Data

For very large unstructured textual datasets:

Thousands of engineered features per instance (statistical NLP features)
50,000 training examples
5 sentiment classes

The same iterative design process applies: use the heuristics as a starting point,. Count parameters, check against compute constraints, adjust neuron counts and layer counts through experimentation, And Fall back to regularization when needed.

7.7.5 Summary: General Design Heuristics

Steps — the design loop:

Start with 2 to 3 hidden layers and 10 to 100 hidden units per layer, depending on problem scale.
If the model underfits (both training and test loss are high), increase complexity: more hidden layers or more neurons per layer.
If the model overfits (training loss very low but test loss shoots up), reduce complexity: fewer layers or fewer neurons.
More data means better expressive power. If data is scarce, use a smaller network.
For time-series data: prefer equal-width architectures.
For classification tasks involving hierarchical feature extraction: prefer pyramid architectures.
For compression/reconstruction tasks: use hourglass architectures.
Always iterate — train, observe, tweak, repeat.

These are all starting points. The real work begins by training, observing, and iterating.

When to use this heuristic / Alternatives: The $lo g_{2}$ heuristic works for tabular data. It breaks for high-dimensional data like images (where features >> instances). For images, start with known architectures like pyramids with powers-of-2 neuron counts (128, 64, 32). The parameter constraint ( $c = 5$ rule) is a rough guide — sometimes you need more parameters than the rule allows. This is where regularization and dropout come in. For text, the same iterative loop applies, but you often need fewer layers than you think because NLP features are already engineered.

Visual Intuition: Picture the design loop as a flowchart. Start box: "Pick 2-3 layers, 10-100 neurons." Arrow to "Train." Arrow to. A decision box: "Check loss." Three branches: (1) Both losses low → Done. (2) Training low, test high → Overfitting → go left to "Reduce complexity." (3) Both high → Underfitting →. Go right to "Increase complexity." Each branch loops back to "Train." The loop continues until the losses are acceptable.

Q: What is the ideal range for a loss function value?

A: There is no fixed ideal range. You can train until loss reaches near zero, but that is impractical for complex datasets. Instead, observe the saturation point — where loss stops decreasing significantly. But do not judge by training loss alone. Take that model, pause training, and apply it to unseen test data. Compare training and test loss:

If both training and test loss are reasonably low → good, stop.
If training loss is very low but test loss is much higher (e.g., 0.5 vs 10 or 20) → overfitting. You are memorizing the training data.
If both training and test loss are high (e.g., both around 10) → underfitting. Increase model complexity.
If the model is saturating at a given loss (say 0.5) and will not go lower, stop. Get the model, check test performance, and decide accordingly.

Architecture design is iterative, not prescriptive. The heuristics give you a reasonable first guess. From there, you watch the loss curves and activation patterns, then adjust. The design loop — start small, check, expand or shrink — is the real algorithm.

Real-world & domain connection: The iterative architecture design loop described here is exactly how practitioners work. Tools like Keras Tuner and Optuna automate this loop using Bayesian optimization and hyperband algorithms. Companies like Google and Meta use Neural Architecture Search (NAS) — an approach. Where a meta-algorithm automatically discovers optimal architectures, sometimes finding designs that outperform human-designed networks. Neuro-evolution, mentioned in the lecture, is one flavor of NAS that uses genetic algorithms.

7.8 Vanishing and Exploding Gradient Problems

Imagine whispering a message through a chain of 50 people. By the time it reaches the last person, the message is either lost (vanished) or distorted beyond recognition (exploded). This is what happens to the gradient in a deep network.

7.8.1 The Problem

When gradients are backpropagated through many layers, they are multiplied by. The weight matrices and the derivatives of activation functions at each layer. Two failure modes emerge:

Vanishing gradient: Repeated multiplication by numbers less than 1 causes the gradient to shrink exponentially toward zero.

Early layers (near the input) receive effectively no update signal. They stop learning.

Exploding gradient: Repeated multiplication by numbers greater than 1 causes the gradient to grow exponentially.

Weights diverge; the loss oscillates or goes to infinity.

Think of a bank account with compound interest. If the interest rate is 0.5 (50% loss each step), your money shrinks to nothing — vanishing. If the rate is 2.0 (doubling each step), it explodes. The gradient in a deep network compounds the same way — But the rate differs at each layer. It depends on weights and activation derivatives.

7.8.2 Why Sigmoid Causes Vanishing Gradients

Look at the Sigmoid function: $σ (z) = \frac{1}{1 + e ^{- z}}$ .

For large positive $z$ (e.g., 10, 2000, 20000), $σ (z) \approx 1$ . The function saturates — all large values get squashed to 1. Similarly, large negative values get squashed to 0.

The derivative of Sigmoid is:

σ^{'} (z) = σ (z) \cdot (1 - σ (z))

Plot this derivative. It peaks at $z = 0$ (where $σ^{'} (0) = 0.25$ ) and rapidly approaches zero as $∣ z ∣$ exceeds about 5. For $∣ z ∣ > 5$ , the gradient is essentially zero.

Why the derivative never exceeds 0.25: At $z = 0$ , $σ (0) = 0.5$ . Then $σ^{'} (0) = 0.5 \cdot (1 - 0.5) = 0.5 \cdot 0.5 = 0.25$ . This is the maximum. For any other $z$ , either $σ (z)$ or $1 - σ (z)$ is less than 0.5, so the product is always less than 0.25. This means every Sigmoid layer can multiply the gradient by at most 0.25 — and usually less. Stack enough layers with weights also below 1, and the gradient vanishes.

Consequence: A hidden neuron receiving inputs that produce large $∣ z ∣$ values will have a near-zero gradient. When this happens across many layers, the error signal vanishes before it reaches early layers.

Why this matters for learning: If one neuron's pre-activation is 10 and another's is 2000, both produce Sigmoid ≈ 1. The difference between them — which carries important information — is lost. The network cannot distinguish between moderately strong signals and extremely strong signals.

7.8.3 Worked Example: Vanishing Gradient with Sigmoid

Consider a 4-layer network with Sigmoid in all hidden layers.

Assume (for illustration) these values. The spectral norm of every weight matrix is 0.8. The maximum Sigmoid derivative is 0.25, which occurs at $z = 0$ .

The spectral norm of a weight matrix $W$ roughly tells you how much that matrix stretches a vector. It is a compressed summary of the matrix's magnitude.

Let the initial gradient at the output layer be 1.0. Propagate it backward step by step.

At layer 3 (one step back): multiply by the weight spectral norm (0.8) and the Sigmoid derivative (0.25).

$\nabla_{3} = 1.0 \times 0.8 \times 0.25 = 0.2$

At layer 2 (two steps back): propagate $\nabla_{3}$ through the same factors.

$\nabla_{2} = 0.2 \times 0.8 \times 0.25 = 0.04$

At layer 1 (three steps back):

$\nabla_{1} = 0.04 \times 0.8 \times 0.25 = 0.008$

After just 3 layers of backpropagation, the gradient has shrunk from 1.0 to 0.008 — a reduction of over 99%. In a 10-layer network, this number becomes $1.0 \times (0.8 \times 0.25)^{10} \approx 0.0000001$ — essentially zero.

The compounding factor: Each backpropagation step multiplies by at most $0.8 \times 0.25 = 0.2$ . After $k$ steps, the gradient is at most $0. 2^{k}$ of its original size.

Layers back (k)	Gradient fraction
1	0.2
2	0.04
3	0.008
5	0.00032
10	~0.0000001

Sense-check: This is a worst-case scenario — in practice, weight norms and activation derivatives vary, and not all paths shrink equally. But the pattern is real: deeper networks with Sigmoid will have gradients that decay exponentially in the number of layers. The early layers effectively receive zero learning signal.

What this means: The layers near the output get reasonable adjustment signals and learn. But layers near the input receive almost no signal. Their weights barely change. You see saturation — training and test loss plateau, no patterns get predicted, and no weights update in early layers. This is the vanishing gradient problem in action.

7.8.4 The General Form

For a network with $L$ layers, the gradient at layer $l$ is:

\frac{\partial L}{\partial W ^{l}} = \frac{\partial L}{\partial A ^{L}} \cdot k = L \prod l + 1 (\frac{\partial A ^{k}}{\partial Z ^{k}} \cdot W^{k}) \cdot \frac{\partial Z ^{l}}{\partial W ^{l}}

The product term $\prod$ is the culprit. If each factor is less than 1 (as with Sigmoid), the product shrinks exponentially with depth. If each factor is greater than 1, it explodes.

Visual Intuition: Draw a plot where the x-axis is "layers back from output". (0 to 10) and the y-axis is "gradient magnitude" on a log scale. Plot three lines: (1) Sigmoid with small weights — starts at 1.0, drops exponentially to ~0.0000001 by layer 10. This is the vanishing gradient. (2) Weights near 1 with ReLU — stays roughly constant around 1.0. This is healthy learning. (3) Weights above 1 with linear activation — starts at 1.0, grows exponentially to millions by layer 10. This is the exploding gradient. The ReLU line is what you want — steady gradient flow.

Common pitfalls: (1) Using Sigmoid in all hidden layers of a very deep network and wondering why early layers don't learn — this is the textbook vanishing gradient. Switch to ReLU. (2) Assuming all gradients vanish equally — some paths through the network may have larger gradients than others, leading to uneven learning. (3) Confusing the vanishing gradient with underfitting — vanishing gradients cause early layers to not learn at all. Underfitting means the architecture is too simple even if gradients flow properly.

The vanishing gradient is a compounding problem: each Sigmoid layer can shrink the gradient by up to 75%. Each set of small weights shrinks it further. After a few layers, the error signal is effectively zero for early layers. ReLU solves this because its derivative is 1 for positive inputs — no shrinkage. The exploding gradient is the opposite problem, solved by gradient clipping and careful weight initialization.

Real-world & domain connection: The vanishing gradient problem was a major bottleneck in deep learning until around 2011-2015. It stopped researchers from training networks deeper than about 3-4 layers for decades. The combination of ReLU activation (Nair & Hinton, 2010), better weight initialization (Glorot/Xavier, 2010; He, 2015), And Batch Normalization (Ioffe & Szegedy, 2015) finally cracked this open. These three innovations — All aimed at maintaining healthy gradient flow — Enabled the training of networks with hundreds of layers (ResNet-152, 2015) and kickstarted the deep learning revolution.

7.9 Activation Functions for Hidden Layers

You have five tools on your workbench: ReLU, Leaky ReLU, ELU, Tanh, and Sigmoid. Each does the same job — adding wiggle to your network — but they differ in speed, gradient health, and quirks. Knowing which tool to reach for saves hours of debugging.

7.9.1 ReLU — The Default Starting Point

The Rectified Linear Unit is:

ReLU (z) = max (0, z)

It outputs $z$ if $z > 0$ , and 0 otherwise.

Why ReLU helps with vanishing gradients:

The derivative of ReLU is:

ReLU^{'} (z) = {1& if z > 0 0& if z < 0

For positive inputs, the derivative is exactly 1 — no attenuation. The gradient passes through unchanged. As long as most neurons receive positive pre-activations, the gradient does not shrink when backpropagating through ReLU layers. This is why ReLU largely solves the vanishing gradient problem.

ReLU is also computationally fast: Computing $max (0, z)$ is a simple comparison — much cheaper than computing exponentials (as in Sigmoid or Tanh).

Recommendation: For most deep feed-forward networks, start with ReLU in all hidden layers. It is the default choice for many practitioners.

ReLU has a beautiful simplicity. The forward pass is a binary gate: positive signals pass through, negative signals are blocked. The backward pass is even simpler: the gradient is either 1 (pass it through) or 0 (stop). This binary nature means ReLU layers act like a sparse feature selector — only the "active" neurons contribute to the output. They receive gradient updates. The sparsity can actually help generalization.

Think of ReLU like a bouncer at a club. If your signal is positive (you're on the guest list), you walk straight in. If negative, you're turned away. The gradient is the same — the bouncer either lets the learning signal through (value 1) or blocks it (value 0). The simplicity makes the door fast to manage.

7.9.2 Sigmoid — Output Layer Only

Sigmoid is rarely used in hidden layers of deep feed-forward networks because of the vanishing gradient problem. However, it appears frequently in specialized architectures like recurrent neural networks (which you will encounter later).

For standard feed-forward networks, use Sigmoid only in the output layer. That is the place for binary classification. Squashing to $(0, 1)$ produces a meaningful probability.

7.9.3 Tanh

Tanh squashes inputs to $(- 1, 1)$ . It is zero-centered (unlike Sigmoid, which outputs only positive values). Tanh is frequently used in hidden layers for time-series data and sequential models.

tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}

The derivative is $tanh^{'} (z) = 1 - tanh^{2} (z)$ , with a maximum of 1.0 at $z = 0$ . This is four times larger than Sigmoid's maximum (0.25), so Tanh has milder vanishing gradient issues. The zero-centered output also helps — when all outputs are positive (Sigmoid), the gradients for all weights feeding into the next layer have the same sign. This causes zigzagging optimization.

7.9.4 The Dying ReLU Problem

ReLU has a weakness: if a neuron's pre-activation $z$ is consistently negative, ReLU outputs.

The gradient is also 0. That neuron receives no updates — it is dead.

A dead neuron permanently outputs 0 and contributes nothing to learning.

How to detect it: at each layer, plot a histogram of activation values. If you see a large spike at 0 — meaning many neurons consistently output 0 — you have dying ReLU neurons.

This is more pronounced when the network is deep. As depth increases, it becomes more likely that some neurons will fall into permanently negative regions.

7.9.5 Variations of ReLU That Avoid Dying Neurons

Leaky ReLU:

LeakyReLU (z) = max (α z, z) where α is a small positive constant (e.g., 0.01)

Instead of outputting 0 for negative inputs, it outputs a small fraction of the input. This small positive slope means the gradient is never zero — dead neurons are avoided because there is always some learning signal.

Parametric ReLU (PReLU): Like Leaky ReLU, But $α$ is a learnable parameter rather than a fixed constant. The network learns the optimal slope for negative inputs during training.

Exponential ReLU (ELU): Uses an exponential function for negative inputs:

ELU (z) = {z & if z > 0 α (e^{z} - 1) & if z \leq 0

7.9.6 Activation Function Selection Guidelines

Scenario	Recommendation
Hidden layers — default start	ReLU
Hidden layers — deep network, neurons dying	Leaky ReLU, PReLU, or ELU
Hidden layers — time-series / sequential data	Tanh
Output — regression	Linear (identity)
Output — binary classification	Sigmoid (1 node)
Output — multi-class classification	Softmax (K nodes)
Output — multi-label classification	Sigmoid (multiple nodes)

Why not go straight to Leaky ReLU if it avoids dying neurons? ReLU is faster. Computing $max (0, z)$ is cheaper than computing $max (α z, z)$ with an exponential or parametric variant. Always try the simplest, fastest option first. Only switch if you observe problems — dying neurons or vanishing gradients.

Visual Intuition: Draw a single plot with the x-axis as $z$ (pre-activation, from -3 to +3) and. Overlay five activation curves: (1) ReLU — flat at 0 for negatives, then a 45° line for positives. Sharp corner at z=0. (2) Leaky ReLU — same but with a barely. Visible upward slope for negatives. (3) ELU — smooth curve approaching -1 for. Negatives, 45° line for positives. (4) Tanh — S-curve from -1 to +1,. Symmetric around origin. (5) Sigmoid — S-curve from 0 to 1, not zero-centered. The key landmark: ReLU's corner at (0,0) is where neurons live or die.

Common pitfalls: (1) Using Sigmoid in hidden layers of a deep network — guaranteed vanishing gradients. Reserve Sigmoid for the output layer. (2) Using Softmax in hidden layers — Softmax normalizes across neurons, killing independence. Use it only at the output. (3) Ignoring dying ReLU neurons — if 40% of your neurons are dead, you have effectively 40% fewer parameters. Check activation histograms. (4) Using a high learning rate with ReLU — large gradient updates can push many neurons into negative territory permanently. This kills them en masse.

ReLU is the default choice because it fixes vanishing gradients and is fast. If neurons die, switch to Leaky ReLU or ELU. For output layers, match the activation to your task: linear for regression, Sigmoid for binary, Softmax for multi-class. The activation function is not an architectural detail — it determines whether your gradients survive the backward pass.

Real-world & domain connection: ReLU (Nair & Hinton, 2010) was one of the key innovations that enabled the deep learning boom. Before ReLU, training networks deeper than 5 layers was nearly impossible. Today, ReLU and its variants appear in virtually every computer vision model (ResNet, EfficientNet), natural language processing model (transformer feed-forward blocks), And reinforcement learning system. The choice between ReLU variants often comes down to a simple trade-off: compute speed vs. robustness to dying neurons.

7.10 Monitoring Training Progress

Training a neural network is like baking a cake. You cannot just set a timer and walk away — you need to peek through the oven window. Loss curves and activation histograms are your window into the network's brain.

7.10.1 Two Essential Plots

During training, track these two diagnostics:

1. Loss curves (training loss vs test loss over epochs):

Is the loss decreasing? If yes, learning is happening.
Is the loss saturating — decreasing extremely slowly or not at all? The.

Model may have reached its capacity, or there may be an issue with architecture/hyperparameters.

Is the loss decreasing too slowly? Try increasing the learning rate.
Compare training loss and test loss:
Both low → good fit
Training low, test high → overfitting
Both high → underfitting

The loss curve is the single most important diagnostic in deep learning. It answers one question: "Is my network getting better over time?" A healthy loss curve slopes downward and eventually flattens near zero. A flat curve means no learning is happening — check your learning rate, architecture, or data. A diverging curve (training loss goes down, test loss goes up) means overfitting — your network is memorizing the training set.

2. Activation histograms (per layer):

Plot the distribution of pre-activation ( $Z$ ) and post-activation ( $A$ ) values at each hidden layer.
All zeros on a layer → dying ReLU. Switch to Leaky ReLU or reduce depth.
Values consistently at extreme ends (e.g., Sigmoid saturating at 0 or 1) → saturation. Consider different activation or initialization.
Unbounded, wildly varying values (one iteration gives 10, next gives -100, next gives +1000) → exploding gradient or too much variance.

Check initialization and consider gradient clipping.

7.10.2 Early Stopping

When the loss stops improving significantly across several consecutive epochs, stop training. This is early stopping — an optimization technique that prevents wasting computation on epochs that produce no meaningful improvement.

Purpose: Early stopping is both an efficiency technique (saving compute) and a regularization technique (preventing overfitting). When the test loss plateaus or starts increasing while training loss continues to drop, further training is just memorizing noise.

Steps:

Track both training and test loss after every epoch.
If test loss does not improve for N consecutive epochs (e.g., N=5), stop training.
Restore the model weights from the epoch with the lowest test loss.

Visual Intuition: Plot two lines on the same graph. X-axis: epochs (0 to 1000). Y-axis: loss. The training loss (blue line) starts high and drops steadily — it always goes down because the optimizer directly minimizes it. The test loss (orange line) also drops at first, But around epoch 400 it starts to flatten. Around epoch 600, it begins to creep up, even though the blue training line keeps dropping. The point where the orange line is lowest (epoch ~400) is your early-stopping point. Training beyond this point only hurts generalization.

Common pitfalls: (1) Trusting only the training loss — it always decreases (by definition of gradient descent). The test loss tells you about generalization. (2) Stopping too early because the loss curve had a small bump — losses can fluctuate. Use a patience parameter (wait N epochs before declaring convergence). (3) Not checking activation histograms at all — you can train for hours with half your neurons dead and never know it.

Monitor two things: the loss curve (tells you if learning is happening) and activation histograms (tells you if each neuron is contributing). When the test loss stops improving, stop training. Early stopping saves time and prevents overfitting in one simple check.

Real-world & domain connection: In production ML pipelines, training monitoring is automated. Tools like TensorBoard, Weights & Biases, And MLflow stream loss curves and activation distributions in real time. Engineers set up alerts when the test loss diverges or when a layer's activations indicate dead neurons. Early stopping is built into every major framework (Keras EarlyStopping callback, PyTorch Lightning's EarlyStopping). For large models that cost thousands of dollars to train, detecting a dying run early can save enormous amounts of money.

7.11 Summary of Module 5

You started this module with a single neuron drawing straight lines. You end it building deep hierarchies that can carve the input space into intricate, nonlinear shapes. Here is the map of everything you learned.

7.11.1 The Four Pillars

The deep feed-forward neural network has four components:

Data — tabular, image, text, or other structured/unstructured input
Architecture — number of hidden layers (depth) and number of neurons per layer (width), plus activation functions
Loss function — differentiable objective that measures prediction error
Learning process — gradient-based optimization (SGD, mini-batch, or batch) using backpropagation

7.11.2 Depth and Width

Depth builds hierarchy by combining simple features into complex patterns and concepts. Width captures more features at a given level.

The more hidden units, The more hierarchical representations the network can understand. The first hidden layer detects small, basic features. Those features combine into complex patterns in the next layer. Those patterns are further consolidated into concepts in even higher layers. This progression — from fine details to abstract concepts — is what depth provides.

A deep feed-forward network is a function approximator built from four ingredients: data, architecture, loss, and optimizer. Depth and width give it representational power. Activation functions give it nonlinearity. Backpropagation gives it a way to learn. Monitoring gives you visibility into whether it is actually learning.

7.11.3 What Comes Next

For the next module, specialized networks for computer vision, speech recognition, And time-series/sequential data will be introduced. These go beyond the standard feed-forward architecture.

Next session agenda: Two Python code demonstrations — showing how to actually implement the architectures discussed, And how to generate the training diagnostics (loss plots and activation histograms) to monitor what is happening during training.

Exam note: Module 5 is confirmed on the exam. Numerical problems will be small in scale. Focus on: parameter counting. Forward propagation computation. Loss calculation. Gradient/error computation. One-step weight updates. Computational graph drawing. Architecture design heuristics. Vanishing gradient understanding. Activation function selection.

Real-world & domain connection: The feed-forward network is the foundation of modern deep learning. Every specialized architecture you will encounter. — CNNs, RNNs, Transformers, GANs — is a variation on this four-component framework. Understanding why depth matters, how gradients flow, And how to pick activations and monitor training are skills that transfer directly to every advanced architecture.

7.12 Exam Guidance Summary

Exam note: Module 5 (Deep Feed-Forward Neural Networks) is confirmed on the exam. The guidance below tells you exactly what to prepare for.

In exam numerical problems, values will be very small. Intermediate values may be.

Provided. You will be asked to compute a single-level weight update or error component.

The gradient form (derivative formula) will be provided, though sometimes.

It may be customized. Apply the given form to the weight adjustments.

Expect at most one or two values to compute. Full-fledged multi-layer forward + backward propagation is unlikely to be expected.
You may be asked to draw a computational graph for a given network — at least for one or two layers.

Understand the diagram: forward graph maps input → Z → A → output; backward graph maps loss → gradients → weight adjustments.

A full numerical example of forward propagation, loss computation, and backward propagation.

Is provided in the uploaded materials. Work through it step-by-step with pen and paper.

Types of exam problems: forward propagation computation, loss calculation, gradient/error computation, one-step weight update, computational.

Graph drawing, architecture design heuristics, and concept-based questions on vanishing gradient, activation function selection, and architecture patterns.

Convolutional neural networks are likely not on the exam (pending final confirmation), but this module (Module 5) is confirmed.
Spend 1-2 hours working through the uploaded problem sets step by step.

7.13 Key Industry Applications

GPU computation prefers hidden layer sizes in powers of 2 (64, 128, 256, 512) for efficient thread-based matrix.

Computations. GPUs organize work into warps of 32 threads, and powers of 2 align cleanly with this hardware parallelism.

Neuro-evolution: Evolutionary algorithms automatically search through combinations of hidden units and layers to.

Find optimal architectures. This eliminates manual experimentation. Approaches like NEAT (NeuroEvolution of Augmenting Topologies). Evolve both weights and architecture simultaneously. This is used when manual architecture search becomes. Infeasible — for example, in large-scale industrial deployments where every percentage point of accuracy matters.

Digit classification: Handwritten digit recognition using 28×28 pixel images fed into deep.

Neural networks to classify 10 digit classes. This is the precursor to convolutional neural. Networks. The MNIST dataset (70,000 handwritten digits) remains the "hello world" of deep. Learning — if your architecture cannot achieve >98% on MNIST, something is fundamentally wrong.

Sentiment analysis: Thousands of engineered NLP features (TF-IDF vectors, word embeddings,.

Part-of-speech tags) fed into a deep network to classify text sentiment into categories. (positive, negative, neutral, or fine-grained like 1-5 stars). Used by companies to monitor. Brand perception on social media, analyze product reviews, and route customer support tickets.

Customer segmentation: Tabular data (demographics, purchase history, browsing behavior) used to classify customers into segments.

For targeted marketing strategies. Deep networks can discover nonlinear segment boundaries that traditional clustering (k-means) misses.

House price prediction: Regression with deep networks for real-estate valuation. Inputs include square footage, bedrooms, bathrooms, location features, school ratings, and.

Market trends. Output is a continuous price. The equal-width architecture from Section 7.2 is a natural starting point for this type of task.

DNN Lecture 07 notes · Deep Feed-Forward Neural Networks

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

1(Preamble)

27.1 Review of Deep Neural Network Fundamentals

Why depth and width matter, and the vanishing/exploding gradient problems deep networks face.

37.2 Designing a Deep Feed-Forward Neural Network Architecture

The four design components and three common architecture patterns: equal-width, pyramid, and hourglass.

47.3 Forward Propagation: A Full Numerical Example

How an input flows through weights and activations to produce a prediction, worked numerically.

57.4 Computing the Loss

Binary cross-entropy loss for classification and a worked loss calculation.

67.5 Backward Propagation and Gradient Computation

Using the chain rule to assign blame to each weight and update it.

77.6 The Effect of Nonlinearity — Interactive Demonstration

87.7 Architecture Design Heuristics — Getting a Starting Point

97.8 Vanishing and Exploding Gradient Problems

Why Sigmoid shrinks gradients and how ReLU fixes it.

107.9 Activation Functions for Hidden Layers

ReLU, Leaky ReLU, ELU, Tanh, Sigmoid and when to use each.

117.10 Monitoring Training Progress

Loss curves, activation histograms, and early stopping.

127.11 Summary of Module 5

The four pillars of a deep feed-forward network and what comes next.

137.12 Exam Guidance Summary

What to prepare for the exam: numerical problems, computational graphs, and key concepts.

147.13 Key Industry Applications

Real-world uses: digit classification, sentiment analysis, customer segmentation, and hardware-aligned design.

Postgraduate students studying deep learning and neural network fundamentals.

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Depth, Width, and the Role of Nonlinearity

Must-know: Depth (number of layers) builds a hierarchy of features; width (neurons per layer) captures more features at one level. But a stack of layers only helps if each layer has a nonlinear activation — a stack of linear layers collapses to a single linear map.

$A^{2} = W^{2} (W^{1} x + b^{1}) + b^{2} = (W^{2} W^{1}) x + (W^{2} b^{1} + b^{2}) = W_{eff} x + b_{eff}$

⚠️ Top pitfall: Assuming 'more layers = smarter network.' Without activation functions, a 10-layer network is mathematically identical to a 1-layer network.

Self-check: Remove every activation function from a 3-hidden-layer network. What single mathematical object does the whole network become?

Connects to: Activation functions, Forward propagation, Universal Approximation Theorem

Architecture Patterns: Equal-width, Pyramid, Hourglass

Must-know: Three reusable layouts: equal-width (same neurons per layer, for time-series), pyramid (narrowing, for hierarchical classification), and hourglass (compress then expand, for autoencoders and generative models). Input and output sizes are fixed by your data; the hidden middle is your design.

$parameters between layer i and i + 1 = n_{i} \times n_{i + 1} + n_{i + 1}$

⚠️ Top pitfall: Counting parameters but forgetting the bias term — every receiving neuron adds one bias, so add $n_{i + 1}$ , not just the weight products.

Self-check: A network is input(3) → hidden(4) → hidden(4) → output(1). How many learnable parameters are there in total?

Connects to: Parameter counting, Forward propagation, Architecture heuristics

Forward Propagation

Must-know: Forward propagation is a repeating two-step pipeline: multiply by weights and add bias to get $Z^{l}$ , then apply the activation to get $A^{l}$ . The output activation matches the task: linear for regression, Sigmoid for binary, Softmax for multi-class, Sigmoid-per-node for multi-label.

$Z^{l} = W^{l} A^{l - 1} + b^{l}, A^{l} = σ (Z^{l})$

⚠️ Top pitfall: Using Softmax for binary classification or Sigmoid in hidden layers. One Sigmoid node already covers two classes; Sigmoid in deep hidden layers causes vanishing gradients.

Self-check: For a binary classification problem, how many output nodes and which activation should you use?

Connects to: Activation functions, Computing the loss, Backward propagation

Binary Cross-Entropy Loss

Must-know: For a Sigmoid output, use binary cross-entropy. It measures the distance between the true label $y$ (0 or 1) and the predicted probability $y^$ . The log heavily penalizes confident wrong answers.

$L = - [y lo g (y^) + (1 - y) lo g (1 - y^)]$

⚠️ Top pitfall: Applying loss to raw logits $Z^{2}$ instead of the activated output $A^{2} = σ (Z^{2})$ . Log of a negative number produces NaNs.

Self-check: If the true label is 1 and the network predicts 0.001, is the loss small or large? Why?

Connects to: Forward propagation, Backward propagation, Output layer design

Backward Propagation (Backprop)

Must-know: Backprop is the chain rule applied across the whole network. The output error for BCE+Sigmoid simplifies to $y^- y$ , and this delta ripples backward, multiplied at each layer by the transpose of the next weight matrix and the activation derivative.

$δ = y^- y, \frac{\partial L}{\partial W ^{l}} = δ^{l} (A^{l - 1})^{⊤}$

⚠️ Top pitfall: Using $y^$ itself as the error signal instead of $y^- y$ . The raw prediction is not the error; the difference from truth is.

Self-check: Write the single scalar weight-update rule that uses the gradient and the learning rate $η$ .

Connects to: Computing the loss, Forward propagation, Vanishing gradients

Why Nonlinearity Is What Makes Depth Useful

Must-know: A linear dataset needs no hidden layers; a nonlinear dataset needs activations. Adding hidden layers without activations still collapses to one line. Sigmoid learns but slowly; Tanh plus more depth learns hard boundaries much faster.

$A^{2} = W^{2} (W^{1} x + b^{1}) + b^{2} = W_{eff} x + b_{eff} (linear collapse)$

⚠️ Top pitfall: Believing that adding layers alone increases capacity. Layers without activation add zero representational power.

Self-check: Two hidden layers with linear activation cannot separate spirals. Why not?

Connects to: Depth and width, Activation functions, Architecture patterns

Architecture Design Heuristics

Must-know: Start with 2–3 hidden layers and 10–100 neurons, then iterate: increase complexity if underfitting, reduce if overfitting. For tabular data a log-rule gives a layer ceiling; keep total parameters well below your training-set size.

$max_layers \approx ⌊ lo g_{2} (\frac{instances}{features \times classes}) ⌋ + 1$

⚠️ Top pitfall: Trusting the log-rule for high-dimensional data (images, text), where features ≫ instances and it suggests zero layers. Fall back to experimentation.

Self-check: Your network has 800 instances, 20 features, 3 classes. What layer ceiling does the heuristic suggest?

Connects to: Parameter counting, Overfitting and underfitting, Monitoring training

Vanishing and Exploding Gradients

Must-know: Backpropagating through many layers multiplies gradients by weights and activation derivatives. Sigmoid's derivative never exceeds 0.25, so gradients shrink exponentially (vanish); weights > 1 make them explode. ReLU's derivative is 1 for positive inputs, fixing vanishing.

$\frac{\partial L}{\partial W ^{l}} = \frac{\partial L}{\partial A ^{L}} k = L \prod l + 1 (\frac{\partial A ^{k}}{\partial Z ^{k}} W^{k}) \frac{\partial Z ^{l}}{\partial W ^{l}}$

⚠️ Top pitfall: Confusing vanishing gradients (early layers stop learning) with underfitting (architecture too simple). They need different fixes.

Self-check: After 3 Sigmoid layers with weight norm 0.8, by what factor has a gradient of 1.0 shrunk?

Connects to: Backward propagation, Activation functions, Monitoring training

Activation Functions

Must-know: Use ReLU as the default hidden activation (fast, fixes vanishing gradients). If neurons die, switch to Leaky ReLU or ELU. Use Sigmoid only at the output for binary classification, Softmax for multi-class, linear for regression.

$ReLU (z) = max (0, z), ReLU^{'} (z) = {10 z > 0 z < 0$

⚠️ Top pitfall: Putting Sigmoid or Softmax in hidden layers. Sigmoid vanishes; Softmax across neurons kills their independence. Reserve both for outputs.

Self-check: A histogram of a layer's activations spikes at 0. What problem does that signal?

Connects to: Vanishing gradients, Forward propagation, Output layer design

Monitoring Training

Must-know: Track two diagnostics: the loss curve (training vs test) and activation histograms per layer. Stop when test loss stops improving (early stopping). Training loss always falls, so judge generalization by test loss.

$generalization gap = L_{test} - L_{train}$

⚠️ Top pitfall: Trusting training loss alone — it always decreases by construction. A rising test loss while training loss falls means overfitting.

Self-check: Both training and test loss are high. Is the model overfitting or underfitting?

Connects to: Overfitting and underfitting, Architecture heuristics, Activation functions

The Four Pillars of a Deep Feed-Forward Network

Must-know: Every deep feed-forward network is built from four ingredients: data, architecture (depth, width, activations), loss function, and a learning process (backprop + gradient descent). Depth and width give power; activations give nonlinearity; monitoring gives visibility.

$y^= f_{NN} (x; W, b)$

⚠️ Top pitfall: Memorizing formulas without understanding the four-component framework; exam questions test why each piece exists, not just the math.

Self-check: Name the four components you must specify before training a feed-forward network.

Connects to: All sections above

Previous LectureLecture 06 · Deep Feedforward Neural Networks

Next LectureLecture 08 · Deep Neural Networks — Core Concepts and Introduction to CNNs

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Deep Feed-Forward Neural Networks

7.1 Review of Deep Neural Network Fundamentals

7.1.1 Quick Recap of Module 5

7.1.2 Visualizing What a Deep Network Does

7.2 Designing a Deep Feed-Forward Neural Network Architecture

7.2.1 The Four Components

7.2.2 Three Common Architecture Patterns

7.2.3 Worked Example: Parameter Counting in an Equal-Width Network

7.2.4 A Note on Layers, Depth, and Notation

7.3 Forward Propagation: A Full Numerical Example

7.3.1 Network Setup

7.3.2 Symbol Registry — Forward Propagation Example

7.3.3 Step-by-Step Forward Pass

7.3.4 Output Layer Design by Task Type

7.3.5 Computational Graph — Forward Propagation

7.4 Computing the Loss

7.4.1 Binary Cross-Entropy Loss

7.4.2 Worked Example: Loss Calculation

7.5 Backward Propagation and Gradient Computation

7.5.1 The Delta (Error) at the Output Layer

7.5.2 Propagating Errors Backward

7.5.3 The Chain Rule for Multiple Layers

7.5.4 Weight Updates

7.5.5 Computational Graph — Backward Propagation

7.6 The Effect of Nonlinearity — Interactive Demonstration

7.6.1 Setup

7.6.2 Experiment 1 — Linearly Separable Data, No Hidden Layers

7.6.3 Experiment 2 — Nonlinear Data, No Hidden Layers

7.6.4 Experiment 3 — Nonlinear Data, With Hidden Layers but No Activation

7.6.5 Experiment 4 — Nonlinear Data, Sigmoid Activation

7.6.6 Experiment 5 — Even More Complex Data, Tanh Activation with More Layers

7.6.7 How Capacity Affects Overfitting

7.7 Architecture Design Heuristics — Getting a Starting Point

7.7.1 Heuristic for Number of Hidden Layers

7.7.2 Heuristic for Number of Neurons per Layer with Parameter Constraints

7.7.3 Scaling to Image Data: The Design Loop

7.7.4 Scaling to Text Data

7.7.5 Summary: General Design Heuristics

7.8 Vanishing and Exploding Gradient Problems

7.8.1 The Problem

7.8.2 Why Sigmoid Causes Vanishing Gradients

7.8.3 Worked Example: Vanishing Gradient with Sigmoid

7.8.4 The General Form

7.9 Activation Functions for Hidden Layers

7.9.1 ReLU — The Default Starting Point

7.9.2 Sigmoid — Output Layer Only

7.9.3 Tanh

7.9.4 The Dying ReLU Problem

7.9.5 Variations of ReLU That Avoid Dying Neurons

7.9.6 Activation Function Selection Guidelines

7.10 Monitoring Training Progress

7.10.1 Two Essential Plots

7.10.2 Early Stopping

7.11 Summary of Module 5

7.11.1 The Four Pillars

7.11.2 Depth and Width

7.11.3 What Comes Next

7.12 Exam Guidance Summary

7.13 Key Industry Applications

DNN Lecture 07 notes · Deep Feed-Forward Neural Networks

Sections Breakdown

Exam Revision Notes

Depth, Width, and the Role of Nonlinearity

Architecture Patterns: Equal-width, Pyramid, Hourglass

Forward Propagation

Binary Cross-Entropy Loss

Backward Propagation (Backprop)

Why Nonlinearity Is What Makes Depth Useful

Architecture Design Heuristics

Vanishing and Exploding Gradients

Activation Functions

Monitoring Training

The Four Pillars of a Deep Feed-Forward Network

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat