Deep Neural Networks

Deep Feedforward Neural Networks

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Deep Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Gradient descent and its variants (batch, mini-batch, stochastic) — covered in Lecture 5
Binary & multi-class classification, confusion matrix, precision, recall, F1 — covered in Lecture 5
Sigmoid, softmax, logistic regression, cross-entropy loss — covered in Lecture 5
Linear neural networks, activation functions (ReLU, identity), MSE/MAE — covered in Lecture 4
Perceptron, forward/backward propagation, XOR non-linear separability — covered in Lecture 3
DNN architecture: layers, feed-forward structure, hyperparameters — covered in Lecture 2

Introduction to Deep Feedforward Neural Networks

A deep feedforward network is a function approximation machine. You feed it data. It learns to map inputs to outputs. It does this by adjusting millions of weights. The "deep" comes from stacking many layers. "Feedforward" means information flows one way. There are no loops. This lecture builds from a single neuron to a full deep network. It shows why depth works and what can go wrong.

6.1 Linear Neural Networks: Code Walkthrough and Implementation

6.1.1 Binary Classification with a Linear Neural Network

A single number can tell you if a tumor is dangerous. But can a computer learn to produce that number from raw measurements? That is what a linear neural network does for binary classification. It learns a weighted vote of the input features. Then it squashes that vote into a probability between 0 and 1.

Think of a linear classifier as a hiring committee with one rule book. Each feature (GPA, experience, interview score) gets a weight — how much the committee cares about it. The committee adds up all the weighted evidence. Then it runs the result through a "hiring threshold." Any score above 0.5 means hire. Below 0.5 means reject. The sigmoid is that threshold, but smoothed out — it never says 0 or 1 with absolute certainty.

Where the analogy breaks: the committee's weights are fixed by human judgment. The network's weights are learned from data — the committee rewrites its own rule book after seeing thousands of past hiring decisions.

A linear neural network for binary classification has no hidden layers. The $d$ input features connect directly to a single output neuron. They connect through a weight vector $w \in R^{d}$ and a scalar bias $b$ . The computation has three stages:

1. Linear combination: $z = w^{⊤} x + b$ — a weighted sum of the inputs, shifted by the bias. 2. Activation: $y^= σ (z) = \frac{1}{1 + e ^{- z}}$ — the sigmoid squeezes $z$ into $(0, 1)$ , giving a probability. 3. Decision: Compare $y^$ against the true label $y \in {0, 1}$ . Compute the loss. Backpropagate the gradient. Update $w$ and $b$ . Repeat.

The entire network is a function $f (x; w, b) = σ (w^{⊤} x + b)$ . It maps a feature vector to a predicted probability. The output is always a number between 0 and 1.

The professor showed this model on the breast cancer diagnostic dataset. The task: classify a tumor as malignant ( $y = 1$ ) or benign ( $y = 0$ ) based on diagnostic features.

6.1.2 Symbol Registry — Linear Binary Classifier

$x$ — input feature vector — $x \in R^{d}$ — vector
$w$ — weight vector — $w \in R^{d}$ — vector
$b$ — bias term — $b$ — scalar
$z$ — linear pre-activation (the weighted sum before activation) — $z = w^{⊤} x + b$ — scalar
$σ$ — sigmoid activation (squashes any real number into $(0, 1)$ ) — $σ (z) = \frac{1}{1 + e ^{- z}}$ — scalar in $(0, 1)$
$y^$ — predicted probability (the model's guess for class 1) — $y^= σ (z)$ — scalar in $[0, 1]$
$y$ — true label (ground truth from the dataset) — $y \in {0, 1}$ — scalar
$η$ — learning rate (step size for weight updates) — $η$ — small positive scalar (e.g., 0.01)

6.1.3 Implementation Steps

The code builds everything from scratch — no scikit-learn shortcuts for the core logic.

Purpose: Train a single-neuron binary classifier end-to-end. Go from raw data loading through evaluation. See every formula in action. This is the simplest possible neural network. It is the foundation for everything that follows.

Inputs & Outputs:

Input: A dataset with $d$ numerical features per instance and binary labels ${0, 1}$ .
Hyperparameters: Learning rate $η = 0.01$ , epochs = 1000, train/test split = 80/20.
Output: A trained weight vector $w \in R^{d}$ and bias $b$ , plus evaluation metrics (accuracy, confusion matrix, precision, recall, F1).

Steps — the training pipeline:

1. Load the data. Read the dataset from file (CSV, Excel, or built-in source). Adapt the loader to your data format.

2. Inspect the data. Print the instance count, feature count, and class distribution (how many malignant vs. benign). This sanity check catches loading errors early.

3. Set hyperparameters. Fix $η = 0.01$ and epochs = 1000. Initialize arrays to record loss history and weight history for later plotting.

4. Define the sigmoid function. Implement $σ (z) = \frac{1}{1 + e ^{- z}}$ . Every call returns a number in $(0, 1)$ .

5. Initialize the weights. For $d$ features and 1 output, you need $d$ link weights plus 1 bias — $d + 1$ total. Initialize them randomly near zero (factor 0.01) to break symmetry. Do NOT set all weights to zero — that makes every weight update identical and prevents learning. Proven schemes like He and Xavier initialization exist and are covered after the midterm.

6. Forward propagation. For one instance $x$ : compute $z = w^{⊤} x + b$ , then $y^= σ (z)$ .

7. Compute the loss. Use binary cross-entropy (log loss):

$L = - [y lo g (y^) + (1 - y) lo g (1 - y^)]$

When $y^$ is close to $y$ , the loss is small. When $y^$ is far from $y$ , the loss is large. The gradient from this loss tells each weight which way to move.

8. Backward propagation. Compute $\frac{\partial L}{\partial w}$ and $\frac{\partial L}{\partial b}$ using the chain rule. Multiply by $η$ . Subtract from current weights. For a single-layer network, the chain is short: $L \to y^\to z \to w, b$ . In multi-layer networks, the chain rule propagates through every layer from output back to input.

9. Compute accuracy. Threshold $y^$ at 0.5 to get a predicted class. Count matches against $y$ .

10. Training loop. Split data into 80% train / 20% test. Normalize features to zero mean and unit variance. For each of 1000 epochs: forward-propagate the training batch, compute loss and accuracy, compute gradients, update weights. Print results every 100 epochs to avoid console noise.

11. Evaluate on test data. Run the trained model on the held-out 20%. Compute predictions. Never peek at test labels during training.

12. Build the confusion matrix. Count true positives (TP), true negatives (TN), false positives (FP), false negatives (FN). Compute precision $= \frac{T P}{T P + F P}$ , recall $= \frac{T P}{T P + F N}$ , and F1 $= 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$ . The code implements these manually so you see each formula — scikit-learn offers one-line equivalents.

Trace on the breast cancer dataset:

Start with $d = 30$ features, $η = 0.01$ , random weights near zero.

Epoch 1: $y^\approx 0.5$ for most samples (random predictions). Loss is high (around 0.69 for balanced classes — the baseline of guessing 0.5). Weights shift a little.
Epoch 100: Loss has dropped noticeably. Accuracy on training set rises above random chance.
Epoch 500: Loss is low. Most predictions are correct.
Epoch 1000: Loss has flattened. Final accuracy on test set is measured.

Result: 4 false negatives (malignant predicted as benign — dangerous). 1 false positive (benign predicted as malignant). The diagonal of the confusion matrix dominates, showing good overall performance.

6.1.4 Understanding the Confusion Matrix

For binary classification, the confusion matrix is a $2 \times 2$ table:

	Predicted Negative	Predicted Positive
Actual Negative	True Negative (TN)	False Positive (FP)
Actual Positive	False Negative (FN)	True Positive (TP)

The diagonal entries (TN, TP) are correct predictions. The off-diagonal entries (FP, FN) are errors. From these four numbers you derive precision, recall, and F1.

In the breast cancer demo: 4 malignant cases were falsely detected as benign (FN). 1 benign case was falsely detected as malignant (FP). The diagonal showed much higher numbers — the model performed well.

Q: Between predicting a benign case as malignant and predicting a malignant case as benign — which is more dangerous?

A: A false negative is far more dangerous. It means predicting "no cancer" when the patient actually has it. The patient goes untreated. A false positive means an unnecessary follow-up test — inconvenient, but not life-threatening. In medical diagnostics, recall (catching all actual positives) often matters more than precision.

Scope: The confusion matrix interpretation depends entirely on the application. In spam detection, a false positive (good email marked as spam) might be worse than a false negative (spam reaching the inbox). Always ask: which error costs more in this domain?

You should think about this from the application perspective. In this use case, which component of the confusion matrix matters most? Between precision and recall, which metric carries more weight when your goal is to detect benign cases? Work this out as an exercise.

6.1.5 Decision Boundary Visualization

A linear neural network with one output node produces a linear decision boundary. The boundary is the set of points where $w^{⊤} x + b = 0$ . In 2D this is a line. In 3D it is a plane. In higher dimensions it is a hyperplane.

Picture a room split by a wall. Points on one side are "class 0." Points on the other side are "class 1." The wall is the decision boundary. Its angle and position are determined by $w$ and $b$ . Changing a weight tilts the wall. Changing the bias shifts the wall.

The visualization shows data points colored by their true class. Some points cross the boundary — those correspond to the mispredictions in the confusion matrix.

6.1.6 Hyperparameter Experiments

Several knobs you can turn:

Change the learning rate. If you make it very large, you will see wild oscillations in the loss. The weights overshoot the minimum.
Change the number of epochs. Let it run longer or shorter and observe convergence. After the loss flattens, more epochs are wasted.
Do feature engineering. Use a technique like random forest to find the most important features. Take only the top few and compare against using all features. Sometimes fewer, better features beat more, noisy ones.
Create polynomial features. If feature $X_{1}$ interacts with feature $X_{2}$ , create $X_{1} \times X_{2}$ or $X_{1}^{2} \times X_{3}$ and feed those into the network. You may have seen this as "linear basis functions" in your machine learning course.

Pitfall: A very large learning rate causes the loss to diverge instead of converge. The weights update so aggressively that they overshoot the minimum and climb the opposite wall of the loss surface. Always start small ( $η = 0.01$ or less) and increase gradually.

Pitfall: Normalizing features is essential. If one feature ranges from 0 to 1000 and another from 0 to 1, the first dominates the weight updates. The optimizer spends most of its effort adjusting the weight for the large-scale feature while ignoring the small-scale one. Always normalize to zero mean and unit variance before training.

The beauty of deep neural networks: you do not have to do this feature engineering manually. Dedicated architectures perform automated feature extraction by design. How they do it becomes clear as you proceed through the upcoming modules.

6.1.7 Industry Applications

The confusion matrix and its components — precision, recall, F1 — are available as one-line calls in scikit-learn. The code here reimplements them manually so you can see the formulas at work. In production, use the library functions — they are tested, optimized, and battle-hardened.

Random forest can identify feature importance. It has built-in functions for this. Use it to order features by importance, take the top few, and feed them into any model. This is a common preprocessing step in industry ML pipelines.

A linear neural network for binary classification is a single neuron with sigmoid output. It computes $y^= σ (w^{⊤} x + b)$ , learns from labeled data via gradient descent, and produces a linear decision boundary. The same training loop — forward pass, loss, backprop, update — scales to networks with millions of parameters. Only the architecture changes.

6.2 Multi-Class Classification with Linear Neural Networks

6.2.1 Architecture Differences from Binary Classification

Binary classification asks: "Is this a cat?" Multi-class classification asks: "Is this a cat, a dog, or a bird?" The answer is one of $K$ options, not just yes/no. The network needs $K$ output neurons. One for each class. Each neuron votes for its class. Softmax converts those votes into a probability distribution. The distribution sums to 1.

Imagine a talent show with three judges. Each judge scores the performance. The binary case has one judge saying "good or bad." The multi-class case has $K$ judges, each championing their own category. Softmax is the rule that says: "The highest-scoring judge wins, but we'll report everyone's score as a share of the total."

Where the analogy breaks: judges are independent. In softmax, probabilities are coupled. Raising one class's score lowers every other class's probability. They must sum to 1.

For $K$ classes, the architecture needs $K$ output neurons. Two things change from the binary case:

1. Softmax replaces sigmoid. Given raw scores $Z_{1}, Z_{2}, \dots, Z_{K}$ , softmax converts them into $K$ probabilities:

$y^_{k} = softmax (Z_{k}) = \frac{e ^{Z_{k}}}{\sum _{j = 1}^{K} e ^{Z_{j}}}$

Each $y^_{k} \in (0, 1)$ . The sum $\sum_{k} y^_{k} = 1$ . The class with the largest $Z_{k}$ gets the largest probability.

2. Categorical cross-entropy replaces binary cross-entropy. The loss for one instance:

$L = - k = 1 \sum K y_{k} lo g (y^_{k})$

Since $y$ is a one-hot vector, only the term for the true class survives. If the true class is $c$ , the loss is simply $- lo g (y^_{c})$ . The binary cross-entropy term $(1 - y) lo g (1 - y^)$ has no analog here — it was a special case for $K = 2$ .

6.2.2 Symbol Registry — Multi-Class Classifier

$K$ — number of distinct classes — integer, $K \geq 2$
$Z_{k}$ — logit (raw unnormalized score) for class $k$ — $Z_{k} = w_{k}^{⊤} x + b_{k}$ — scalar
$y^_{k}$ — predicted probability for class $k$ — $y^_{k} = softmax (Z_{k}) = \frac{e ^{Z_{k}}}{\sum _{j = 1}^{K} e ^{Z_{j}}}$ — scalar in $(0, 1)$
$y$ — true label in one-hot encoding — $y \in {0, 1}^{K}$ — vector, exactly one entry is 1

6.2.3 One-Hot Encoding

A one-hot vector has length $K$ . Exactly one element is 1. The rest are 0. The position of the 1 encodes the class.

For 4 classes (A, B, C, D) and an instance of class B:

$y = [0, 1, 0, 0]$

During training, the model outputs $K$ probabilities $y^_{1}, \dots, y^_{K}$ . The cross-entropy loss compares this probability vector against the one-hot vector. Because 0 $\times$ log(anything) = 0, the loss simplifies: only the probability assigned to the true class matters.

For $K$ classes, the one-hot encoding has size $K$ . Every instance gets a one-hot label. This is different from label encoding. Label encoding stores class as integer 1, 2, 3. That would imply an ordering that does not exist.

Pitfall: Do not use integer labels with categorical cross-entropy expecting the model to treat them as class indices. The loss function expects one-hot vectors (or class indices handled internally by the framework). Using raw integers without proper conversion leads to silent bugs.

6.2.4 Iris Dataset Demo

The iris dataset contains four features: petal length, petal width, sepal length, sepal width. The task: classify each flower into one of three species — Setosa, Versicolor, Virginica.

The code structure is identical to the binary case. Only three modules change: softmax activation, categorical cross-entropy loss, and one-hot encoded labels.

Worked: Iris classification with mini-batch SGD

Setup: $K = 3$ classes, $d = 4$ features, batch size 16, $η = 0.1$ , 1000 epochs.

Each epoch: shuffle the data, form batches of 16 instances.
For each batch: forward-propagate all 16 instances, compute softmax probabilities (each row sums to 1), compute categorical cross-entropy averaged over 16 instances, backpropagate, update weights.
One epoch = roughly $⌈ n /16 ⌉$ weight updates, where $n$ is the training set size.

The weight matrix from input to output has shape $3 \times 4$ (3 output classes, 4 input features).

6.2.5 Confusion Matrix for Multi-Class

For $K$ classes, the confusion matrix is $K \times K$ . The diagonal still shows correct predictions. Off-diagonal entry $(i, j)$ shows how many instances of true class $i$ were predicted as class $j$ .

Precision and recall are computed per class. Use the one-vs-all approach:

Treat class $c$ as "positive." Treat all other classes as "negative."
Compute TP, FP, FN, TN for class $c$ . Then precision = TP/(TP+FP), recall = TP/(TP+FN).

The F1 score is the harmonic mean of precision and recall:

$F 1 = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$

Why F1? Precision alone can be misleading. A model that predicts "positive" only once and gets it right has precision = 1.0, but it misses almost everything. Recall alone can be misleading too. Predicting "positive" for every instance gives recall = 1.0 but terrible precision. F1 balances both. It is also less sensitive to class imbalance. If one class has 100 instances and another has 5, F1 gives a fairer picture than raw accuracy.

6.2.6 Multi-Label vs Multi-Class — Architectural Question

Q: What is the difference between multi-class and multi-label classification?

A: In multi-class, each instance belongs to exactly one class. In multi-label, one instance can belong to multiple classes at once. A single image can contain both a cat and a dog. That is multi-label. The correct output is "cat AND dog," not "cat OR dog."

Q: How would you design the architecture for multi-label?

A: In multi-class, softmax forces $\sum_{k} y^_{k} = 1$ — the classes compete. For multi-label, you need the classes to be independent. Replace softmax with $K$ independent sigmoid units. Each output neuron asks: "Does this image contain class $k$ ?" and produces a probability in $(0, 1)$ . The loss becomes the sum of $K$ binary cross-entropy terms — one per label. Think through the implications for the output layer.

Q: Does scikit-learn internally handle multi-class with one-vs-one or one-vs-all tricks?

A: Yes. For 4 classes, scikit-learn builds 6 binary classifiers for one-vs-one. Or 4 binary classifiers for one-vs-all. These are tricks. They make binary classifiers handle multi-class. In a deep network, you do not need these tricks. $K$ output nodes with softmax handle it directly. Each output gives a probability automatically.

6.2.7 Weight Count for Multi-Class

With 4 input features and 3 output classes, the weight matrix from input to output has shape $3 \times 4$ — that is $3 \times 4 = 12$ link weights. Add 1 bias per output neuron: 3 biases. Total: $12 + 3 = 15$ learnable parameters.

General formula for a single-layer multi-class network with $d$ inputs and $K$ classes: $d \times K + K$ parameters.

6.2.8 Batch Stability and Class Imbalance

Consider this scenario during mini-batch SGD. One batch has only class A instances. The next has only class B. The model sees a distorted picture each update. The loss oscillates wildly. Each batch pulls the weights in a different direction.

Fix: Shuffle the data randomly every epoch before forming batches. This gives each batch a better mix of classes. It leads to more stable learning.

Class imbalance is a broader problem. Suppose Setosa has 100 training instances. Versicolor has only 10. Virginica has only 5. The model may learn to always predict Setosa. That gets 100 out of 115 correct. But it learned nothing about the other classes. F1 score helps. It penalizes this majority-class bias. Beyond F1, consider oversampling, undersampling, or weighted loss functions. These are covered in machine learning courses.

6.2.9 Dimensionality Reduction for Visualization

When you have many features (say 10), you cannot plot them all on a 2D screen. PCA — Principal Component Analysis — finds linear combinations of the features that capture the most variance. Each principal component is a new axis. The components are orthogonal (uncorrelated with each other). Take the top two components and use them for a scatter plot.

Think of PCA as taking a photo of a 3D object from the best angle. You lose depth information, but the 2D photo shows the most informative view possible. PCA finds the "best angle" mathematically — the projection that preserves the most variance in the data.

This gives a much better visualization than arbitrarily picking two raw features. In the iris dataset, PCA on the four features often reveals clear clusters for the three species.

6.2.10 Underfitting and Overfitting Indicators

Scope: Underfitting. If both training and test accuracy are poor — around 50-60% for a balanced problem — the model is underfitting. It is too simple to capture the patterns in the data. Remedies for a linear network: engineer better features, remove noisy features, or increase the dataset size.

Scope: Overfitting. If training accuracy is excellent but test accuracy is poor, the model is overfitting. The test confusion matrix shows many off-diagonal entries. The model memorized training data. It cannot generalize. For deeper networks, use regularization to fight overfitting. Dropout, weight decay, and early stopping all help.

Assumption: These diagnostics assume the train/test split is random and representative. If your test set comes from a different distribution, a perfect model will still have poor test accuracy. That is data mismatch, not overfitting.

Multi-class classification replaces the single sigmoid output with $K$ softmax outputs that compete to sum to 1. The cross-entropy loss only penalizes the probability assigned to the correct class. Everything else — the forward pass, backpropagation, gradient descent loop — is identical to the binary case. The architecture is the only difference.

6.3 Limitations of Linear Networks and the XOR Problem

Symbol Registry

$w$ — weight vector — $w \in R^{d}$ — determines the orientation of the decision boundary
$b$ — bias — scalar — determines the offset of the decision boundary
$x$ — input feature vector — $x \in R^{d}$
$h_{1}, h_{2}$ — hidden neuron outputs — scalars — outputs of two neurons in a hidden layer
$W^{(1)}$ — weight matrix for the first layer — $W^{(1)} \in R^{d \times h}$
$W^{(2)}$ — weight matrix for the second layer
XOR — exclusive OR function — outputs 1 when exactly one input is 1

6.3.1 Why a Single Linear Boundary Fails

A linear network can draw one straight line. One line. That is its entire toolbox. If your data needs two lines — or a curve — the linear network fails. This is not a bug. It is a mathematical fact. And the simplest problem that exposes this is XOR.

Think of a linear classifier as a single fence across a field. You want to separate sheep from goats with one straight fence. If the sheep are clustered in one corner and the goats in another, one fence works. But what if the sheep are in opposite corners? No single straight fence can isolate both corners at once. You need two fences — or a curved enclosure. XOR is the "sheep in opposite corners" problem.

A linear neural network with no hidden layers can only draw one straight line (or hyperplane). The decision boundary is $w^{⊤} x + b = 0$ . Any data that requires more than one line to separate classes is beyond its reach.

The simplest example is XOR. Take this truth table:

$X_{1}$	$X_{2}$	XOR Output
0	0	0
0	1	1
1	0	1
1	1	0

Plot these four points on a 2D plane. (0,0) and (1,1) are class 0. (0,1) and (1,0) are class 1. No single straight line can separate all the class-0 points from all the class-1 points. Draw any line you like — horizontal, vertical, diagonal — none perfectly separates the classes.

This is the canonical demonstration that linear models have limited capacity (the set of functions they can represent). XOR requires a nonlinear decision boundary.

6.3.2 Solving XOR with Two Lines

What if you use two lines instead of one? Line 1 and Line 2. Then combine their results.

Line 1 says: "Points above me are class 1." Line 2 says the same thing. Now combine them with a rule: if above Line 1 OR Line 2 but not both, classify as 1. Otherwise, classify as 0. This is XOR logic built from two linear models.

In neural network terms: the two lines correspond to two neurons in a hidden layer. Each neuron extracts one linear feature. A subsequent layer combines those features. Together they can perfectly separate XOR.

Mathematically, using two hidden neurons with ReLU activation and a specific set of weights:

Let the first hidden layer compute:

$h_{1} = max (0, x_{1} + x_{2}), h_{2} = max (0, x_{1} + x_{2} - 1)$

The output layer then combines them: $y = h_{1} - 2 h_{2}$ . Evaluating on the four XOR points:

$(x_{1}, x_{2})$	$h_{1}$	$h_{2}$	$y = h_{1} - 2 h_{2}$
(0, 0)	0	0	0
(0, 1)	1	0	1
(1, 0)	1	0	1
(1, 1)	2	1	0

The network has perfectly solved XOR.

6.3.3 The Deeper Problem: Combining Linear Models Is Still Linear

Here is the catch. Say neuron A gives output $A$ . Neuron B gives output $B$ . The next layer computes:

$W_{C} \cdot A + W_{B} \cdot B + b_{C}$

This is still a linear combination. Add more layers. Stack more neurons. As long as every operation is linear, the whole network collapses to one big linear model.

Proof of collapse: Layer 1 computes $h^{(1)} = W^{(1)} x + b^{(1)}$ . Layer 2 computes $h^{(2)} = W^{(2)} h^{(1)} + b^{(2)}$ . Combine them:

$h^{(2)} = W^{(2)} (W^{(1)} x + b^{(1)}) + b^{(2)} = (W^{(2)} W^{(1)}) x + (W^{(2)} b^{(1)} + b^{(2)})$

This is a single transformation $W^{'} x + b^{'}$ . No matter how deep you go without nonlinearity, you get a linear model.

To break out of linearity, you need something nonlinear between layers — an activation function.

6.3.4 The Limitation as Motivation

Even if you add many hidden layers and many hidden units, the network stays linear without nonlinearity. The whole thing reduces to a single linear transformation. The depth adds nothing. Complexity is wasted.

This is the core motivation for activation functions — not just at the output layer, but inside every hidden layer. The XOR problem is the simplest case that proves why you need them.

Pitfall: A common beginner mistake: "more layers = more power." This is wrong. Without nonlinear activations, you have one layer. It does not matter how many you stack. The network has the same capacity as a single linear model.

Pitfall: Some beginners try identity activations ( $g (z) = z$ ) in hidden layers. This is the same as no activation — the network stays linear.

A purely linear network, no matter how deep, is still a linear model. XOR proves this with four points — no single line can separate them. The fix is nonlinear activation functions between layers. This is the bridge to the next section: what makes ReLU, sigmoid, and tanh different, and why ReLU dominates in practice.

6.4 Activation Functions: Introducing Non-Linearity

6.4.1 The Core Idea

Linear layers stack like pancakes. Each pancake looks different, but together they still taste like one big pancake. Pour syrup (a nonlinear activation) between each layer, and suddenly you have a stack of distinct flavors. The nonlinearity is what makes depth count.

Put a nonlinear activation function at every hidden layer. The architecture becomes:

$input \to hidden layer 1 \to activation \to hidden layer 2 \to activation \to \dots \to output layer \to output activation$

This is a multilayer perceptron (MLP) — a feedforward deep neural network. Two ingredients define deep learning: multiple layers (depth) and nonlinear activations (complexity). Remove either one, and you are back to a linear model.

6.4.2 Symbol Registry — Activation Functions

$σ (z)$ — logistic sigmoid — $σ (z) = \frac{1}{1 + e ^{- z}}$ — output in $(0, 1)$
$tanh (z)$ — hyperbolic tangent — $tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}$ — output in $(- 1, 1)$
$ReLU (z)$ — Rectified Linear Unit — $ReLU (z) = max (0, z)$ — output in $[0, \infty)$
$LeakyReLU (z)$ — Leaky ReLU — $LeakyReLU (z) = max (α z, z)$ with small $α$ (e.g., 0.01) — output in $(- \infty, \infty)$

6.4.3 Mathematical Definition of Linearity

A function $f$ is linear if it satisfies two properties:

1. Additivity: $f (x + y) = f (x) + f (y)$ . The output of a sum equals the sum of the outputs. 2. Homogeneity: $f (a x) = a \cdot f (x)$ . Scaling the input scales the output by the same factor.

Sigmoid, tanh, and ReLU all violate at least one of these properties. They are nonlinear.

Quick check on each:

Sigmoid: $σ (0 + 0) = 0.5$ , but $σ (0) + σ (0) = 1.0$ . Additivity fails.
ReLU: $ReLU (- 3 + 5) = 2$ , but $ReLU (- 3) + ReLU (5) = 0 + 5 = 5$ . Additivity fails.
Tanh: $tanh (2 \cdot 1) \neq = 2 \cdot tanh (1)$ . Homogeneity fails.

Without nonlinearity, deep learning collapses. It cannot extract complex patterns. The XOR example proves exactly this.

6.4.4 Sigmoid — Properties and Usage

The sigmoid squashes any real input into $(0, 1)$ :

$σ (z) = \frac{1}{1 + e ^{- z}}$

Its derivative: $σ^{'} (z) = σ (z) (1 - σ (z))$ . The maximum derivative is 0.25 (at $z = 0$ ).

This range makes sigmoid natural for the output layer of a binary classifier. The output reads directly as a probability.

Pitfall — Vanishing gradients in hidden layers: When $z$ is large positive, $σ (z) \approx 1$ and $σ^{'} (z) \approx 0$ . When $z$ is large negative, $σ (z) \approx 0$ and $σ^{'} (z) \approx 0$ . The derivative is at most 0.25, and it approaches 0 at the tails. Backpropagating through many sigmoid layers multiplies many numbers $\leq 0.25$ . The product shrinks exponentially. Weights in early layers receive gradients so small they barely change. Training stalls.

6.4.5 Tanh — Properties and Usage

Tanh squashes into $(- 1, 1)$ :

$tanh (z) = \frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}$

Its derivative: $tanh^{'} (z) = 1 - tanh^{2} (z)$ . The maximum derivative is 1 (at $z = 0$ ).

Tanh is zero-centered, unlike sigmoid (which is centered at 0.5). Zero-centered outputs can help training converge faster because the gradients do not all share the same sign.

Pitfall: Tanh also suffers from vanishing gradients in deep networks. Its derivative approaches 0 for large absolute inputs. Multiply through many layers with $∣ z ∣ ≫ 0$ , and the gradient vanishes.

6.4.6 ReLU — Properties and Usage

ReLU (Rectified Linear Unit) is dead simple:

$ReLU (z) = max (0, z)$

Derivative: $1$ for $z > 0$ , $0$ for $z < 0$ , undefined at $z = 0$ (set to 0 in practice).

Why ReLU dominates hidden layers:

1. Cheap computation. No exponentials, no divisions. Just check if $z > 0$ . 2. No vanishing gradient for $z > 0$ . The derivative is exactly 1. Multiply by 1 through 100 layers and you still have... 1. The gradient flows cleanly. 3. Sparse activation. Negative inputs produce zero output. Out of, say, 2000 neurons, many produce negative values and contribute nothing to the forward pass. The activations become sparse — mostly zeros. Sparse representations can be more efficient and sometimes generalize better.

Scope: ReLU is not differentiable at exactly $z = 0$ . In practice, you define the subgradient as 0 at that point. Hitting exactly zero is extremely rare with floating-point arithmetic. This technicality rarely causes issues in training.

6.4.7 The Dying ReLU Problem

Pitfall — Dying ReLU: If a ReLU neuron always receives negative inputs, it always outputs zero. Its gradient is always zero. Gradient descent never updates its weights. It never contributes again. It is dead.

This can cascade: if many neurons die, information stops flowing forward. The loss plateaus. Weights in early layers stop changing. The network is stuck.

When it happens: A learning rate that is too large can push weights into a region where all inputs to a neuron become negative. Poor initialization can also cause it. Once dead, a ReLU neuron stays dead — its gradient is zero, so no update can revive it.

6.4.8 Leaky ReLU — A Fix for Dying ReLU

Leaky ReLU gives negative inputs a small, nonzero slope instead of zero:

$LeakyReLU (z) = {z α z if z > 0 if z \leq 0$

Where $α$ is a small constant like 0.01. The derivative for negative inputs is $α$ instead of 0. The neuron never completely dies — a tiny gradient always flows.

Variants:

PReLU (Parametric ReLU): $α$ is learned during training, not fixed.
ELU (Exponential Linear Unit): Uses $e^{z} - 1$ on the negative side, which is smooth at $z = 0$ .

6.4.9 Where to Use Each Activation

These are heuristics, not rigid rules:

Layer	Recommended Activation	Why
Hidden layers (default)	ReLU	Fast, no vanishing gradient for $z > 0$
Hidden layers (if dying ReLU)	Leaky ReLU, ELU	Prevents dead neurons
Output — binary classification	Sigmoid	Output in $(0, 1)$ , reads as probability
Output — multi-class	Softmax	Outputs sum to 1, probability distribution
Output — regression	Linear (identity)	No range restriction needed

Sigmoid and tanh are rarely used in hidden layers of deep networks because of vanishing gradients.

6.4.10 Computational Cost Comparison

ReLU: $max (0, z)$ — one comparison. Extremely fast.
Sigmoid: $\frac{1}{1 + e ^{- z}}$ — one exponential, one addition, one division. Relatively slow.
Tanh: $\frac{e ^{z} - e ^{- z}}{e ^{z} + e ^{- z}}$ — two exponentials, additions, division. Relatively slow.

In large networks with millions of neurons, the difference matters. ReLU trains noticeably faster than sigmoid or tanh.

6.4.11 Why Differentiability Matters for Activation Functions

Backpropagation uses the chain rule. To compute $\frac{\partial L}{\partial w}$ for a weight $w$ , you multiply derivatives along the path from the loss to $w$ . Every activation function along that path contributes its derivative as a factor in the product.

If an activation function is not differentiable at some points, you cannot compute its contribution to the gradient. The function must be differentiable — at least almost everywhere (everywhere except a set of measure zero, like a finite number of points).

This is the same reason the loss function (MSE, cross-entropy) must be differentiable. If you design your own activation function, it must be differentiable almost everywhere.

6.4.12 Practitioner Wisdom

There is no mathematical rule that says "use exactly 2 layers with exactly 100 neurons each." You try configurations. You train. You evaluate. You iterate. Architecture design is part art, part experimentation.

The same holds for activation functions. ReLU tends to work well. Sigmoid in hidden layers tends to cause trouble. But there is no proof that one is universally better. You try and see.

Q: Can different layers use different activation functions?

A: Yes. The choice is per-layer, not global. You might use ReLU for hidden layers and sigmoid for the output. Or ReLU for one hidden layer and tanh for another. Everything is experimental.

Q: Is there a mathematical formula to determine the number of layers or neurons?

A: No. These are hyperparameters — like learning rate. You set them, experiment, and adjust. Some heuristics exist (e.g., next layer having roughly half the neurons of the previous layer), but none are universally valid. There is a field called neuroevolution that uses genetic algorithms to evolve optimal architectures automatically.

Activation functions are the nonlinearity that makes depth useful. ReLU is the default for hidden layers — fast, cheap, and avoids vanishing gradients for positive inputs. Its main weakness is dying neurons, which Leaky ReLU fixes. Sigmoid and tanh are now mostly reserved for output layers where their bounded ranges serve a purpose.

6.5 Deep Feedforward Neural Network Architecture

6.5.1 Width and Depth

A deep network has two knobs: how many layers (depth) and how many neurons per layer (width). Adding layers lets the network build hierarchical abstractions. Adding neurons lets each layer capture more patterns. Both increase capacity. Neither has a magic formula for the right setting.

A deep neural network has two structural degrees of freedom:

Depth: The number of layers. More layers = deeper network. Each layer composes a new transformation on top of the previous one.
Width: The number of neurons per layer. More neurons = wider layer. Each neuron computes one scalar feature.

Both changes increase the network's expressive power. This is the set of functions it can represent. One hidden layer can approximate any continuous function. This is the universal approximation theorem. But deep networks can do it with far fewer neurons total.

6.5.2 Symbol Registry — Network Architecture

$L$ — total number of layers (counting input as layer 0, hidden layers, and output as layer $L$ ) — integer
$N_{0}$ — number of input neurons (= number of features, $d$ ) — integer
$N_{l}$ — number of neurons in hidden layer $l$ — integer, for $l = 1, 2, \dots, L - 1$
$N_{L}$ — number of neurons in output layer (= number of classes for classification) — integer
$h^{(l)}$ — activations of layer $l$ — $h^{(l)} \in R^{N_{l}}$ — vector
$W^{(l)}$ — weight matrix connecting layer $l - 1$ to layer $l$ — $W^{(l)} \in R^{N_{l - 1} \times N_{l}}$ — matrix. Each column $W_{:, j}^{(l)}$ holds the weights feeding into neuron $j$ of layer $l$ .
$b^{(l)}$ — bias vector for layer $l$ — $b^{(l)} \in R^{N_{l}}$ — vector
$Z^{(l)}$ — pre-activation at layer $l$ (before the activation function) — $Z^{(l)} \in R^{N_{l}}$ — vector

Notation note: The professor writes $Z^{(l)} = W^{(l) ⊤} h^{(l - 1)} + b^{(l)}$ . Standard texts often drop the transpose and write $Z^{(l)} = W^{(l)} h^{(l - 1)} + b^{(l)}$ with $W^{(l)} \in R^{N_{l} \times N_{l - 1}}$ . Both forms are equivalent — the transpose just sweeps the shape convention under notation. Throughout these notes we follow the professor's convention with the transpose explicit.

6.5.3 Forward Propagation — Mathematical Formulation

Forward propagation computes the output of each layer, one after another, from input to output.

For the first hidden layer ( $l = 1$ ), the input is the raw features $X$ :

$Z^{(1)} h^{(1)} = W^{(1) ⊤} X + b^{(1)} = g^{(1)} (Z^{(1)}) (linear combination) (apply activation)$

For subsequent hidden layers ( $l = 2, \dots, L - 1$ ), the input is the previous layer's output:

$Z^{(l)} h^{(l)} = W^{(l) ⊤} h^{(l - 1)} + b^{(l)} = g^{(l)} (Z^{(l)}) (linear combination) (apply activation)$

For the output layer ( $l = L$ ), the final prediction:

$Z^{(L)} y^= W^{(L) ⊤} h^{(L - 1)} + b^{(L)} = g^{(L)} (Z^{(L)}) (linear combination) (output activation)$

Here $g^{(l)}$ is the activation function for layer $l$ . Different layers can use different $g$ . Hidden layers typically use ReLU; the output layer uses sigmoid (binary), softmax (multi-class), or identity (regression).

The key insight: the output of layer $l - 1$ becomes the input (the "features") for layer $l$ . This is hierarchical feature learning. Early layers near the input detect simple patterns (edges, basic shapes). Middle layers combine simple patterns into intermediate ones (textures, parts). Later layers combine intermediate patterns into high-level concepts (objects, faces). No human designed these features — gradient descent learned them from data.

6.5.4 Information Flow

Think of a deep network as a factory assembly line. Raw materials (input features) enter at the start. Each station (layer) transforms them. The finished product (prediction) exits at the end. Quality control (loss) checks the product and sends adjustment instructions backward through every station. No station can send work back upstream — that is why it is "feedforward."

At every node, the forward pass does two things: 1. Linear combination: Sum of (weight $\times$ incoming value) + bias. 2. Nonlinear transformation: Pass that sum through the activation function.

Both happen inside the same node. The activation function is not a separate layer — it is applied elementwise to each neuron's pre-activation.

Information flows forward: input $\to$ hidden layers $\to$ output. Gradients flow backward: loss $\to$ output layer $\to$ hidden layers $\to$ input layer. This two-pass structure — forward for prediction, backward for learning — is the engine of every deep network.

Q: Does the same neuron continue from one hidden layer to the next?

A: No. The outputs of all neurons in the previous layer are passed to all neurons in the next layer (that is what "fully connected" means). Each layer has its own set of neurons with their own weights and biases. Between layers sit weighted links. Each neuron in layer $l$ receives input from every neuron in layer $l - 1$ .

6.5.5 Mini-Batch Vectorized Formulation

When processing mini-batches, you stack $B$ instances into a matrix and process them together. The batch formulation is:

$H^{(l)} = g^{(l)} (H^{(l - 1)} W^{(l)} + b^{(l)})$

Where:

$H^{(l - 1)} \in R^{B \times N_{l - 1}}$ — the outputs from the previous layer for all $B$ instances, stacked as rows.
$W^{(l)} \in R^{N_{l - 1} \times N_{l}}$ — the weight matrix.
$b^{(l)} \in R^{N_{l}}$ — the bias vector, broadcast (added to every row).
$H^{(l)} \in R^{B \times N_{l}}$ — the outputs for all $B$ instances.

Shape check: $H^{(l - 1)}$ is $B \times N_{l - 1}$ . $W^{(l)}$ is $N_{l - 1} \times N_{l}$ . Their product is $B \times N_{l}$ . Adding $b^{(l)}$ (size $N_{l}$ ) broadcasts to each of the $B$ rows. The result has the correct shape $B \times N_{l}$ .

This is the same math as the single-instance version. The matrix form lets GPUs parallelize — all $B$ instances are processed in one matrix multiplication instead of looping $B$ times.

6.5.6 Example Weight Matrices

Consider an architecture with 3 input features, 4 neurons in hidden layer 1, and 3 neurons in hidden layer 2.

Between input and hidden layer 1: $W^{(1)} \in R^{3 \times 4}$ — 12 weights, plus $b^{(1)} \in R^{4}$ — 4 biases. Total: 16 parameters.
Between hidden layer 1 and hidden layer 2: $W^{(2)} \in R^{4 \times 3}$ — 12 weights, plus $b^{(2)} \in R^{3}$ — 3 biases. Total: 15 parameters.

At every neuron, the computation is the same: sum of (weight $\times$ incoming value) + bias, then activation.

Worked: Forward pass on a tiny input

Take the architecture above (3 inputs, 4 hidden-1, 3 hidden-2). Feed it $x = [1.0, 0.5, - 0.3]$ :

Hidden layer 1: Compute $Z^{(1)} = W^{(1) ⊤} x + b^{(1)}$ . The result is a vector of 4 numbers, one per neuron. Apply ReLU: negative values become 0, positives stay. Get $h^{(1)} = ReLU (Z^{(1)})$ .

Hidden layer 2: Use $h^{(1)}$ as the new input. Compute $Z^{(2)} = W^{(2) ⊤} h^{(1)} + b^{(2)}$ . Apply ReLU again. Get $h^{(2)} = ReLU (Z^{(2)})$ — a vector of 3 numbers.

Output: Multiply by $W^{(3)}$ , add $b^{(3)}$ , apply softmax. The result is 3 probabilities summing to 1. The class with the highest probability wins.

Shape check: (3,) → (4,) → (3,) → (3 output classes). Every step preserves the expected dimensions.

6.5.7 What Differs from the Linear Model

Only two things change from the single-layer linear model: 1. Multiple layers (stacked perceptrons), not just one. 2. Nonlinear activations between layers.

Everything else — the loss function, the backpropagation algorithm, the gradient descent update — remains structurally identical. The loss is computed at the output. Gradients flow backward through the chain of layers via the chain rule. Weights at every layer get updated. The training loop you learned for the linear model scales directly to networks with 100 layers.

A deep feedforward network stacks linear transformations and nonlinear activations. Each layer's output becomes the next layer's input. This lets the network learn hierarchical features — simple patterns near the input, complex patterns near the output. The batch formulation $H^{(l)} = g^{(l)} (H^{(l - 1)} W^{(l)} + b^{(l)})$ captures all of this in one clean matrix equation.

6.6 Backpropagation and the Chain Rule

Symbol Registry

$L$ — loss (scalar) — measures the error between prediction and true label
$W_{3}$ — a specific weight in the first hidden layer — scalar parameter
$y^$ — predicted output — vector
$Z^{(l)}$ — pre-activation at layer $l$ — vector, before applying the activation function
$h^{(l)}$ — activation at layer $l$ — vector, after applying the activation function
$\frac{\partial L}{\partial θ}$ — gradient of the loss with respect to parameter $θ$ — tells us which way to move $θ$ to reduce $L$
$σ^{'} (z)$ — derivative of the sigmoid — $σ (z) (1 - σ (z))$
$ReLU^{'} (z)$ — derivative of ReLU — 1 for $z > 0$ , 0 for $z < 0$

6.6.1 Why the Chain Rule Matters

A weight deep in the first hidden layer has no direct line to the loss. It sits at the input side. The loss sits at the output. How do you figure out which way to nudge that weight to reduce the loss? The chain rule is the answer — it connects the loss to every weight in the network through a chain of partial derivatives.

Imagine a long row of dominoes. You push the last one. You want to know how much the first one moved. You cannot see it directly. But you know how each domino affects the next. Multiply those effects together. Now you know how your push at the end reached the start. Backpropagation works the same way. It starts from the loss and works backward. It multiplies local effects at each layer. The result is the gradient for every weight.

Where the analogy breaks: dominoes fall once. Backpropagation does this for every batch. It runs thousands of times during training.

Backpropagation computes the gradient of the loss $L$ for every parameter. It uses the chain rule of calculus. If $a$ affects $b$ and $b$ affects $c$ , then $\frac{\partial c}{\partial a} = \frac{\partial c}{\partial b} \cdot \frac{\partial b}{\partial a}$ . In a network, the loss is the final output. The weights are the inputs to the computation graph. The chain rule connects them. It chains together partial derivatives from the loss backward to each weight.

6.6.2 Walking Through the Chain

Suppose you want $\frac{\partial L}{\partial W _{3}}$ , the gradient of the loss with respect to a weight in the first hidden layer. The path backward is:

1. $\frac{\partial L}{\partial y ^}$ — How much does the loss change when the output prediction changes? This is the gradient of the loss function (e.g., cross-entropy).

2. $\frac{\partial y ^}{\partial Z ^{(L)}}$ — The output prediction came from applying the output activation (sigmoid/softmax) to the last layer's pre-activation. What is the derivative of that activation?

3. $\frac{\partial Z ^{(L)}}{\partial h ^{(L - 1)}}$ — The last layer's pre-activation is $W^{(L) ⊤} h^{(L - 1)} + b^{(L)}$ . How much does it change when the previous layer's activations change? This gives $W^{(L)}$ .

4. $\frac{\partial h ^{(L - 1)}}{\partial Z ^{(L - 1)}}$ — Those activations came from applying the activation function $g^{(L - 1)}$ to $Z^{(L - 1)}$ . Differentiate that activation.

5. $\dots$ continue backward through all hidden layers $\dots$

6. $\frac{\partial Z ^{(l + 1)}}{\partial W _{3}}$ — Eventually, you reach the layer containing $W_{3}$ . Its pre-activation uses $W_{3}$ as one of the weights in the linear combination. The derivative of a linear function with respect to one of its weights is the corresponding input value.

Each step is one partial derivative. Multiply them all together:

$\frac{\partial L}{\partial W _{3}} = \frac{\partial L}{\partial y ^} \cdot \frac{\partial y ^}{\partial Z ^{(L)}} \cdot \frac{\partial Z ^{(L)}}{\partial h ^{(L - 1)}} \cdot \dots \cdot \frac{\partial Z ^{(l + 1)}}{\partial W _{3}}$

This is the chain rule. It is the mathematical engine of deep learning.

Worked: Gradient for a 2-layer network with one hidden neuron

Consider a tiny network. It has input $x$ . Hidden neuron: $h = ReLU (w_{1} x + b_{1})$ . Output: $y^= σ (w_{2} h + b_{2})$ . The loss is MSE: $L = \frac{1}{2} (y^- y)^{2}$ .

To find $\frac{\partial L}{\partial w _{1}}$ (how much the first weight should change):

$\frac{\partial L}{\partial w _{1}} = \partial L / \partial y^(y^- y) \cdot σ^{'} (z_{2}) y^(1 - y^) \cdot \partial z_{2} / \partial h w_{2} \cdot ReLU^{'} (z_{1}) 1 [w_{1} x + b_{1} > 0] \cdot \partial z_{1} / \partial w_{1} x$

Each factor comes from differentiating one step of the forward pass. The chain rule strings them together.

6.6.3 The Activation Function Gets Differentiated Too

Every activation function along the path contributes its derivative. For sigmoid: $σ^{'} (z) = σ (z) (1 - σ (z))$ . For tanh: $tanh^{'} (z) = 1 - tanh^{2} (z)$ . For ReLU: derivative is 1 for $z > 0$ , 0 for $z < 0$ .

This is why activation functions must be differentiable (at least almost everywhere). If a function has no derivative, you cannot include its factor in the chain rule product.

6.6.4 Connection to Lecture 5

In the previous lecture, you saw the sigmoid function differentiated as part of backpropagation for a single-layer network. The same idea extends to every layer. The backward pass differentiates every activation function in the chain. The only difference is the length of the chain. One layer becomes many layers. But the rule is the same.

Pitfall: A common beginner mistake is to think "backpropagation is a separate algorithm from gradient descent." Backpropagation computes the gradient. Gradient descent uses that gradient to update weights. They are two parts of one training loop — forward pass, backward pass (backprop), update. Not separate.

Backpropagation is the chain rule applied to neural networks. Starting from the loss, it walks backward through every layer, multiplying local derivatives. The result is the gradient of the loss with respect to every weight. Gradient descent then uses those gradients to update the weights. This single algorithm scales from simple perceptrons to networks with billions of parameters.

6.7 Vanishing and Exploding Gradients

Symbol Registry

$L$ — loss (scalar) — the objective we minimize
$θ_{l}$ — a parameter (weight or bias) in layer $l$
$h^{(l)}$ — activation of layer $l$ — vector
$Z^{(l)}$ — pre-activation of layer $l$ — vector
$\frac{\partial L}{\partial θ _{l}}$ — gradient of the loss with respect to $θ_{l}$ — determines the weight update
$σ^{'} (z)$ — derivative of sigmoid — maximum 0.25, approaches 0 at tails
$tanh^{'} (z)$ — derivative of tanh — maximum 1, approaches 0 at tails
$ReLU^{'} (z)$ — derivative of ReLU — 1 for $z > 0$ , 0 for $z < 0$
$α$ — slope for negative inputs in Leaky ReLU — small constant (e.g., 0.01)

6.7.1 What Is the Vanishing Gradient Problem?

You multiply 0.1 by 0.1 by 0.1 ten times. You get $1 0^{- 10}$ — essentially zero. Now imagine those numbers are gradients flowing backward through ten layers. The weights in the first layer get a gradient so tiny that they barely move. Training stalls. This is the vanishing gradient problem, and it is why deep networks were nearly impossible to train before ReLU.

During backpropagation, the gradient for a weight in layer $l$ is a product of many factors from all later layers:

$\frac{\partial L}{\partial θ _{l}} = \frac{\partial L}{\partial h ^{(L)}} \cdot \frac{\partial h ^{(L)}}{\partial Z ^{(L)}} \cdot \frac{\partial Z ^{(L)}}{\partial h ^{(L - 1)}} \cdot \dots \cdot \frac{\partial h ^{(l + 1)}}{\partial Z ^{(l + 1)}} \cdot \frac{\partial Z ^{(l + 1)}}{\partial θ _{l}}$

Each factor is a derivative. If most factors are less than 1, the product shrinks exponentially with depth. The gradient vanishes — it becomes too small for meaningful weight updates. Early layers learn nothing. Training stalls.

This happens especially with sigmoid and tanh activations. The sigmoid derivative is at most 0.25 and approaches 0 for large $∣ z ∣$ . Tanh's derivative is at most 1 and also approaches 0 for large $∣ z ∣$ . Through 5-10 layers of sigmoids, the gradient effectively disappears.

6.7.2 What Is the Exploding Gradient Problem?

The opposite problem: factors in the chain rule are consistently greater than 1. Multiply through many layers and the gradient grows exponentially. The weight updates become enormous. The loss oscillates wildly or hits NaN (not a number). Weights diverge instead of converging.

This is more common with poor weight initialization — if initial weights are too large, the activations and their derivatives are large, and the gradient product explodes.

Think of vanishing gradients as a whisper that fades to silence — early layers hear nothing. Exploding gradients are a shout that becomes a deafening roar — early layers are blasted with updates so large the network breaks. You want a steady, clear signal that reaches every layer at a usable volume.

6.7.3 Activation Functions and Gradient Flow

Activation	Derivative Range	Vanishing Risk
Sigmoid	max 0.25, approaches 0 at tails	Very high
Tanh	max 1, approaches 0 at tails	High
ReLU	1 for $z > 0$ , 0 for $z < 0$	Low (for positive inputs)
Leaky ReLU	1 for $z > 0$ , $α$ (small) for $z < 0$	Low

ReLU is preferred in hidden layers because its derivative for positive inputs is exactly 1. Backpropagating through a chain of 1s does not shrink or expand the gradient. The gradient flows cleanly.

Q: During backpropagation, does the gradient value always decrease?

A: No. It depends on the activation function's derivative. With ReLU, the derivative is 1 for positive inputs — no shrinking. With some functions, the derivative can be larger than 1, making the gradient grow (exploding). With many sigmoid layers, the derivative is well below 1, so the gradient shrinks (vanishing). There is no guarantee it only goes one way.

Q: What does vanishing gradient look like in practice?

A: You train for thousands of epochs. The loss stays high. The weights in early layers show almost no change — track them and you will see they are static. The error is large but the adjustments never reach the front of the network. A common culprit: sigmoid in hidden layers of a deep network.

Q: How do you detect exploding gradients?

A: The loss goes to NaN or oscillates wildly. Weight values change by orders of magnitude between updates. If you monitor weights and see them blowing up ( $10, 100, 10000, \dots$ ), that is exploding gradients. The standard fix: gradient clipping — cap the gradient magnitude at a threshold (e.g., clip all gradients to have norm $\leq 1.0$ ). This is built into PyTorch and TensorFlow. More on this after the midterm.

6.7.4 Why ReLU Avoids the Problem

For any positive input, the ReLU derivative is exactly 1. So the gradient passes through unchanged. There is no multiplication by a tiny fraction. The gradient does not vanish.

For negative inputs, the ReLU derivative is 0 — no gradient flows at all. That is a different problem. It is dying ReLU (covered in section 6.4.7). It is not vanishing gradient in the traditional sense. A dead neuron does not contribute, but it does not shrink the gradients of other neurons.

6.7.5 Sparse Activation in ReLU

When ReLU is applied to a layer with 2000 neurons, many produce negative outputs. Those get clipped to zero. The neurons that produce zeros are "off." The others are "on." The representation is sparse — most entries are zero.

Sparse representations can be easier to interpret, more memory-efficient, and sometimes generalize better. But if too many neurons are always off, no information passes — the dying ReLU scenario.

6.7.6 Industry Practice

In practice: start with ReLU for hidden layers. If you observe dying ReLU symptoms — loss plateaus early, many dead neurons — try Leaky ReLU or ELU. Sigmoid and tanh are reserved for output layers. Use them when the output range must match a specific interval, like probability $[0, 1]$ for binary classification or $[- 1, 1]$ for some generative models. Gradient clipping is a standard safeguard against exploding gradients in all deep learning frameworks.

6.8 Parameter Counting in Deep Networks

Symbol Registry

$L$ — total number of weight layers in the network — integer
$N_{l - 1}$ — number of neurons in layer $l - 1$ (the "previous" layer) — integer
$N_{l}$ — number of neurons in layer $l$ (the "current" layer) — integer
$N_{l - 1} \times N_{l}$ — count of link weights between two fully connected layers — integer
$N_{l}$ — count of biases in layer $l$ (one per neuron) — integer

6.8.1 The Counting Formula

Every connection between two neurons is a learnable number — a weight. Every neuron has one more — a bias. Count them all up and you know how many knobs gradient descent will tune. For a fully connected network, the count is mechanical: multiply the neurons in two adjacent layers, then add the biases of the second layer.

For any two consecutive layers (layer $l - 1$ with $N_{l - 1}$ neurons and layer $l$ with $N_{l}$ neurons), the learnable parameters are:

$Parameters = N_{l - 1} \times N_{l} + N_{l}$

Where:

$N_{l - 1} \times N_{l}$ — the link weights. Every neuron in layer $l - 1$ connects to every neuron in layer $l$ .
$+ N_{l}$ — the biases. One per neuron in layer $l$ .

Total parameters across the entire network (with $L$ weight layers, counting input as layer 0):

$Total = l = 1 \sum L (N_{l - 1} \times N_{l} + N_{l})$

The input layer (layer 0) has no learned parameters — it just holds the data.

6.8.2 Worked Numerical Example

Architecture: Input $N_{0} = 784$ (flattened 28×28 image), Hidden 1 $N_{1} = 256$ , Hidden 2 $N_{2} = 128$ , Output $N_{3} = 10$ (10 classes).

Layer 1 (Input → Hidden 1):

$N_{0} \times N_{1} + N_{1} = 784 \times 256 + 256 = 200, 704 + 256 = 200, 960$

Layer 2 (Hidden 1 → Hidden 2):

$N_{1} \times N_{2} + N_{2} = 256 \times 128 + 128 = 32, 768 + 128 = 32, 896$

Layer 3 (Hidden 2 → Output):

$N_{2} \times N_{3} + N_{3} = 128 \times 10 + 10 = 1, 280 + 10 = 1, 290$

Total: $200, 960 + 32, 896 + 1, 290 = 235, 146$ parameters.

Sense check: Most parameters live between the input and the first hidden layer — that is where the high-dimensional raw data meets the network. The input layer is the widest, so the first weight matrix dominates the parameter count.

This is a modest network. It already has over 235,000 parameters. Deep networks can have millions or billions. This scale explains why deep learning needs powerful hardware (GPUs, TPUs) and why backpropagation must be computationally efficient.

Pitfall: Forgetting to count biases. Every layer with $N_{l}$ neurons has $N_{l}$ biases. They are small in number compared to weights, but they are not zero. On an exam, omitting biases from the count costs marks.

Pitfall: Counting the input layer as having parameters. The input layer ( $N_{0}$ neurons) stores data, not learnable weights. Parameters only exist between layers.

6.8.3 The Formula Generalized

For $L$ total weight layers (input = layer 0, output = layer $L$ ), sum over $l = 1$ to $L$ :

$Total parameters = l = 1 \sum L (N_{l - 1} \cdot N_{l} + N_{l})$

This is the formula to memorize. It appears on exams.

Parameter counting is mechanical: for each pair of adjacent layers, multiply the sizes and add the biases. The first weight matrix dominates when the input dimension is large (e.g., images). The formula is $\sum_{l = 1}^{L} (N_{l - 1} \times N_{l} + N_{l})$ . Count biases — exam questions check for this.

6.9 Summary of Key Architectural Decisions

Symbol Registry

$η$ — learning rate — small positive scalar — controls step size of each weight update
$L$ — depth — number of layers including input, hidden, and output
$N_{l}$ — width of layer $l$ — number of neurons in that layer
Early stopping — monitoring validation loss and halting when improvement falls below a threshold

6.9.1 Hyperparameters of a Deep Network

Building a neural network is like cooking without a recipe. You know the ingredients — layers, neurons, activation functions, learning rate. But there is no formula that says "exactly 3 layers, 128 neurons each." You try things. You taste (evaluate). You adjust. This is the art of deep learning.

Hyperparameters are all the choices you make before training begins. They are not learned from data — you set them. They include:

Number of layers (depth)
Number of neurons per layer (width)
Choice of activation function per layer
Learning rate $η$
Batch size
Number of epochs (or a stopping criterion)
Weight initialization scheme (He, Xavier, random-small)

None of these have a mathematical formula that guarantees optimality. Unlike linear regression, which has a closed-form solution for the optimal weights, deep networks have no such guarantee for their hyperparameters. You experiment.

6.9.2 Early Stopping as a Convergence Mechanism

Instead of fixing the number of epochs, you can watch the validation loss and stop when it stops improving. This is early stopping.

When the improvement between successive epochs becomes negligible — say the delta falls below 0.006 — stop training. This avoids wasted computation and can help prevent overfitting. If the validation loss starts increasing, the model is beginning to overfit. Training loss may still be decreasing. Early stopping catches this divergence.

In the linear model demo from section 6.1, the loss curve went flat. This happened after about 700 to 800 epochs. There was no further improvement. Running all 1000 epochs was unnecessary. Early stopping would have halted around epoch 750. That saves 25% of the training time.

6.9.3 Feedforward Definition

A deep feedforward neural network has one or more hidden layers between input and output.

Forward pass: Input $\to$ hidden layers $\to$ output. At every hidden node, the forward signal undergoes a nonlinear transformation via an activation function.
Backward pass: Loss $\to$ output $\to$ hidden layers $\to$ input. Gradients flow back via the chain rule.

The network is called "feedforward" because there are no cycles — information never loops back. The output of layer $l$ depends only on the output of layer $l - 1$ , not on layer $l + 1$ or on layer $l$ itself from a previous time step. (Recurrent networks, covered later, do have cycles.)

6.9.4 What Stays the Same from the Linear Model

The objective function (loss), the learning algorithm (backpropagation + gradient descent), and the overall training loop stay structurally identical. The only new elements are multiple layers and nonlinear activations between them.

Everything you learned about classification, gradient descent, confusion matrices, and evaluation metrics still applies. The network got deeper. The fundamentals did not change.

Deep feedforward networks are built from the same parts as the simple linear model — just more of them, stacked with nonlinearities between. Hyperparameters control depth, width, learning rate, and initialization. None have a guaranteed optimum — you experiment. Early stopping saves time and fights overfitting. ReLU is the default hidden-layer activation. Everything else scales directly from the single-neuron case.

Exam Guidance Summary

These are the key examinable points from Lecture 6, consolidated from the professor's guidance throughout the lecture.

Parameter Counting

Exam note: Expect numerical questions on parameter counting. You will be given an architecture with specific layer sizes. Compute the total learnable parameters. The formula per layer pair is $N_{prev} \times N_{curr} + N_{curr}$ . Remember to count biases. They are often forgotten and cost marks. Common format: "An MLP has input size 784, two hidden layers of 256 and 128, and 10 output classes. How many parameters?" Answer: 235,146.

XOR and Nonlinearity

Exam note: Explain why a linear neural network cannot solve XOR. Explain how nonlinear activation functions enable it. Key points: a single linear boundary cannot separate XOR points. Stacking linear layers without activations collapses to one linear model. Nonlinear activations break the chain. They add expressive power.

Activation Functions

Exam note: Know the properties of common activation functions. Study sigmoid, tanh, ReLU, Leaky ReLU. For each: the formula, output range, derivative behavior, and main use case. Be able to compare them on: computational cost, vanishing gradient risk, and where each is used.

Vanishing and Exploding Gradients

Exam note: Understand vanishing and exploding gradient problems. Describe the chain rule's role. Explain why multiplying many small derivatives causes gradients to vanish. Explain why multiplying many large derivatives causes them to explode. Know that ReLU avoids vanishing for positive inputs. Its derivative is exactly 1.

Confusion Matrix

Exam note: Interpret a confusion matrix. Identify which off-diagonal entry maps to which error type. Connect the error type to precision and recall. Know which error is more dangerous in context. For example, in medical diagnosis, false negatives cost lives.

One-Hot Encoding and Output Activations

Exam note: Know the difference between one-hot encoding and label encoding. Understand when to use softmax vs sigmoid at the output layer. Softmax is for multi-class. Outputs sum to 1. Sigmoid is for binary or multi-label. Each output is independent.

Training Mechanics

Exam note: Explain early stopping. Monitor validation loss and stop when improvement plateaus. Explain batch shuffling. It prevents class-skewed batches from destabilizing training. Explain the effect of learning rate on convergence. Too high gives oscillations. Too low gives slow convergence.

Practical Exercises

Exam note: The professor recommends these hands-on exercises. Change the learning rate in the provided code. Observe convergence behavior. Use random forest to identify important features. Retrain with only the top features. Create polynomial features. Compare against the baseline. Try different datasets beyond breast cancer and iris.

Key Industry Applications

Evaluation Metrics

scikit-learn provides one-line implementations for confusion matrices, precision, recall, and F1 score. In production code, use the library functions — they are tested and optimized. The manual implementations in the lecture are for learning the formulas. Knowing what is under the hood matters when you debug unexpected metrics.

Weight Initialization

He initialization, Xavier (Glorot) initialization, and LeCun initialization are built into every major deep learning framework (PyTorch, TensorFlow, JAX). They are the defaults when you create a layer. Xavier initialization draws weights from a distribution. Its variance is $σ^{2} = \frac{2}{n _{in} + n _{out}}$ . This keeps the variance of activations and gradients stable across layers. He initialization is optimized for ReLU networks. It uses $σ^{2} = \frac{2}{n _{in}}$ .

Dimensionality Reduction

PCA (Principal Component Analysis) is widely used for visualizing high-dimensional data in 2D or 3D. It is a standard preprocessing step in exploratory data analysis. In industry, PCA also serves as a feature reduction technique. It projects high-dimensional data onto fewer dimensions. It preserves maximum variance.

Feature Importance

Random forest is commonly used for feature importance. It is a standard preprocessing step in industry ML pipelines. Before training a neural network, practitioners run a random forest first. It identifies the most predictive features. The network then trains on only those features. This reduces dimensionality. It can speed up training without hurting accuracy.

Architecture Search

Neuroevolution uses genetic algorithms to automatically discover optimal neural network architectures. While not yet mainstream in production, it is an active research area. Companies with large compute budgets (Google, Meta) use automated architecture search (NAS — Neural Architecture Search) to design state-of-the-art models.

Gradient Clipping

Gradient clipping is a standard technique in all deep learning frameworks to prevent exploding gradients. In PyTorch: `torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)`. In TensorFlow: `tf.clip_by_norm(grad, clip_norm)`. It caps the gradient magnitude at a threshold, preventing any single update from being destructively large.

Activation Functions in Production

ReLU is the default activation in almost all modern deep learning architectures — ResNet, Transformer, VGG, Inception, EfficientNet. Leaky ReLU and its variants (PReLU, ELU) are used when dying ReLU is observed. Sigmoid and tanh appear primarily in output layers and in specialized architectures like LSTMs (where sigmoid controls gating).

Development Platforms

Google Colab provides free GPU access for prototyping and learning. It is the recommended platform for the code examples in this lecture. PyTorch and TensorFlow are the two dominant frameworks — both provide built-in implementations of every concept covered in this lecture. Kaggle competitions and computer vision benchmarks (ImageNet, COCO) drive architecture innovation. Successful architectures become standard models — VGG (2014), ResNet (2015), DenseNet (2017), EfficientNet (2019), Transformer (2017). All of these are results of extensive experimentation, not mathematical derivation. The field progresses through empirical discovery.

Deep learning in production uses the same fundamentals taught in this lecture — ReLU activations, cross-entropy loss, gradient descent, backpropagation. The frameworks (PyTorch, TensorFlow) handle the heavy computation. The practitioner's job is architecture design, hyperparameter tuning, and debugging — skills built on the foundations from this lecture.

DNN Lecture 06 notes · Deep Feedforward Neural Networks

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

16.1 Linear Neural Networks: Code Walkthrough and Implementation

Single-neuron binary classifier, sigmoid output, binary cross-entropy, confusion matrix.

26.2 Multi-Class Classification with Linear Neural Networks

Softmax output, one-hot labels, categorical cross-entropy, multi-class confusion matrix.

36.3 Limitations of Linear Networks and the XOR Problem

Why one linear boundary fails on XOR and why stacked linear layers stay linear.

46.4 Activation Functions: Introducing Non-Linearity

Sigmoid, tanh, ReLU, Leaky ReLU, dying ReLU, differentiability and usage.

56.5 Deep Feedforward Neural Network Architecture

Width, depth, forward propagation, hierarchical feature learning, batch vectorization.

66.6 Backpropagation and the Chain Rule

Chain-rule gradient of the loss with respect to every weight, backward pass.

76.7 Vanishing and Exploding Gradients

How gradient products shrink or grow across depth, and ReLU's role.

86.8 Parameter Counting in Deep Networks

Mechanical count of weights and biases per layer pair, summed over the network.

96.9 Summary of Key Architectural Decisions

Hyperparameters, early stopping, feedforward definition, what stays the same.

10Exam Guidance Summary

Professor's consolidated examinable points across the lecture.

11Key Industry Applications

Production practices: evaluation metrics, initialization, PCA, gradient clipping, platforms.

Postgraduate students in Deep Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Linear Neural Network for Binary Classification

Must-know: A linear (single-neuron) binary classifier computes hat-y = sigma(w^T x + b) and is trained by gradient descent on binary cross-entropy. It draws one straight-line decision boundary.

$y^= σ (z) = \frac{1}{1 + e ^{- z}}, z = w^{⊤} x + b$

Top pitfall: Forgetting to normalize features to zero mean and unit variance, or initializing all weights to zero so every update is identical (symmetry).

Self-check: Why does a single linear neuron always produce a straight-line (hyperplane) decision boundary?

Connects to: Sigmoid activation, Binary cross-entropy, Gradient descent, Confusion matrix

Multi-Class Classification

Must-know: For K classes use K output neurons with softmax; compare the softmax probabilities to a one-hot label using categorical cross-entropy. Softmax forces the K probabilities to sum to 1.

$y^_{k} = softmax (Z_{k}) = \frac{e ^{Z_{k}}}{\sum _{j = 1}^{K} e ^{Z_{j}}}, L = - k = 1 \sum K y_{k} lo g (y^_{k})$

Top pitfall: Feeding integer labels instead of one-hot vectors, or forgetting that softmax couples the classes so raising one lowers the others.

Self-check: With one-hot labels, why does only the true class's term survive in the categorical cross-entropy loss?

Connects to: One-hot encoding, Softmax, Binary cross-entropy, Sigmoid

XOR and the Limits of Linear Networks

Must-know: A single linear boundary cannot separate XOR. Worse, stacking linear layers without nonlinear activation still collapses to one big linear model, so depth adds nothing.

$h^{(2)} = W^{(2)} (W^{(1)} x + b^{(1)}) + b^{(2)} = (W^{(2)} W^{(1)}) x + b^{'}$

Top pitfall: Assuming 'more layers = more power' while every layer stays linear (e.g., identity activation). The network is still one linear transform.

Self-check: Why does composing many linear layers never increase the model's capacity beyond a single linear map?

Connects to: Activation functions, ReLU, Perceptron (Lecture 3)

Activation Functions

Must-know: Nonlinear activations (sigmoid, tanh, ReLU, Leaky ReLU) are what give depth its power. ReLU is the default hidden-layer choice: cheap and no vanishing gradient for positive inputs.

$ReLU (z) = max (0, z), LeakyReLU (z) = max (α z, z)$

Top pitfall: Using sigmoid in hidden layers (vanishing gradients), or using identity activation (the network stays linear).

Self-check: Why does ReLU avoid vanishing gradients for z > 0 while sigmoid does not?

Connects to: Vanishing gradients, XOR, Differentiability

Deep Feedforward Architecture & Forward Propagation

Must-know: A deep network stacks linear transforms and nonlinear activations; each layer's output becomes the next layer's input, enabling hierarchical feature learning from edges up to objects.

$Z^{(l)} = W^{(l) ⊤} h^{(l - 1)} + b^{(l)}, h^{(l)} = g^{(l)} (Z^{(l)})$

Top pitfall: Confusing the weight-matrix orientation (the professor's transpose convention) when writing the forward equations.

Self-check: In the batch form, what are the shapes of H^{(l-1)} and W^{(l)}, and how is the bias added?

Connects to: Mini-batch vectorization, Backpropagation, Activation functions

Backpropagation and the Chain Rule

Must-know: Backpropagation is the chain rule applied to the network: it walks backward from the loss, multiplying local derivatives to get the gradient of the loss with respect to every weight. Gradient descent then updates the weights.

$\frac{\partial L}{\partial W _{3}} = \frac{\partial L}{\partial y ^} \cdot \frac{\partial y ^}{\partial Z ^{(L)}} \cdot \dots \cdot \frac{\partial Z ^{(l + 1)}}{\partial W _{3}}$

Top pitfall: Thinking backpropagation is a separate algorithm from gradient descent. It only computes the gradient; gradient descent uses it to update weights.

Self-check: How does the gradient 'reach' a weight sitting in the very first hidden layer?

Connects to: Chain rule, Gradient descent, Vanishing gradients

Vanishing and Exploding Gradients

Must-know: Multiplying many small derivatives (sigmoid/tanh) makes gradients vanish; many large ones make them explode. ReLU's derivative of exactly 1 for positive inputs lets the gradient flow cleanly.

$\frac{\partial L}{\partial θ _{l}} = path \prod (successive derivatives)$

Top pitfall: Assuming the gradient always shrinks. With ReLU it can stay at 1, and with some functions it can grow without bound (exploding).

Self-check: Why does a chain of sigmoid derivatives shrink exponentially as the network gets deeper?

Connects to: Activation functions, ReLU, Gradient clipping

Parameter Counting

Must-know: For each pair of adjacent layers count N_{l-1} x N_l weights plus N_l biases, then sum over all layers. Never forget the biases, and never count parameters in the input layer.

$Total = l = 1 \sum L (N_{l - 1} \times N_{l} + N_{l})$

Top pitfall: Omitting biases from the count, or counting the input layer (which only holds data, not learnable weights).

Self-check: For a 784 -> 256 -> 128 -> 10 network, how many parameters are in the first weight matrix alone?

Connects to: Architecture, Weight matrices

Hyperparameters and Early Stopping

Must-know: Depth, width, learning rate, batch size and initialization are hyperparameters set before training. Early stopping halts when validation loss stops improving, saving time and fighting overfitting.

$w \leftarrow w - η \nabla_{w} L$

Top pitfall: Fixing the number of epochs instead of watching validation loss, which wastes compute and can overfit.

Self-check: What signal tells you it is time to stop training early?

Connects to: Gradient descent, Overfitting, Learning rate

Confusion Matrix, Precision, Recall, F1

Must-know: The confusion matrix's off-diagonal cells are errors. Precision, recall and F1 summarize classifier quality; in medical diagnosis recall (catching actual positives) usually matters most.

$precision = \frac{T P}{T P + F P}, recall = \frac{T P}{T P + F N}, F 1 = 2 \frac{precision \cdot recall}{precision + recall}$

Top pitfall: Ignoring which error type costs more in context. A false negative (missed cancer) is usually far worse than a false positive.

Self-check: In cancer screening, why is a false negative more dangerous than a false positive?

Connects to: Binary classification, Multi-class evaluation, Class imbalance

Mini-Batch Vectorized Forward Pass

Must-know: Stacking B instances into a matrix lets one matrix multiplication process a whole batch at once. This vectorization is what makes GPU training fast.

$H^{(l)} = g^{(l)} (H^{(l - 1)} W^{(l)} + b^{(l)})$

Top pitfall: Forgetting that the bias vector is broadcast across every row of the batch, not multiplied.

Self-check: For a batch of size B and layer widths N_{l-1}, N_l, what is the shape of H^{(l)}?

Connects to: Forward propagation, Architecture

Multi-Label vs Multi-Class

Must-know: Multi-class forces exactly one class via softmax. Multi-label uses K independent sigmoid outputs with a sum of binary cross-entropies, so one instance can belong to several classes at once.

$L_{multi-label} = k = 1 \sum K [- y_{k} lo g (y^_{k}) - (1 - y_{k}) lo g (1 - y^_{k})], y^_{k} = σ (Z_{k})$

Top pitfall: Using softmax when an instance can belong to several classes; softmax makes the classes compete so 'cat AND dog' is impossible.

Self-check: Why can softmax never represent 'this image contains both a cat and a dog'?

Connects to: Softmax, Sigmoid, One-hot encoding

Previous LectureLecture 05 · Gradient Descent Variants, Classification, and Evaluation

Next LectureLecture 07 · Deep Feed-Forward Neural Networks

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.