Deep Neural Networks

Gradient Descent Variants, Classification, and Evaluation

📅 Published: 2026-07-15

🎓 Level: postgraduate

👥 Audience: Postgraduate students studying Deep Neural Networks

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Linear regression and mean squared error — covered in Lecture 3 (Perceptron Learning and Introduction to Regression) and Lecture 4 (Linear Neural Networks for Regression).
Gradient descent (batch, mini-batch, SGD) — covered in Lecture 4, section 4.6.
Perceptron, forward and backward propagation — covered in Lecture 3, sections 3.1 and 3.6.
Activation functions (identity, sigmoid, ReLU) — covered in Lecture 4, section 4.4.

Gradient Descent Variants, Classification, and Evaluation

Hook: You have trained a model. Every line of code looks right. Yet the loss keeps climbing. Before adding more layers or data, check three dials: how much to trust each gradient, how many examples per step. And when to stop.

This lecture picks up from linear regression. It builds toward the full classification toolkit. You will compare three gradient descent strategies. You will apply them to binary and multi-class problems. You will learn to evaluate classifiers with the right metrics.

5.1 Recap of Linear Regression and Debugging Tips

Hook: Why does a model sometimes refuse to learn even when your equations are right? The difference between a model that converges smoothly and one that explodes into NaN often comes down to one number: the learning rate.

Intuition + Analogy: Training a neural network is like adjusting the temperature of a shower. You turn the knob a little, wait. And check the result. Turn too far and the water scalds. Turn too little and you barely notice a change. The learning rate is how much you twist the knob each time. The batch size is how long you wait before checking. The stopping criterion is deciding when the water is warm enough to step in. Like a shower, every knob interacts. A big twist needs less waiting. But risks overshoot.

5.1.1 Morning Session Review

Recap of the single-perceptron training loop. The model uses a linear regression equation. It has an identity activation function. The output equals the input. The loss is mean squared error (MSE):

J (w, b) = \frac{1}{N} i = 1 \sum N (y^{(i)} - y^^{(i)})^{2}

$w$ : weight vector (learned parameters that scale each input feature)
$b$ : bias term (shifts the decision boundary, a scalar learned alongside weights)
$N$ : total number of training instances
$y^{(i)}$ : true target value for instance $i$
$y^^{(i)}$ : predicted value for instance $i$ , computed as $y^= w^{⊤} x + b$

MSE has a useful property: the curve is differentiable everywhere — no kinks, no corners. The gradient is smooth across the entire loss surface, which makes gradient-based optimization work cleanly.

After computing the loss at the output layer, backpropagation runs the computation in reverse. First you figure out the error at the output — how much the prediction missed. Then you propagate that adjustment backward through the network, from the output layer to the input layer. The learning rate $η$ — a small scalar — controls how much you change the weights per step. Weights are updated, and the loop repeats until a stopping condition fires.

Worked Example — Single MSE Step. Suppose you have one training instance: $x = 2$ , $y = 5$ . Initial weights: $w = 0$ , $b = 0$ . Learning rate $η = 0.01$ .

Forward pass: $y^= 0 \cdot 2 + 0 = 0$

Loss: $J = (5 - 0)^{2} = 25$

Gradient: $\frac{\partial J}{\partial w} = 2 \cdot (0 - 5) \cdot 2 = - 20$ , $\frac{\partial J}{\partial b} = 2 \cdot (0 - 5) = - 10$

Update: $w = 0 - 0.01 \cdot (- 20) = 0.2$ , $b = 0 - 0.01 \cdot (- 10) = 0.1$

Sense check: The new prediction is $0.2 \cdot 2 + 0.1 = 0.5$ , closer to the target of 5. One step moved the right way.

Scope — When MSE works well. MSE assumes the errors are roughly symmetric around zero with constant variance. It works best when outliers are rare. If your data has extreme outliers, the squared term amplifies their effect — one bad point can yank the entire model. In that case, consider MAE (mean absolute error) or Huber loss, which are less affected by outliers.

Assumption: The target values are continuous and the noise is roughly Gaussian. If your target is categorical (0 or 1), MSE is a poor fit. Use cross-entropy instead. You will see this in Section 5.6.

Visual Intuition. Imagine a plot of loss (vertical axis) against weight value (horizontal axis). The MSE curve is a smooth U-shaped parabola — one global minimum, no bumps. Gradient descent starts somewhere on the curve and takes steps downhill. The steepness of the slope tells you how big the next step should be. Near the bottom, the slope flattens. So steps naturally shrink. The whole process looks like a ball rolling into the bottom of a bowl and settling there.

5.1.2 Debugging Tips for Training

Pitfall — Loss keeps increasing. You set all parameters correctly, but the loss rises instead of falling. The learning rate is too large — each step overshoots the minimum and lands higher up the opposite side. Fix: Reduce $η$ . Try halving it and watch the loss curve.

Pitfall — Loss becomes NaN. The model is overshooting so badly that the weights explode into numeric overflow. Fix: Reduce $η$ drastically. If the loss was climbing before NaN appeared, cut $η$ by a factor of 10 or more.

Pitfall — Loss refuses to drop. The learning rate may be too small, making progress invisible. Or your weights may be stuck at a plateau. Fix: Try increasing $η$ slightly. If nothing changes, check your update rule — you might have a sign error (adding the gradient instead of subtracting it).

When to stop training. The prof gave four approaches:

Fixed iterations. Set a number and walk away. Simple. But wasteful if you converge early.
Loss threshold. Stop when the change in loss between epochs drops below a tiny number (e.g., $1 0^{- 6}$ ).
Gradient threshold. Stop when the gradient magnitude falls below a threshold. If the slope is nearly flat, you are at a minimum.
Early stopping. Watch the loss trend. If it decreases for a while then starts rising, stop immediately — you are beginning to overfit, or oscillations have started.

Learning rate schedules. For large problems like game-playing AI, the prof described a schedule. Start at $η = 0.1$ for the first 100 iterations. Then drop to $η = 0.05$ for the next 200. This big-start, small-finish pattern helps. The model makes fast early progress and settles carefully near the minimum. Schedules are covered properly under optimization topics later.

Pitfall — Real-world learning rate bounds. In practice, $η$ rarely exceeds 0.1 or 0.2. If you find yourself using $η = 1.0$ or higher, something is probably wrong with your data scaling or your loss formulation.

5.1.3 Weight Initialization

Why initialization matters. Starting all weights at zero puts every neuron in the same state. Every neuron computes the same gradient and updates identically. The network never breaks symmetry. It learns nothing useful.

Named initializations. Three mathematically derived strategies exist:

He initialization — scales weights by $2/ n_{in}$ . Designed for ReLU activations.
Xavier (Glorot) initialization — scales weights by $2/ (n_{in} + n_{out})$ . Designed for tanh/sigmoid.
LeCun initialization — scales weights by $1/ n_{in}$ . Effective for sigmoid activations.

Libraries like scikit-learn and PyTorch provide these as built-in functions. The intuition behind how they preserve variance across layers comes in a later lecture.

5.1.4 Feature Engineering

Feature engineering is the process of selecting, transforming. And scaling the raw inputs before they enter the model. Three actions:

Keep features that carry meaningful signal about the target.
Remove redundant features (highly correlated, constant, or duplicates).
Transform irrelevant or noisy features into useful ones — or drop them during preprocessing.

Scaling for convergence. Raw features often live on different scales — age in [0, 100] and salary in [10,000, 500,000]. Without scaling, the weight update for salary dominates. And the model zigzags instead of converging smoothly. Apply any of these:

Z-score normalization: $x^{'} = \frac{x - μ}{σ}$ (zero mean, unit variance)
Min-max scaling: $x^{'} = \frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}$ (range $[0, 1]$ )
Max scaling: $x^{'} = \frac{x}{x _{m a x}}$ (range $[0, 1]$ , preserves zero)

After scaling, every feature contributes roughly equally to each weight update — no single feature dominates the gradient.

5.1.5 Overfitting and Underfitting

Pitfall — Overfitting. Training error is tiny but test error is huge. The model has memorized the training data. Every noise, every quirk — instead of learning the underlying pattern. It fails on unseen examples.

Pitfall — Underfitting. Both training and test errors are high. The model is too simple to capture the pattern at all.

Fixes for overfitting:

Simplify the model. Reduce the number of neurons per layer. Fewer parameters mean less capacity to memorize.
Regularization. Add a penalty term to the loss that discourages large weights. This pushes the model toward simpler solutions. Covered in detail later.

5.1.6 Monitoring Oscillations

Pitfall — Oscillating loss. You see the loss bounce: iteration 5 gives 0.5, iteration 6 gives 1.5, iteration 7 gives 0.5 again. The model is zigzagging across the loss valley instead of descending. The learning rate is too large — it keeps overshooting the bottom. Reduce $η$ .

Pitfall — Flat loss. If the loss is not decreasing at all, check your update rule first. Are you subtracting the gradient or adding it? Then try a slightly higher learning rate. A dead-flat loss might also mean your weights are initialized at a saddle point — random initialization helps prevent this.

Q: When training gets stuck, should I change the learning rate or the model architecture first?

A: Always check the learning rate first. It is the fastest thing to change and the most common cause of stuck training. If the loss is flat, try a slightly higher rate. If it is oscillating, halve it. Only after exhausting learning rate adjustments should you look at changing the model — adding or removing perceptrons, or adjusting the feature engineering. The prof's debugging hierarchy is: learning rate → update rule correctness → weight initialization → feature scaling → model architecture.

Recap + Bridge: You now have the full debugging toolkit for a single-perceptron linear regressor. Four control knobs decide whether training succeeds: learning rate, stopping criterion, weight initialization. And feature scaling. Next, we ask: should we use all data at once, a handful, or just one point per step? Section 5.2 introduces three gradient descent variants.

Real-World Connection. Game-playing AI systems learn from millions of frames. They use learning rate schedules. Start at 0.1 for the first few hundred thousand frames. Then drop to 0.01, then 0.001. Companies like DeepMind use the same four control knobs at staggering scale.

5.2 Types of Gradient Descent

Hook: You have 60,000 images to classify. You could feed all 60,000 through the network, compute one giant gradient. And take one careful step. Or you could pick one image, update immediately. And repeat. The first feels safer. The second feels reckless. Which one actually produces a better model faster?

Intuition + Analogy — The marble and the funnel (Prof's analogy). The trajectory of a cost curve is like watching a marble roll down an uneven funnel. Batch gradient descent is the heaviest marble. It rolls straight to the bottom with no wobble. Mini-batch gradient descent is lighter. It wiggles a bit. Stochastic gradient descent bounces around, ricocheting off the walls. The surprise: wiggles are not always bad. A heavy marble can get stuck in a shallow dimple. A bouncy one might escape and find a deeper basin.

Where the analogy breaks: A real marble loses energy to friction and eventually stops. A gradient descent optimizer keeps going as long as the learning rate is nonzero — near the minimum, the noise in SGD means it never truly settles.

5.2.1 Three Variants

Three strategies for using training data during gradient descent.

Batch gradient descent. Use all $N$ training instances to compute one gradient. One weight update per epoch. The gradient is the exact average over the entire dataset:

\nabla_{θ} J (θ) = \frac{1}{N} i = 1 \sum N \nabla_{θ} L (y^^{(i)}, y^{(i)})

θ := θ - η \cdot \frac{1}{N} i = 1 \sum N \nabla_{θ} L (y^^{(i)}, y^{(i)})

Mini-batch gradient descent. Split the $N$ instances into batches of size $B$ . Compute the gradient over each batch separately. Multiple weight updates per epoch:

\nabla_{θ} J (θ) = \frac{1}{B} i = 1 \sum B \nabla_{θ} L (y^^{(i)}, y^{(i)})

Stochastic gradient descent (SGD). Use exactly one randomly chosen instance per update. The gradient is a noisy estimate of the true gradient:

θ := θ - η \cdot \nabla_{θ} L (y^^{(i)}, y^{(i)})

From the enrichment docs: the stochastic gradient $\nabla f_{i} (x)$ is an unbiased estimate of the full gradient $\nabla f (x)$ , because $E_{i} [\nabla f_{i} (x)] = \nabla f (x)$ . On average, it points in the right direction — but any individual step can be wildly off.

The choice among them depends on computational resources, speed requirements. And desired accuracy. In deep learning practice, you will almost always use mini-batch gradient descent (also called mini-batch stochastic gradient descent).

5.2.2 Contour Visualization

Visual Intuition. Imagine the cost function as a 3D bowl. Look at it from directly above. What you see are concentric elliptical rings — contour lines. Each ring connects points of equal cost. The innermost rings mark the lowest-cost region (the minimum). The outermost rings are high-cost regions far from the solution. This top-down view tracks how the cost moves across epochs.

What is an epoch? One complete pass through all $N$ training instances. For batch GD with $N = 1, 000$ , one epoch means: gather all 1,000 instances, forward-propagate, compute loss, compute gradient, do one update. For mini-batch with $B = 100$ , one epoch means 10 forward/backward passes and 10 updates.

5.2.3 Trajectory Smoothness

Batch gradient descent traces a smooth, gradual curve from the outermost contour ring toward the center. The trajectory is the smoothest of the three. Every step uses the exact, full-dataset gradient.

Mini-batch gradient descent shows slight wiggles between epochs. The cost deviates a little because each batch gives a slightly different gradient estimate. Not perfectly smooth. But the oscillations are small and the net movement is toward the minimum.

Stochastic gradient descent is noisy. The cost may rise on one update, fall on the next, rise again — all within the same contour region. The oscillations are large. In some cases, SGD may not converge at all. It can wander indefinitely around the minimum without settling.

Trade-off summary:

	Batch GD	Mini-Batch GD	SGD
Smoothness	Highest	Moderate	Lowest
Speed per epoch	Slowest	Medium	Fastest (most updates)
Noise in gradient	None	Moderate	High
Memory requirement	Entire dataset	Batch size $B$	1 instance
Convergence guarantee	Yes (convex)	Yes (convex, with schedule)	Yes (convex, with decaying $η$ )

5.2.4 Speed vs. Stability Ordering

Smoothness (most → least):

Batch gradient descent
Mini-batch gradient descent
Stochastic gradient descent

Speed — updates per unit time (fastest → slowest):

Stochastic gradient descent (one instance per update — no waiting)
Mini-batch gradient descent
Batch gradient descent (must process the entire dataset before one step)

In deep learning, mini-batch gradient descent is the default. It is a best-of-both compromise — not as slow as batch, not as noisy as stochastic. Compared to batch GD, its noisier gradient estimates can sometimes help it escape sharp, shallow local minima that a smooth trajectory would settle into.

5.2.5 Batch Size Computation Example

Worked Example — Batch count and weight updates.

Given: $N = 60, 000$ instances, batch size $B = 128$ .

Batches per iteration: $60, 000/128 = 468.75$ . In practice, this means 468 full batches of 128 + 1 partial batch of 96 instances → 469 batches processed per epoch.

Weight updates per epoch: 469 (one per batch).

Weight updates for 10 epochs: $469 \times 10 = 4, 690$ .

Sense check: For batch GD, 10 epochs would give only 10 updates. For SGD, 10 epochs would give 600,000 updates. Mini-batch sits in between — 4,690 updates in 10 epochs. This is the practical meaning of "best of both worlds."

5.2.6 Decision Framework for Choosing a Variant

Scope — When each variant makes sense.

Factor	Pick Batch GD	Pick Mini-Batch GD	Pick SGD
Dataset size	Small (< 10k)	Medium to large	Huge (millions+)
Memory	Plentiful	Moderate	Very low
Accuracy priority	Highest	High	Lower (noisy)
Speed priority	Low	Medium	High
Dimensionality	Low to medium	Medium to high	Any
GPU available	Yes (load full data)	Yes (batch on GPU)	Not needed
Escaping local minima	Harder (smooth)	Possible	Easier (noisy)

Decision rule: If accuracy is the overriding concern, pick batch GD. If speed matters and you have a GPU, pick mini-batch. If memory is severely limited or the dataset is enormous, pick SGD.

5.2.7 Student Questions and Answers

Q: What kind of exam questions can we expect for this part?

A: The prof outlined four question types:

Numerical: Similar to the morning session — compute forward passes, gradients, and weight updates for a given instance or batch. Also batch-size calculations ( $N / B$ ).
Analytical (scenario-based): Given a use case ("limited memory, need high accuracy, real-time"), pick the right gradient descent variant. Justify based on accuracy, speed, stability, compute capability, number of instances. And data dimensionality.
Plot interpretation: You may get cost-curve plots of different GD trajectories. Identify which curve matches which method. Spot whether the learning rate is too high (extreme oscillations).
Code interpretation: Given a code snippet, identify the hyperparameters, the convergence criterion. And the gradient descent variant. No code writing required.

5.2.8 The Marble-and-Funnel Analogy

The trajectory of a cost curve across epochs is like watching a marble roll down an uneven funnel. Batch GD is the heaviest marble. It rolls straight to the bottom. Mini-batch is lighter. It wiggles a bit. SGD is practically bouncing around. The wiggles are not always bad. They can help you escape sharp, shallow local minima that a heavy marble would miss.

Recap + Bridge: The three gradient descent variants differ only in how many data points go into one gradient. Batch GD = all $N$ , mini-batch = $B$ , SGD = 1. Mini-batch strikes the sweet spot — fast, stable, and GPU-friendly. Section 5.3 formalizes the mini-batch math.

Real-World Connection. Almost every deep learning framework defaults to mini-batch SGD. When you call optimizer.step() on a batch of 32 or 128, that is it in action. Training BERT on 16 GB of GPU memory forces a batch size of 8. Training a small CNN with an A100 lets you use 256. The choice is a hardware constraint, not just a math decision.

5.3 Mini-Batch Gradient Descent Formulation

Hook: You have 1,000 training examples and a capable GPU. Batch GD gives you one weight update per epoch. SGD gives you 1,000 updates but each one is based on just one data point. Is there an option that gives you dozens of reasonably reliable updates per epoch — without waiting for all the data? That is mini-batch gradient descent.

5.3.1 How It Works

Purpose. Mini-batch gradient descent solves the speed-accuracy tradeoff between batch GD and SGD. It splits the dataset into small chunks (batches), computes the gradient on each chunk independently, and updates weights after each chunk. This gives many updates per epoch — faster than batch GD — while each update is averaged over $B$ examples, making it less noisy than SGD.

5.3.2 Mathematical Formulation

Inputs & Outputs.

Inputs:

Training data: $N$ instances, each with feature vector $x^{(i)} \in R^{d}$ and target $y^{(i)}$ .
Initial weights: $θ$ (all trainable parameters — $w$ and $b$ for a single perceptron).
Hyperparameters: learning rate $η$ , batch size $B$ , number of epochs.
Loss function: $L (y^, y)$ — e.g., MSE for regression, cross-entropy for classification.

Outputs:

Updated weights $θ$ that minimize the average loss across the training data.

Steps — The mini-batch training loop.

Repeat for a fixed number of epochs (or until a convergence criterion fires).
Shuffle the dataset. Randomly reorder the $N$ instances before each epoch. This prevents the model from learning spurious patterns from the order of the data.
Split into batches. Divide the shuffled data into batches of size $B$ . For $N = 1, 000$ and $B = 100$ , you get 10 batches per epoch.
For each batch: - Forward-propagate the $B$ instances through the network to get predictions $y^^{(i)}$ . - Compute the loss averaged over the batch: $J_{batch} (θ) = \frac{1}{B} i = 1 \sum B L (y^^{(i)}, y^{(i)})$ - Compute the gradient of the batch loss with respect to each parameter: $\nabla_{θ} J_{batch} = \frac{1}{B} i = 1 \sum B \nabla_{θ} L (y^^{(i)}, y^{(i)})$ - Update weights: $θ_{j} := θ_{j} - η \cdot \frac{1}{B} i = 1 \sum B \frac{\partial L ^{(i)}}{\partial θ _{j}}$
End of epoch. One complete pass through all $N$ instances = one epoch. Return to step 1.

The critical difference from batch GD: the loss and gradient are averaged by $B$ (batch size), not by $N$ (total instances). From the enrichment docs, the mini-batch gradient $\frac{1}{B} \sum_{i \in batch} \nabla f_{i} (x)$ is a more stable estimator than the single-sample SGD gradient — its variance scales as $1/ B$ . Larger $B$ = less noise, slower updates. Smaller $B$ = more noise, faster updates.

Trace — One epoch, three batches. Suppose $N = 6$ instances, $B = 2$ (3 batches), $η = 0.01$ . Initial weights: $θ = [0, 0]$ .

Batch 1 (instances 1, 2): Forward pass → loss $= 0.8$ . Gradient $= \frac{1}{2} (\nabla L_{1} + \nabla L_{2}) = [0.3, - 0.1]$ . Update: $θ = [0, 0] - 0.01 \cdot [0.3, - 0.1] = [- 0.003, 0.001]$ .

Batch 2 (instances 3, 4): Forward pass with $θ = [- 0.003, 0.001]$ → loss $= 0.5$ . Gradient $= [0.2, 0.15]$ . Update: $θ = [- 0.003, 0.001] - 0.01 \cdot [0.2, 0.15] = [- 0.005, - 0.0005]$ .

Batch 3 (instances 5, 6): Forward pass → loss $= 0.3$ . Gradient $= [- 0.1, 0.05]$ . Update: $θ = [- 0.005, - 0.0005] - 0.01 \cdot [- 0.1, 0.05] = [- 0.004, - 0.001]$ .

End of epoch. Loss trend: $0.8 \to 0.5 \to 0.3$ — decreasing. Three updates in one epoch. Batch GD would have made one update; SGD would have made six. Mini-batch gives three updates, each informed by two examples.

5.3.3 Choosing Batch Size

The batch size directly controls the speed-vs-stability tradeoff:

Batch Size	Updates per Epoch	Stability	Speed per Iteration	Memory
Small (32–64)	Many ( $\approx N / B$ )	Less stable — weight can jump from $+ 0.5$ to $- 0.5$ to $+ 5$	Faster	Low
Medium (~128)	Moderate	Moderate	Balanced	Moderate
Large (256+)	Fewer	More stable — smoother trajectory	Slower per iteration (more data per step)	High

Pitfall — Memory constraint. If your batch size does not fit in GPU memory, the training crashes. Reduce $B$ . If memory is so limited that even $B = 16$ is tight, fall back to SGD.

Pitfall — Under-utilized GPU. If your GPU is sitting mostly idle with a small $B$ , increase the batch size. A modern GPU can process hundreds of examples in parallel. A batch size of 2 wastes its parallelism.

Pitfall — Batch size and learning rate. Larger batch sizes produce more stable (lower-variance) gradients. You can often use a larger learning rate with a larger batch. A common heuristic: double the batch size → try multiplying the learning rate by roughly $2$ . This is sometimes called the "linear scaling rule" in practice. However, the relationship is not exact — always validate.

Complexity & Cost. Per batch: forward pass is $O (B \cdot d)$ for $d$ features. Backward pass is also $O (B \cdot d)$ . Per epoch: $O (N \cdot d)$ — same asymptotic cost as batch GD. The advantage is not in total computation but in wall-clock time: you get many updates before seeing the whole dataset once, so the loss curve drops sooner.

When to Use / Alternatives.

Use mini-batch GD when you have a medium-to-large dataset and a GPU. This is the default in deep learning.
Use batch GD when the dataset fits entirely in memory and you want the most stable convergence. Rare in deep learning except for small-scale experiments.
Use SGD when memory is extremely tight or when you need the absolute fastest per-iteration time. Rare in modern practice because the noise hurts convergence speed.

5.3.4 Student Questions and Answers

Q: In the scenario-based questions, do we just pick based on speed vs. accuracy?

A: Read the scenario carefully. Check whether accuracy is the top priority, or whether speed and computational constraints matter more. Then justify your choice of batch, mini-batch, or stochastic gradient descent. The justification — showing you understand the tradeoffs — is what earns marks.

Recap + Bridge: Mini-batch GD is the workhorse of deep learning. Shuffle the data. Slice it into chunks of size $B$ . Update after each chunk. The batch size $B$ trades speed for gradient quality. Next question: what kind of problem are you solving? Section 5.4 introduces classification — where the target is a category, not a number.

Real-World Connection. Every call to DataLoader(batch_size=128) in PyTorch sets up mini-batch gradient descent. When you train ResNet on ImageNet, a typical batch size is 256 across 8 GPUs. Each GPU processes 32 examples. Gradients are averaged across GPUs. Weights get one update. The entire pipeline (1.2 million images, 90 epochs) boils down to the loop you just studied, scaled across hundreds of machines.

5.4 Classification: Concepts and Types

Hook: Regression predicts a number — house price, temperature, speed. But what if the answer is not a number? What if it is "spam" or "not spam"? "Cat" or "dog" or "car"? The moment your target is a label rather than a quantity, you have crossed from regression into classification.

Intuition + Analogy — Sorting mail. Imagine a postal worker standing in front of a conveyor belt of letters. Each letter lands in one of three bins: local, national, or international. The worker looks at the address (the features) and decides (the prediction). If there are only two bins, that is binary classification — local or not-local. Three or more bins is multi-class classification. A letter with multiple stickers — one for "fragile" and one for "priority" — is multi-label classification: two independent yes/no questions about the same item.

Where the analogy breaks: The postal worker knows the rules (postal codes map to bins). A classifier must learn those rules from labeled examples. It does not start with a lookup table.

5.4.1 What Is Classification

Classification is a supervised learning task where the target variable $Y$ is categorical — it takes one of a fixed set of discrete values, not a continuous number.

Supervised: The training data includes both features $x$ and the correct label $y$ .
Categorical target: $y \in {cat, dog, car}$ or encoded as $y \in {0, 1}$ for binary, $y \in {1, 2, \dots, K}$ for multi-class.

The model learns a mapping from feature space to class labels. At test time, it sees new features and predicts a label. Real-world examples:

Spam detection: Features = keyword counts ("money," "discount," "payment"), sender reputation, attachment metadata. Label = spam / not-spam.
Medical diagnosis: Features = symptoms, lab test results, medical history. Label = disease-present / disease-absent.
Image classification: Features = raw pixel values. Label = cat, dog, or car.
Sentiment analysis: Features = words and phrases in a product review. Label = positive, negative, or neutral.

5.4.2 Binary Classification

Binary classification has exactly two possible labels, conventionally encoded as $0$ and $1$ . The model outputs a single number (a probability or a score), and you threshold it to get the final class.

Examples: spam ( $y = 1$ ) vs. not-spam ( $y = 0$ ), disease vs. healthy, pass vs. fail, cat vs. not-cat.

The model architecture is a single output neuron with a sigmoid activation (Section 5.5). The loss function is binary cross-entropy (Section 5.6).

5.4.3 Multi-Class Classification

Multi-class classification has $K > 2$ mutually exclusive labels. Each input belongs to exactly one class. The model outputs $K$ numbers (one per class), and the highest-scoring class is the prediction.

Examples: classifying images into cat / dog / car ( $K = 3$ ), sentiment into positive / negative / neutral ( $K = 3$ ), digit recognition 0–9 ( $K = 10$ ).

For image classification, the simplest feature extraction uses raw pixels: an $N \times N$ image gives $N^{2}$ pixel values as features. But raw pixels are a poor representation — a cat shifted by one pixel looks completely different in feature space. Convolutional neural networks (CNNs) solve this by learning translation-invariant feature extractors automatically. CNNs are covered in a later lecture.

The model architecture uses $K$ output neurons with a softmax activation (Section 5.9). The loss function is categorical cross-entropy.

5.4.4 Multi-Label Classification

Multi-label classification allows a single input to belong to multiple classes simultaneously. This is not one prediction with $K$ choices — it is $K$ independent binary decisions.

Example: an image may need two classification tasks at once. One task: cat, dog, or mouse. Another task (using the same image): real photo or AI-generated. The same input can be both "cat" AND "AI-generated." In practice, you build two separate binary classifiers (or one multi-output model with sigmoid activations), each making its own yes/no decision.

Prof's guidance: if one label can be derived from another with an if-else, do not build a separate model. For example, if you classify images as cat/dog/fox, you can infer "wild" vs. "domestic" from the class name. No extra model needed. Build a multi-label path only when the extra label is truly independent.

Visual Intuition. Picture the feature space as a 2D plane with dots colored by class. For binary classification, a single line (decision boundary) cuts the plane into two regions — blue dots on one side, red on the other. For multi-class ( $K = 3$ ), you need three lines, creating three wedge-shaped regions. Each region is where one class gets the highest score. For multi-label, imagine two independent sets of overlapping regions painted on the same plane — one set for the "animal type" decision and another for the "real vs. fake" decision.

Pitfall — Confusing multi-class with multi-label. Multi-class: one input → one label from $K$ options. Multi-label: one input → potentially multiple labels. If you use a multi-class setup for a multi-label problem, the model can only pick one class. It cannot output "both."

Pitfall — Assuming raw pixels are good features. For an $N \times N$ image, the feature vector has $N^{2}$ dimensions. That is huge and unstructured. CNNs (covered later) exist precisely because raw pixels make terrible direct inputs for a regular perceptron.

Pitfall — Building unnecessary multi-label models. The prof's advice: if you can derive one label from another with a simple if-else, do not waste resources. For example, "fox → wild" needs no extra classifier. Build a multi-label path only when the extra label is truly independent.

5.4.5 Student Questions and Answers

Q: Can we consider "wild vs. domestic animals" as a multi-label classification alongside "cat vs. dog"?

A: Yes. But think it through first. If you already have fine-grained classes (cat, dog, fox, etc.), can you infer "wild" vs. "domestic" from the predicted class name? If yes — just write an if-else mapping from class name to wild/domestic. Do not build a separate costly model. Build a multi-label path only when the category cannot be derived from existing outputs.

Recap + Bridge: Classification is regression's categorical sibling. Binary (two classes), multi-class ( $K$ exclusive classes), and multi-label (independent yes/no decisions) cover every labeling scenario. The next question: how does a perceptron — which outputs any real number — produce a valid class prediction? Section 5.5 answers that with the sigmoid function.

Real-World Connection. Classification drives most deployed ML systems today. Spam filters process billions of emails daily. Each one is a binary classification. Google Photos classifies every uploaded image into thousands of object categories (multi-class) and simultaneously tags images as "live photo" or "screenshot" (multi-label). Medical imaging startups use multi-class classification to detect diabetic retinopathy from retina scans — grading severity on a 5-point scale. All of these systems start with the same fundamental question: is the target a number or a category?

5.5 Sigmoid Activation and Logistic Regression

Hook: A perceptron spits out numbers like $- 3.7$ or $42.1$ — any real value. But your task is to answer a yes/no question: is this email spam? How do you squash an unbounded number into a clean "0.72 probability of spam"?

Intuition + Analogy — The bouncer at the club. Imagine a nightclub bouncer. He evaluates multiple signals: your outfit, ID, demeanor. Then he decides: let you in (1) or turn you away (0). The sigmoid is the bouncer. It takes the weighted sum of all signals and converts it into a probability. If the output crosses 0.5, the door opens. Raise the threshold to 0.8 for a more exclusive club. The model must be more confident before saying "yes."

Where the analogy breaks: A bouncer's rules are fixed (dress code, guest list). A sigmoid learns the weights from data. It figures out which signals matter and how much.

5.5.1 Why Not Linear Regression for Classification

A simple perceptron computes a weighted sum:

z = w^{⊤} x + b

With a linear (identity) activation, the output $z$ can be any real number from $- \infty$ to $+ \infty$ . Classification needs output in ${0, 1}$ .

The naive approach — thresholding at 0.5. "If $z \geq 0.5$ , predict class 1; otherwise, class 0." This breaks for two reasons. Linear regression tries to fit every data point equally. It does not discriminate between classes. A few extreme data points can shift the entire line. The decision boundary gets pulled away from where it should be. What you need is a boundary that separates two regions regardless of how points scatter inside each class. Linear regression minimizes squared distance to all points; classification needs to minimize misclassification errors. These are fundamentally different goals.

Visual Intuition. The prof showed slides contrasting two approaches. A linear regression line (blue/purple) was asked to separate two colored classes. It failed: the line cut through both clusters, trying to be close to every point. The sigmoid-based model (green S-curve) created a clean decision boundary that separated the classes properly. The S-curve is designed for discrimination, not fitting.

Worked Example — Linear regression fails at classification. Suppose you have two data points: $(x = 1, y = 0)$ and $(x = 2, y = 1)$ . Linear regression fits a line: $y = x - 1$ . At $x = 1.5$ , prediction = 0.5 — right on the threshold. Now add one outlier: $(x = 100, y = 1)$ . The line shifts to $y \approx 1$ for all reasonable $x$ — it now predicts class 1 for almost everything. Classification fails catastrophically because the regression line cannot ignore outliers the way a decision-boundary model can.

Sense check: The sigmoid-based model would still separate the two classes properly because the S-curve saturates at 0 and 1 — extreme points do not yank the entire curve.

5.5.2 The Sigmoid Function

The sigmoid function maps any real-valued input $z$ into the interval $(0, 1)$ :

σ (z) = \frac{1}{1 + e ^{- z}}

When $z$ is very large ( $z \to + \infty$ ), $e^{- z} \to 0$ , so $σ (z) \to 1$ .
When $z$ is very negative ( $z \to - \infty$ ), $e^{- z} \to \infty$ , so $σ (z) \to 0$ .
When $z = 0$ , $σ (0) = \frac{1}{1 + 1} = 0.5$ .

The whole model is called logistic regression. The name is a historical quirk — it is a classification technique. "Regression" refers to the fact that it regresses (fits) a linear model $z = w^{⊤} x + b$ — you get the classification label by applying the sigmoid on top.

Notation note: Some texts write the sigmoid as $logit^{- 1} (z)$ or $\frac{1}{1 + e x p ( - z )}$ . The prof uses $σ (z)$ . All are equivalent.

5.5.3 Symbol Registry — Sigmoid

Symbol	Meaning	LaTeX	Type / Domain
$x$	input feature vector	$x$	$R^{d}$
$w$	weight vector	$w$	$R^{d}$
$b$	bias term	$b$	scalar
$z$	pre-activation (weighted sum)	$z = w^{⊤} x + b$	$(- \infty, + \infty)$
$σ (z)$	sigmoid output	$σ (z) = \frac{1}{1 + e ^{- z}}$	$(0, 1)$
$y^$	predicted probability	$y^= σ (z)$	$(0, 1)$
$y$	true class label	$y \in {0, 1}$	scalar

5.5.4 Properties of the Sigmoid

Output range: $(0, 1)$ . The sigmoid never produces exactly 0 or 1 — it only approaches them asymptotically.
Monotonic: As $z$ increases, $σ (z)$ always increases. There is no wiggle.
Differentiable everywhere: The curve is smooth with no kinks. This is critical because backpropagation needs gradients.
Symmetric about $(0, 0.5)$ : If you fold the graph horizontally around the point $(0, 0.5)$ , the two halves match exactly. Equivalently, $σ (- z) = 1 - σ (z)$ .
Default threshold: $σ (z) \geq 0.5 ⟹$ predict class 1. At $z = 0$ , the model is maximally uncertain. You can change the threshold — raise it to 0.8 for higher confidence.
Probability interpretation: $σ (z)$ can be read as $P (y = 1 ∣ x)$ , the estimated probability of class 1 given the input.

Worked Example — Sigmoid at key points.

$σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{2} = 0.5$ (maximum uncertainty)
$σ (2) = \frac{1}{1 + e ^{- 2}} = \frac{1}{1 + 0.135} \approx 0.88$ (high confidence in class 1)
$σ (- 2) = \frac{1}{1 + e ^{2}} = \frac{1}{1 + 7.389} \approx 0.12$ (high confidence in class 0)
$σ (5) = \frac{1}{1 + e ^{- 5}} = \frac{1}{1 + 0.0067} \approx 0.993$ (nearly certain class 1)

Sense check: The values are all between 0 and 1, symmetric around 0.5. And increase monotonically.

5.5.5 Derivative of the Sigmoid

The derivative of the sigmoid has a remarkably clean form. It can be expressed entirely in terms of the sigmoid output itself:

\frac{d σ}{d z} = σ (z) (1 - σ (z))

Full derivation:

Start with $σ (z) = (1 + e^{- z})^{- 1}$ . Apply the chain rule:

\frac{d σ}{d z} = \frac{d}{d z} (1 + e^{- z})^{- 1} = - 1 \cdot (1 + e^{- z})^{- 2} \cdot \frac{d}{d z} (1 + e^{- z}) = - (1 + e^{- z})^{- 2} \cdot (- e^{- z}) = \frac{e ^{- z}}{( 1 + e ^{- z} ) ^{2}} = \frac{1}{1 + e ^{- z}} \cdot \frac{e ^{- z}}{1 + e ^{- z}} = σ (z) \cdot \frac{e ^{- z}}{1 + e ^{- z}} = σ (z) \cdot (1 - \frac{1}{1 + e ^{- z}}) = σ (z) \cdot (1 - σ (z)) (power rule + chain rule) (derivative of e^{- z} is - e^{- z}) (negatives cancel) (split the denominator) (first factor is σ (z)) (add and subtract 1 in numerator) (recognize the sigmoid again)

This compact form is why sigmoid is computationally elegant: once you have computed $σ (z)$ during the forward pass, the derivative costs just one extra multiplication. No need to recompute $e^{- z}$ .

Special-case check: At $z = 0$ , $σ (0) = 0.5$ , so $σ^{'} (0) = 0.5 \cdot 0.5 = 0.25$ . The slope is steepest at the center. At $z = \pm 5$ , $σ^{'} (5) = 0.993 \cdot 0.007 \approx 0.007$ — the slope is nearly flat at the extremes. This makes intuitive sense: near the decision boundary, small changes in $z$ matter a lot. Far from it, the output is already saturated and hardly budges.

The derivative curve is bell-shaped, symmetric, and centered at $z = 0$ . You will use $σ^{'} (z) = σ (z) (1 - σ (z))$ directly in numerical problems and backpropagation computations.

5.5.6 The Logit

The sigmoid maps the linear predictor $z$ to a probability $y^$ . The inverse mapping — going from probability back to the linear score — is called the logit (or log-odds):

lo g (\frac{y ^}{1 - y ^}) = z = w^{⊤} x + b

Full derivation of the inverse:

Start with $y^= \frac{1}{1 + e ^{- z}}$ . Solve for $z$ :

y^y^(1 + e^{- z}) y^+ y^e^{- z} y^e^{- z} e^{- z} e^{z} z = \frac{1}{1 + e ^{- z}} = 1 = 1 = 1 - y^= \frac{1 - y ^}{y ^} = \frac{y ^}{1 - y ^} = lo g (\frac{y ^}{1 - y ^}) (multiply both sides by denominator) (divide by y^) (take reciprocal of both sides) (take natural log)

Substituting $z = w^{⊤} x + b$ gives the logit form:

lo g (\frac{y ^}{1 - y ^}) = w^{⊤} x + b

The quantity $\frac{y ^}{1 - y ^}$ is the odds. It is the ratio of class-1 probability to class-0 probability. If the odds are 3, class 1 is three times more likely than class 0. Taking the log makes the relationship linear in $x$ . The prof explained: "Logit is a proportion of probability of data belonging to class 1 against probability of data belonging to class 0."

Dimensional check: $y^\in (0, 1)$ , so the odds $\frac{y ^}{1 - y ^} \in (0, \infty)$ . The log-odds (logit) can be any real number — matching the unrestricted range of $w^{⊤} x + b$ . The inverse relationship is consistent.

5.5.7 Student Questions and Answers

Q: Instead of sigmoid, why not use min-max normalization to scale the output to $[0, 1]$ ?

A: Min-max normalization needs the minimum and maximum of the entire domain. For the perceptron output $z$ , the domain is $(- \infty, + \infty)$ — unbounded. You cannot plug $\pm \infty$ into a min-max formula. The sigmoid compresses an unbounded range into $(0, 1)$ naturally, without needing to know min or max values.

Q: How do we know the sigmoid is symmetric?

A: Look at the graph. Draw a horizontal line through 0.5. The curve above and below are mirror images. Mathematically, $σ (- z) = 1 - σ (z)$ . Also, the derivative $σ^{'} (z)$ is symmetric about $z = 0$ — the left and right sides match if folded along the vertical axis.

Q: Do we still need a sigmoid for a logic gate problem where the target is already exactly 0 or 1?

A: The perceptron's raw output $z$ is continuous — it can be 0.2, 10.2, -3.7, anything. The sigmoid converts that continuous value into a probability-like number. After the sigmoid, you apply a threshold (default: 0.5) to get the final binary decision. Libraries like scikit-learn's LogisticRegression do this automatically.

Q: With multiple features, do we apply the sigmoid to each feature individually?

A: No. You compute the weighted sum $z = w^{⊤} x + b$ — which already combines all features into a single number. You apply the sigmoid once to that combined result. You do not transform each feature individually.

Q: For classification, how is the error calculated differently from regression?

A: The loss function is binary cross-entropy, not MSE. Its mathematical form is completely different. However, after differentiation, the gradient has the same structure as linear regression: (predicted $-$ actual) $\times$ feature. You will see this in detail in Section 5.6.

Recap + Bridge: The sigmoid squashes any real number into a probability. This turns a linear perceptron into a binary classifier. Its derivative is self-contained: $σ^{'} (z) = σ (z) (1 - σ (z))$ . Backpropagation is cheap. The logit shows the inverse. Section 5.6 answers: once you have these probabilities, how do you measure how wrong the model is?

Real-World Connection. Logistic regression is one of the most deployed ML models in the world. Credit card fraud detectors use it to score transactions in real time. If the probability exceeds 0.9, they block the transaction. Online ad platforms estimate click-through probability with it. In medicine, it predicts hospital readmission risk from patient vitals. The weight $w_{j}$ tells you exactly how the log-odds change per unit of feature $j$ . No black box — just weighted evidence.

5.6 Binary Cross-Entropy Loss

Hook: A model that predicts 0.99 when the answer is 1 sounds nearly perfect. But a model that predicts 0.01 when the answer is 1 sounds catastrophically wrong. Should these two mistakes get the same penalty? The answer is no. And that is exactly what cross-entropy enforces.

Intuition + Analogy — The lie detector. A confident wrong prediction is like someone looking you in the eye and lying with absolute certainty. A borderline wrong prediction is like someone shrugging and guessing wrong. The first is much worse. The $- lo g$ function behaves like this: when the model is certain and right ( $y^\approx 1, y = 1$ ), cost ≈ 0. When it is certain and wrong ( $y^\approx 0, y = 1$ ), cost → ∞. When it is uncertain ( $y^\approx 0.5$ ), cost is moderate regardless of the answer. The log curve is nature's way of punishing overconfidence in the wrong direction.

Where the analogy breaks: A lie detector gives a binary output (lie/truth). Cross-entropy gives a continuous cost. The magnitude of the penalty matters for gradient-based learning.

5.6.1 Designing a Classification Cost Function

What a good classification loss must deliver.

In regression, error is simple: subtract predicted from actual and square it. In classification, the error is binary — right or wrong per instance. A good loss function for classification must satisfy two requirements:

Convexity. The loss surface must have one global minimum (no local minima to trap gradient descent). This guarantees convergence to the best possible solution under gradient-based optimization.
Confidence sensitivity. A confident wrong prediction must be penalized far more heavily than a borderline wrong one. If the true label is 1 and the model outputs 0.99 (wrong with near-certainty), the penalty should be massive. If the model outputs 0.51 (wrong but hesitant), the penalty should be mild.

MSE fails both requirements for classification. It is not the most natural convex form for binary targets. And it penalizes all errors roughly proportionally to squared distance. It does not exponentially punish confidence mistakes.

5.6.2 The Two Components

Binary classification has two ways to be wrong:

True label $y = 1$ , but predicted $y^\approx 0$ (missed a positive).
True label $y = 0$ , but predicted $y^\approx 1$ (false alarm).

These are fundamentally different errors. So the loss splits into two components. Only one is active per instance. The one matching the true label.

Component 1 — when the true label is 1 (the "cat" case):

L (y^, y = 1) = - lo g (y^)

Behavior:

$y^\to 1$ : $- lo g (1) = 0$ — perfect prediction, zero cost.
$y^= 0.5$ : $- lo g (0.5) \approx 0.693$ — uncertain, moderate cost.
$y^\to 0$ : $- lo g (0) \to + \infty$ — confident wrong, infinite cost.

Component 2 — when the true label is 0 (the "dog" case):

L (y^, y = 0) = - lo g (1 - y^)

Behavior:

$y^\to 0$ : $- lo g (1) = 0$ — perfect, zero cost.
$y^= 0.5$ : $- lo g (0.5) \approx 0.693$ — uncertain, moderate cost.
$y^\to 1$ : $- lo g (0) \to + \infty$ — confident wrong, infinite cost.

Domain check: $y^\in (0, 1)$ because the sigmoid never outputs exactly 0 or 1. So $lo g (y^)$ and $lo g (1 - y^)$ are always defined (never $lo g (0)$ in exact arithmetic). In practice, numerical stability requires clipping $y^$ to a tiny range like $[1 0^{- 15}, 1 - 1 0^{- 15}]$ .

5.6.3 Combined Form — Binary Cross-Entropy

Both components combine into one expression using $y$ as an on/off switch:

L (y^, y) = - [y \cdot lo g (y^) + (1 - y) \cdot lo g (1 - y^)]

If $y = 1$ : the $(1 - y) lo g (1 - y^)$ term is multiplied by 0 and vanishes. Only $- lo g (y^)$ remains.
If $y = 0$ : the $y lo g (y^)$ term vanishes. Only $- lo g (1 - y^)$ remains.

The cost averaged over $N$ training instances is:

J (θ) = - \frac{1}{N} i = 1 \sum N [y^{(i)} lo g (y^^{(i)}) + (1 - y^{(i)}) lo g (1 - y^^{(i)})]

Why "cross-entropy"? Entropy $H (p) = - \sum p lo g p$ measures the uncertainty in a distribution. Cross-entropy $H (p, q) = - \sum p lo g q$ measures how well distribution $q$ approximates the true distribution $p$ . Here, $p$ is the true label distribution (deterministic — it is $[1, 0]$ or $[0, 1]$ ) and $q$ is the model's predicted distribution $[y^, 1 - y^]$ . Minimizing cross-entropy makes the model's predictions match the true labels.

From the enrichment docs: the cross-entropy between $P$ and $Q$ is $H (P, Q) = - \sum_{j} P (j) lo g Q (j)$ . The minimum is achieved when $P = Q$ . This is the information-theoretic foundation of the binary cross-entropy loss.

Worked Example — Loss values for three scenarios.

Case A: True label $y = 1$ , model predicts $y^= 0.99$ (confident and right).

L = - [1 \cdot lo g (0.99) + 0 \cdot lo g (0.01)] = - lo g (0.99) = 0.010

Near zero cost. The model is confident and correct.

Case B: True label $y = 1$ , model predicts $y^= 0.51$ (barely right, unconfident).

L = - lo g (0.51) = 0.673

Moderate cost — correct but hesitant.

Case C: True label $y = 1$ , model predicts $y^= 0.01$ (confident and wrong).

L = - lo g (0.01) = 4.605

Very high cost. The model is confident in the wrong answer.

Sense check: The penalty ratio between "confident wrong" and "confident right" is $4.605/0.010 \approx 461$ . The model is punished about 460 times more for being confident and wrong than for being confident and right. The log function matches the design requirement perfectly.

5.6.4 The Gradient of Cross-Entropy with Sigmoid

When cross-entropy is paired with the sigmoid activation, the gradient simplifies dramatically:

\frac{\partial L}{\partial w _{j}} = \frac{1}{N} i = 1 \sum N (y^^{(i)} - y^{(i)}) \cdot x_{j}^{(i)}

Full derivation (showing why the complex-looking chain rule collapses):

The loss for a single instance is $L = - [y lo g (y^) + (1 - y) lo g (1 - y^)]$ , where $y^= σ (z)$ and $z = w^{⊤} x + b$ .

Step 1 — derivative of $L$ with respect to $y^$ :

\frac{\partial L}{\partial y ^} = - [y \cdot \frac{1}{y ^} + (1 - y) \cdot \frac{- 1}{1 - y ^}] = - \frac{y}{y ^} + \frac{1 - y}{1 - y ^} = \frac{- y ( 1 - y ^ ) + ( 1 - y ) y ^}{y ^ ( 1 - y ^ )} = \frac{- y + y y ^ + y ^ - y y ^}{y ^ ( 1 - y ^ )} = \frac{y ^ - y}{y ^ ( 1 - y ^ )} (common denominator)

Step 2 — derivative of $y^$ with respect to $z$ (sigmoid derivative):

\frac{\partial y ^}{\partial z} = σ^{'} (z) = y^(1 - y^)

Step 3 — chain rule:

\frac{\partial L}{\partial z} = \frac{\partial L}{\partial y ^} \cdot \frac{\partial y ^}{\partial z} = \frac{y ^ - y}{y ^ ( 1 - y ^ )} \cdot y^(1 - y^) = y^- y

The $y^(1 - y^)$ terms cancel perfectly. This is why sigmoid + cross-entropy is such a natural pair. In the chain rule, the sigmoid derivative $y^(1 - y^)$ appears in the denominator of the cross-entropy derivative. It cancels with the sigmoid derivative from the chain rule. The result is the simple form $y^- y$ .

Step 4 — derivative with respect to weights (for $z = w_{0} x_{0} + w_{1} x_{1} + \dots$ ):

\frac{\partial L}{\partial w _{j}} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial w _{j}} = (y^- y) \cdot x_{j}

Averaging over $N$ instances:

\frac{\partial L}{\partial w _{j}} = \frac{1}{N} i = 1 \sum N (y^^{(i)} - y^{(i)}) \cdot x_{j}^{(i)}

The key insight: This is structurally identical to the MSE gradient for linear regression: $(y^- y) \cdot x_{j}$ . The only difference is what $y^$ means — for regression, $y^= z$ ; for classification, $y^= σ (z)$ . The backward-pass logic is identical. You can swap between regression and binary classification by changing only the activation function and the loss, while the gradient update rule stays the same.

Dimensional check: $y^\in (0, 1)$ , $y \in {0, 1}$ , so $(y^- y) \in (- 1, 1)$ . Multiplied by feature $x_{j}$ and averaged over $N$ , the gradient has the same dimension as $w_{j}$ . Consistent.

5.6.5 Student Questions and Answers

Q: If we solve the two cost curve equations, would their intersection point be the minimum?

A: The two components never fire together for the same instance. When $y = 1$ , only $- lo g (y^)$ is active. When $y = 0$ , only $- lo g (1 - y^)$ is active. You are not solving two equations at once — for a given data point, exactly one log term contributes to the error. The minimum of the active component occurs at $y^= y$ .

Q: Why do we take the log in the cross-entropy form?

A: The log captures a specific intuition. A confident wrong prediction should be punished far more than a borderline mistake. As $y^\to 0$ when $y = 1$ , the $- lo g$ curve shoots to infinity. It punishes overconfidence exponentially. A linear penalty cannot distinguish "wrong by a little" from "wrong with certainty." The log also connects to information theory. Maximum likelihood estimation gives it deep theoretical backing. The full statistical derivation comes in the ML course.

Recap + Bridge: Binary cross-entropy punishes two kinds of mistakes — missing a positive and raising a false alarm — with a logarithmic penalty that explodes for confident wrong answers. Paired with sigmoid, the math collapses to the same gradient form as MSE in linear regression: (predicted $-$ actual) $\times$ feature. Next, Section 5.7 walks through a complete numerical example — training a binary classifier with SGD on real numbers.

Real-World Connection. Binary cross-entropy (or "log loss") is the standard for binary classifiers in production. Facebook's click-through models, Google's spam filters. And bank fraud detectors all minimize it. The math — sigmoid plus cross-entropy collapsing to a clean gradient — lets these systems train efficiently at massive scale. When you see loss='binary_crossentropy' in Keras or nn.BCELoss() in PyTorch, this is exactly the loss you have derived.

5.7 Binary Classification: Worked Numerical Example with SGD

Hook: You have four students. Two passed, two failed. You know only how many hours each studied. Can a single perceptron with a sigmoid learn the pattern. And can you compute every step by hand to prove it?

Intuition + Analogy — Learning from mistakes, one student at a time. SGD is like a teacher grading papers one by one. She adjusts her approach after each paper. She does not wait for all 60,000 exams before tweaking her method. Mark's paper (1 hour, failed) nudges her expectation lower. Priya's paper (3 hours, passed) nudges it back up. Each paper causes a nudge. The nudges are noisy but frequent. Over many passes, her grading criteria converge to something sensible.

Where the analogy breaks: A real teacher has intuition about what constitutes a passing grade. SGD starts with zero knowledge — weights are all zeros. And every prediction begins at 0.5 (pure guess).

5.7.1 Problem Setup

Problem: Predict pass (1) or fail (0) from hours studied.

Model: Single perceptron with sigmoid activation:

z = w_{0} \cdot x_{0} + w_{1} \cdot x_{1}, y^= σ (z) = \frac{1}{1 + e ^{- z}}

$x_{0} = 1$ (bias term, always 1)
$x_{1}$ = hours studied
$y^$ = predicted probability of passing
Two learnable weights: $w_{0}$ (bias), $w_{1}$ (hours weight)

Training data (4 instances):

Instance	Bias $x_{0}$	Hours $x_{1}$	Label $y$	Meaning
1	1	1	0	1 hour → fail
2	1	2	1	2 hours → pass
3	1	3	1	3 hours → pass
4	1	4	0	4 hours → fail

Hyperparameters:

Learning rate $η = 0.5$ (large — chosen for rapid demonstration; real training uses ~0.01)
Gradient descent variant: Stochastic Gradient Descent (SGD) — one instance per update
Loss function: Binary cross-entropy (implicit in the gradient formula $(y^- y) \cdot x$ )

5.7.2 Step 1: Initialize Weights to Zero

w_{0} = 0, w_{1} = 0

Starting at zero means the first prediction for any input will be $σ (0) = 0.5$ — the model is maximally uncertain about every student.

5.7.3 Forward Pass and Update Formulas

Forward pass:

z = w_{0} \cdot x_{0} + w_{1} \cdot x_{1}

y^= σ (z) = \frac{1}{1 + e ^{- z}}

SGD update rule (no division by $N$ — one instance at a time):

w_{j}^{new} = w_{j}^{old} - η \cdot (y^- y) \cdot x_{j}

The error term is $(y^- y)$ — predicted probability minus true label. The gradient is error × feature. For SGD, you do not average over $N$ .

5.7.4 Iteration 1 — First Random Instance

Trace — Iteration 1: Student with 1 hour, failed.

Instance chosen: $x_{0} = 1$ , $x_{1} = 1$ , $y = 0$ (1 hour study → fail).

Forward pass:

z = 0 \cdot 1 + 0 \cdot 1 = 0

y^= σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{2} = 0.5

Error: $y^- y = 0.5 - 0 = 0.5$ . The model predicted 50% chance of passing for a student who actually failed. It was wrong by 0.5.

Gradients: Since $\frac{\partial L}{\partial w _{j}} = (y^- y) \cdot x_{j}$ and we use SGD (no $1/ N$ ):

\frac{\partial L}{\partial w _{0}} = 0.5 \cdot 1 = 0.5

\frac{\partial L}{\partial w _{1}} = 0.5 \cdot 1 = 0.5

Weight updates:

w_{0} = 0 - 0.5 \cdot 0.5 = - 0.25

w_{1} = 0 - 0.5 \cdot 0.5 = - 0.25

After iteration 1: $w_{0} = - 0.25$ , $w_{1} = - 0.25$ .

Interpretation: The model saw a student who studied 1 hour and failed. It nudged both weights negative — making it slightly harder to predict "pass" for low study hours.

5.7.5 Iteration 2 — Next Random Instance

Trace — Iteration 2: Student with 3 hours, passed.

Instance chosen: $x_{0} = 1$ , $x_{1} = 3$ , $y = 1$ (3 hours → pass).

Forward pass with current weights $w_{0} = - 0.25$ , $w_{1} = - 0.25$ :

z = (- 0.25) (1) + (- 0.25) (3) = - 0.25 - 0.75 = - 1.0

y^= σ (- 1.0) = \frac{1}{1 + e ^{1.0}} = \frac{1}{1 + 2.71828...} = 0.2689

The exact value of $σ (- 1)$ is $1/ (1 + e) \approx 0.2689$ . This is confirmed: the sigmoid of $- 1$ is about 0.269.

Error: $y^- y = 0.2689 - 1 = - 0.7311$

The model predicted only 26.9% chance of passing for a student who actually passed. It was quite wrong — in the opposite direction from iteration 1.

Gradients:

\frac{\partial L}{\partial w _{0}} = (- 0.7311) \cdot 1 = - 0.7311

\frac{\partial L}{\partial w _{1}} = (- 0.7311) \cdot 3 = - 2.1933

Weight update with $η = 0.5$ :

w_{0}^{new} w_{1}^{new} = - 0.25 - 0.5 \cdot (- 0.7311) = - 0.25 + 0.3656 = 0.1156 = - 0.25 - 0.5 \cdot (- 2.1933) = - 0.25 + 1.0966 = 0.8466

After iteration 2: $w_{0} \approx 0.1156$ , $w_{1} \approx 0.8466$ .

Prof's confirmation: "The values changed from $- 0.25, - 0.25$ to 0.11 and 0.87." Our computed values (0.1156 and 0.8466) match these rounded values — the computation is verified.

Interpretation: The model saw a student who studied 3 hours and passed. It corrected aggressively — the hour weight $w_{1}$ jumped from $- 0.25$ to $0.85$ , now strongly favoring "pass" for higher study hours.

5.7.6 Subsequent Iterations and Convergence

Keep randomly selecting instances, forward-propagating, computing error. And updating weights. Shuffle the data between epochs. Repeat until the weights stabilize.

Convergence pattern: The loss should trend downward across epochs, with noise from the single-instance updates. After many epochs, the weights settle to values that produce correct predictions (when thresholded appropriately) for all four training instances.

5.7.7 Making Predictions with Learned Weights

Once trained, the model predicts:

y^= σ (w_{0} + w_{1} \cdot x_{1})

Apply a threshold to get the binary class. The default threshold is 0.5. But the prof noted a practical issue: after convergence in this example, all predictions might cluster near 1. Using threshold = 0.5 would label everything as class 1. The fix is to raise the threshold — say, to 0.86 — so that only clear "pass" predictions get classified as pass. Threshold tuning is part of model evaluation, covered in Section 5.8.

Pitfall — Large learning rate. The prof used $η = 0.5$ for demonstration. In real practice, this is far too large. A learning rate this big can cause the weights to oscillate wildly or even diverge. Use 0.01, 0.001, or smaller for real training. The large rate here is purely to make the weight changes visible in a hand-worked example.

Pitfall — Zero initialization. Starting all weights at zero is fine for a single perceptron. But for deep networks it breaks symmetry. Every neuron learns the same thing. Random initialization (He, Xavier, LeCun) is the standard for multi-layer networks.

Pitfall — Threshold default. Do not blindly use 0.5. If your classes are imbalanced or your model's output distribution is skewed, tune the threshold on a validation set.

5.7.8 Student Questions and Answers

Q: In the numerical calculation, the learning rate was 0.5 and the gradient was also 0.5. Are they related?

A: No. Pure coincidence. The learning rate $η$ is fixed at 0.5 throughout training. The gradient $(y^- y) \cdot x$ varies at every iteration based on the current weights and the chosen instance. In iteration 1, it happened to be 0.5. In iteration 2, it was $- 0.731$ . They are independent quantities.

Q: Is the weight update applied to all features at once?

A: Yes. Both $w_{0}$ and $w_{1}$ are updated simultaneously in the same iteration. Each weight has its own gradient computed from the same error signal $(y^- y)$ . If you had $d$ features, all $d + 1$ weights (including bias) would be updated together.

Q: Is the selection of instances for SGD completely random?

A: Yes. In any given epoch, you shuffle the data and process instances one by one. The first update could use instance 1, instance 4, or any other. The order does not matter as long as each instance eventually gets used.

Recap + Bridge: You traced SGD through two iterations with real numbers. Forward pass, sigmoid, error, gradient, weight update. The pattern repeats for every instance, every epoch. Next: once the model is trained, how do you measure whether it is any good? Section 5.8 covers accuracy, precision, recall. And the confusion matrix.

Real-World Connection. This tiny example has 4 instances, 2 weights, and 1 feature. Yet the same algorithm trains binary classifiers on millions of examples with thousands of features. The only differences at scale are the ones listed above. The only differences at scale: (a) the learning rate is much smaller (0.001–0.01). (b) Mini-batch SGD replaces pure SGD. (c) The features and weights are matrices instead of scalars. (d) The math runs on GPUs. When a self-driving car classifies "pedestrian / not-pedestrian" from a camera frame, the update rule is still $w := w - η \cdot (y^- y) \cdot x$ . It just multiplies across millions of parameters in parallel.

5.8 Evaluation Metrics for Classification

Hook: Your model says a patient has a rare disease. Should you celebrate? Not yet. If the disease affects only 1 in 10,000 people, a model that always says "healthy" gets 99.99% accuracy. That is a useless model wearing a perfect score. Accuracy can lie. You need metrics that look deeper.

Intuition + Analogy — The security checkpoint. Think of an airport security scanner. It can make two mistakes: flag a harmless bag or miss a dangerous item. A false positive is an inconvenient delay. A false negative could be catastrophic. Precision asks: "When the scanner beeps, how often is there a threat?" Recall asks: "Out of all real threats, how many did it catch?" A scanner that never beeps has perfect precision but zero recall. A scanner that beeps at everything has perfect recall but terrible precision. F1 score balances them.

Where the analogy breaks: Airport security has asymmetric costs. A false negative (missed threat) is far worse than a false positive (extra bag check). In classification, you choose which metric to prioritize based on your domain's real-world costs.

5.8.1 Why Different Metrics

Regression has a universal yardstick: how far is the predicted number from the actual number? MSE, MAE, and $R^{2}$ all measure this. Classification needs more nuanced questions:

What fraction of predictions are correct overall? (Accuracy)
When the model says "positive," how often is it right? (Precision)
How many of the actual positives did the model find? (Recall)
Can we capture both precision and recall in one number? (F1 Score)

5.8.2 Train / Validation / Test Split

Three data partitions, three purposes.

Training set — The data used to learn model weights. The model sees these examples and their labels during training.
Validation set — A small held-out portion used exclusively for tuning hyperparameters. If you want to test learning rates of 0.01, 0.001. And 0.0001, you do not retrain on billions of examples each time. You train quickly on a subset, evaluate on the validation set. And pick the best hyperparameter. Then you train the final model with that hyperparameter on the full training set.
Test set — Completely unseen data used exactly once: at the very end, after all training and tuning is done. This gives the final, honest performance estimate. You must never peek at the test set during training or hyperparameter tuning — doing so leaks information and inflates your reported accuracy.

5.8.3 The Confusion Matrix

Confusion matrix. A 2×2 table that counts every possible outcome of binary classification. Pick one class to call "positive" (the one you care about) and the other as "negative."

	Predicted Positive	Predicted Negative
Actual Positive	True Positive (TP)	False Negative (FN)
Actual Negative	False Positive (FP)	True Negative (TN)

Definitions:

TP — Actual positive, predicted positive. The model got it right.
TN — Actual negative, predicted negative. The model got it right.
FP — Actual negative, predicted positive. A false alarm (Type I error).
FN — Actual positive, predicted negative. A miss (Type II error).

Every classification metric is built from these four numbers.

Visual Intuition. Draw a 2×2 grid. The diagonal (top-left to bottom-right) contains correct predictions: TP and TN. The off-diagonal contains mistakes: FP and FN. A perfect classifier has zeros on the off-diagonal and all counts on the diagonal. A random classifier scatters counts across all four cells.

5.8.4 Accuracy

Accuracy = \frac{TP + TN}{TP + TN + FP + FN}

The fraction of all predictions that are correct. Simple and intuitive — but can be dangerously misleading when classes are imbalanced.

5.8.5 Precision

Precision = \frac{TP}{TP + FP}

Of all instances the model called "positive," how many were actually positive? High precision means the model rarely cries wolf. When it says "this is a positive case," you can trust it.

5.8.6 Recall (Sensitivity)

Recall = \frac{TP}{TP + FN}

Of all instances that are actually positive, how many did the model manage to find? High recall means the model rarely misses a positive case.

Worked Example — Computing all metrics from a confusion matrix.

6 test instances:

Instance	Actual $y$	Predicted $y^$
1	1	1
2	1	0
3	0	1
4	0	0
5	0	1
6	0	0

Confusion matrix: TP = 1, FN = 1, FP = 2, TN = 2.

Accuracy: $\frac{1 + 2}{1 + 2 + 2 + 1} = \frac{3}{6} = 0.5$

Precision: $\frac{1}{1 + 2} = \frac{1}{3} \approx 0.33$

Only 1 in 3 predicted positives was actually positive. The model cries wolf a lot.

Recall: $\frac{1}{1 + 1} = \frac{1}{2} = 0.5$

The model found only half of the actual positive cases.

F1 Score: $2 \cdot \frac{( 1/3 ) \cdot ( 1/2 )}{( 1/3 ) + ( 1/2 )} = 2 \cdot \frac{1/6}{5/6} = 2 \cdot \frac{1}{5} = 0.4$

Sense check: Both precision and recall are low. So F1 is also low (0.4). An F1 of 1.0 would mean perfect precision and perfect recall. An F1 of 0 means at least one of them is zero.

5.8.7 F1 Score

F_{1} = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}

The F1 score is the harmonic mean of precision and recall. Why harmonic mean instead of regular (arithmetic) mean? The harmonic mean is sensitive to imbalance — if either precision or recall is near zero, F1 is near zero. The arithmetic mean of 0.9 and 0.1 is 0.5, which is misleading. The harmonic mean is $2 \cdot (0.9 \cdot 0.1) / (0.9 + 0.1) = 0.18$ — much more honest.

Domain check: Since precision and recall are both in $[0, 1]$ , F1 is also in $[0, 1]$ . Higher is better. F1 = 1 means perfect precision and perfect recall.

5.8.8 Class Imbalance and When to Use F1

Pitfall — Accuracy with imbalanced data. Suppose 99% of emails are not-spam. A model that always predicts "not-spam" gets 99% accuracy — but it catches zero spam. Accuracy masks the failure.

Scope — When to prefer each metric:

Accuracy works when classes are roughly balanced.
Precision matters when false positives are costly (e.g., flagging a legitimate transaction as fraud — you risk annoying a customer).
Recall matters when false negatives are costly (e.g., missing a cancer diagnosis. The cost of a miss is enormous).
F1 score is the go-to single-number summary for imbalanced datasets. It is experimentally proven to give reasonably informed results even when one class dominates.

5.8.9 Student Questions and Answers

Q: The terminology of positive/negative — is it arbitrary?

A: Yes. Whatever class you want to focus on becomes the "positive" class. If you care about detecting pass cases, pass is positive. If you care about detecting disease, disease-present is positive. It is a labeling convention — nothing mathematical depends on which class you call positive. Just be consistent once you choose.

Exam note: Given a confusion matrix, you must be able to compute accuracy, precision, recall. And F1 score — both for binary and per-class (Section 5.11). The prof confirmed this will appear on the exam. Know the formulas by heart.

Recap + Bridge: Accuracy tells you "how often right." Precision says "how trustworthy when positive." Recall says "how many positives caught." F1 balances them. Pick the metric that matches the real-world cost of mistakes. Section 5.9 extends classification to three, ten, or a thousand categories.

Real-World Connection. Medical screening lives and dies by these metrics. A mammogram classifier with 90% recall but 10% precision floods radiologists with false positives. That wastes time and causes unnecessary biopsies. One with 99.9% precision but 50% recall misses half the cancers. The FDA requires manufacturers to report sensitivity and specificity. In spam filtering, a false positive is far worse than a false negative. Losing one important email angers users more than seeing fifty spam messages. Gmail's spam filter is tuned for near-zero false positive rate. It lets some spam through deliberately.

5.9 Multi-Class Classification

Hook: A single perceptron draws one line — splitting the world into "yes" and "no." But what if you need three answers? Or ten? Or a thousand? One line cannot carve three regions. You need more lines. And a way to turn their outputs into a single coherent prediction.

Intuition + Analogy — The panel of judges. Imagine a talent show with three judges. Each judge specializes in one contestant. Judge A watches for signs of "Cat," Judge B watches for "Dog," and Judge C watches for "Fox." They all see the same performance (the same input features). Each judge gives a raw score — Judge A says 5, Judge B says 3, Judge C says -5. These raw scores are on different scales and not directly comparable. The softmax function acts like a normalization panel: it converts each judge's score into a probability, ensuring they all sum to 1. The contestant with the highest probability wins. The judges don't talk to each other — they just each learn, through training, what their assigned class looks like.

Where the analogy breaks: Real judges have preconceived biases. Softmax neurons start from random weights and learn purely from labeled data.

5.9.1 The Problem

A single perceptron with sigmoid draws exactly one decision boundary, splitting the feature space into two regions. With $K > 2$ classes, you need $K$ decision boundaries — each one separating one class from all others.

The solution: place $K$ perceptrons at the output layer. Each perceptron learns to answer: "Is this instance class $k$ or not?" The input is the same for all $K$ perceptrons. Each perceptron has its own weight vector $w^{(k)}$ and bias $b^{(k)}$ .

5.9.2 Multi-Output Perceptron Architecture

Architecture for $K$ classes:

Perceptron 1: Weight vector $w^{(1)}$ , bias $b^{(1)}$ . Trained to say "class 1 vs. the rest."
Perceptron 2: Weight vector $w^{(2)}$ , bias $b^{(2)}$ . Trained to say "class 2 vs. the rest."
...
Perceptron $K$ : Weight vector $w^{(K)}$ , bias $b^{(K)}$ . Trained to say "class $K$ vs. the rest."

With $d$ features (plus bias), each perceptron has $d + 1$ weights. With $K$ perceptrons, that is $K \times (d + 1)$ weights total. For 3 classes and 2 features plus bias: $3 \times 3 = 9$ weights.

No communication between output neurons. Each perceptron operates independently. They receive the same input $x$ but learn different weight vectors through training — each matched against its own component in the one-hot label.

5.9.3 Why Not Use a Single Neuron for Multi-Class

A single perceptron gives one real number as output. With three classes, how would you map a single number to three categories? You cannot — a scalar has no room for three-way decisions. You need at least $K$ output values (one per class) to make a $K$ -way choice. Each output neuron provides one score; softmax converts them all into a probability distribution.

5.9.4 The Softmax Activation Function

The $K$ perceptrons produce raw scores $z_{1}, z_{2}, \dots, z_{K}$ , each in $(- \infty, + \infty)$ . These are on different, uncalibrated scales. The softmax function converts all $K$ scores into probabilities that:

Are each in $(0, 1)$ ,
Sum to exactly 1,
Preserve the relative ordering of the raw scores.

softmax (z_{k}) = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}

$z_{k} = w^{(k) ⊤} x + b^{(k)}$ : raw score from perceptron $k$ .
$e^{z_{k}}$ : exponentiate to make all scores positive (exponential is always > 0).
$\sum_{j = 1}^{K} e^{z_{j}}$ : normalization constant — sum of all exponentiated scores.
$y^_{k} = softmax (z_{k})$ : predicted probability that the input belongs to class $k$ .

Worked Example — Softmax computation.

Raw scores: $z_{1} = 5$ , $z_{2} = 3$ , $z_{3} = - 5$ .

Step 1 — Exponentiate:

e^{5} = 148.41, e^{3} = 20.09, e^{- 5} = 0.0067

Step 2 — Sum: $148.41 + 20.09 + 0.0067 = 168.51$

Step 3 — Divide each by the sum:

y^_{1} = \frac{148.41}{168.51} \approx 0.881

y^_{2} = \frac{20.09}{168.51} \approx 0.119

y^_{3} = \frac{0.0067}{168.51} \approx 0.00004

Check: $0.881 + 0.119 + 0.00004 \approx 1.0$ . The model predicts class 1 with 88.1% confidence.

Sense check: The highest raw score (5) maps to the highest probability. The lowest raw score ( $- 5$ ) maps to nearly zero. The exponential function amplifies differences — a score gap of 2 (from 5 to 3) becomes a probability gap of ~76 percentage points. Softmax aggressively favors the maximum.

5.9.5 Symbol Registry — Softmax and Multi-Class

Symbol	Meaning	LaTeX	Type / Domain
$K$	number of classes	$K$	integer, $K \geq 2$
$z_{k}$	raw score for class $k$	$z_{k} = w^{(k) ⊤} x + b^{(k)}$	$(- \infty, + \infty)$
$w^{(k)}$	weight vector for class $k$	$w^{(k)}$	$R^{d}$
$b^{(k)}$	bias for class $k$	$b^{(k)}$	scalar
$y^_{k}$	softmax probability for class $k$	$y^_{k} = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}$	$(0, 1)$

5.9.6 Properties of Softmax

Outputs sum to 1. The softmax output is a valid probability distribution. This is its defining property.
Differentiable everywhere. Required for backpropagation. The derivative is well-behaved (though slightly more complex than sigmoid's derivative).
Translation invariant. Adding a constant $c$ to every $z_{k}$ does not change the softmax output: $\frac{e ^{z_{k} + c}}{\sum _{j} e ^{z_{j} + c}} = \frac{e ^{c} \cdot e ^{z_{k}}}{e ^{c} \cdot \sum _{j} e ^{z_{j}}} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$ In practice, implementations subtract $max (z)$ from all scores before exponentiation for numerical stability (to avoid overflow from $e^{500}$ ).
Reduces to sigmoid for $K = 2$ . When $K = 2$ , $softmax (z_{1}) = \frac{1}{1 + e ^{- (z_{1} - z_{2})}}$ . This is a sigmoid of the score difference. If you fix $z_{2} = 0$ , it is exactly the sigmoid.
Order-preserving. If $z_{a} > z_{b}$ , then $y^_{a} > y^_{b}$ . The class with the highest raw score always gets the highest probability.

5.9.7 One-Hot Encoding

The model outputs a vector $y^= [y^_{1}, y^_{2}, \dots, y^_{K}]$ of $K$ probabilities. The true label must also be a vector of length $K$ for the loss computation. One-hot encoding converts a class label into a vector with exactly one "1" and $K - 1$ "0"s:

Class 1 → $[1, 0, 0, \dots, 0]$
Class 2 → $[0, 1, 0, \dots, 0]$
Class $K$ → $[0, 0, 0, \dots, 1]$

Each component maps to one perceptron's output. If the true class is 2 and $K = 3$ , the one-hot label is $[0, 1, 0]$ . Perceptron 1's output compares against 0, perceptron 2's against 1, and perceptron 3's against 0. Training pushes $y^_{2} \to 1$ and $y^_{1}, y^_{3} \to 0$ .

5.9.8 Categorical Cross-Entropy Loss

Binary cross-entropy generalizes naturally to $K$ classes. With one-hot labels $y = [y_{1}, \dots, y_{K}]$ where exactly one $y_{k} = 1$ :

L (y^, y) = - k = 1 \sum K y_{k} \cdot lo g (y^_{k})

Since only one $y_{k} = 1$ (all others are 0), this simplifies to $L = - lo g (y^_{k^{*}})$ where $k^{*}$ is the true class. Only the probability assigned to the correct class matters.

Gradient simplification. When paired with softmax, the gradient again collapses to a clean form:

\frac{\partial L}{\partial z _{k}} = y^_{k} - y_{k}

This is the same structure as binary classification: predicted minus actual. The backward pass for multi-class classification is structurally identical — each perceptron sees $(y^_{k} - y_{k})$ as its error signal.

For mini-batch GD with batch size $B$ , the loss is averaged over $B$ instances. The gradient form is:

\nabla_{W} L = \frac{1}{B} \cdot X_{batch}^{⊤} \cdot (Y^- Y)

where $X_{batch}$ is $B \times d$ , $Y^$ is $B \times K$ (softmax probabilities), and $Y$ is $B \times K$ (one-hot labels).

5.9.9 Student Questions and Answers

Q: For multi-class, is it always a multi-perceptron model? Can we use a single perceptron?

A: You need more than one perceptron at the output layer. A single perceptron outputs one scalar — you cannot map one number to three categories. For $K$ classes, you need $K$ perceptrons. For 4 classes, 4 perceptrons. For 10 classes, 10 perceptrons. Each provides one score; softmax combines them into a probability distribution.

Q: Does each perceptron learn the characteristics of a particular class?

A: Yes. Each perceptron becomes specialized to its assigned class through training. The output neurons have no direct connections to each other — they do not communicate. The same input feeds all of them. Through weight updates (matching each perceptron's output against its corresponding one-hot component), they individually learn to represent their class.

Q: Is the gradient formula the same for multi-class as it was for binary classification?

A: Yes — structurally the same. Each perceptron is doing "class k or not," which is essentially a binary classification. The error signal for perceptron $k$ is $y^_{k} - y_{k}$ . The same form carries through the backward pass.

Q: Can a single-layer perceptron handle non-linear data in multi-class?

A: No. Single-layer models (one layer of $K$ perceptrons with softmax) can only draw linear decision boundaries. For non-linearly separable data — where classes are intertwined in curved regions — you need multiple layers (a deep neural network / multilayer perceptron). The prof confirmed this is coming in later sessions.

Recap + Bridge: Multi-class uses $K$ perceptrons, each learning a "class k vs. rest" boundary. Softmax converts raw scores to probabilities. Categorical cross-entropy generalizes the binary loss. The gradient structure remains: (predicted − actual) × feature. The backward pass is no harder than binary. Section 5.10 walks through a complete mini-batch numerical example.

Real-World Connection. Softmax with categorical cross-entropy is the final layer in almost every image classifier. ResNet-50, trained on ImageNet's 1,000 classes, ends with softmax over 1,000 outputs. GPT and other language models use softmax over 50,000+ tokens. The model outputs a probability distribution over every possible word. The highest-scoring token is chosen or sampled. The same softmax runs billions of times per day across every deployed language model. From smartphone keyboards to ChatGPT, it is the same formula.

5.10 Multi-Class Classification: Worked Numerical Example with Mini-Batch GD

Hook: Nine weights. Four data points. Three classes. Two features. Two batches. Can you trace the entire mini-batch forward pass, softmax, gradient computation. And weight update — in matrix form — with a pencil and paper?

Intuition + Analogy — The three detectives. Three detectives stand in a room. Each detective has a different theory about "what kind of animal is in this photo?" Detective A is biased toward finding cats. Detective B looks for dogs. Detective C is the fox specialist. They all see the same photo (same input features). Each one whispers a raw score: A says 5, B says 3, C says -5. The softmax moderator normalizes these into probabilities — A gets 88%, B gets 12%, C gets ~0%. The moderator declares the photo is most likely a cat. But then the ground truth arrives: it was actually a dog (one-hot label = [0, 1, 0]). Detective B should have scored higher. All three detectives adjust their theories (weights) accordingly — B increases the weight on dog-like features, A and C decrease theirs. This is one mini-batch update.

5.10.1 Problem Setup

Problem: 3-class classification (cat / dog / fox) using 2 features plus bias.

Input: Each instance is a vector $x = [x_{0}, x_{1}, x_{2}]^{⊤}$ where $x_{0} = 1$ (bias).

Model: 3 perceptrons at the output layer. Each perceptron has 3 weights ( $w_{0}, w_{1}, w_{2}$ ). Total weights: $3 \times 3 = 9$ .

Weight matrix $W$ (3 × 3):

W = w_{01} w_{11} w_{21} w_{02} w_{12} w_{22} w_{03} w_{13} w_{23}

Row 0: bias weights for all 3 classes
Row 1: weights for feature $x_{1}$ for all 3 classes
Row 2: weights for feature $x_{2}$ for all 3 classes
Column $k$ : weight vector for class $k$ perceptron

Data (4 instances, 3 classes):

Instance	Original Label	One-Hot Vector
1	1 (Cat)	$[1, 0, 0]$
2	2 (Dog)	$[0, 1, 0]$
3	3 (Fox)	$[0, 0, 1]$
4	2 (Dog)	$[0, 1, 0]$

The one-hot encoding follows the prof's convention: class 1 → $[1, 0, 0]$ , class 2 → $[0, 1, 0]$ , class 3 → $[0, 0, 1]$ . Instance 4 being class 2 (dog) is the verified mapping from the lecture.

5.10.2 Hyperparameters

Gradient descent variant: Mini-batch
Batch size $B = 2$ : 4 instances → 2 batches per epoch
Learning rate $η = 0.1$
Weight initialization: Random small values (not all zeros — to show variation)
Activation: Softmax at output
Loss: Categorical cross-entropy

Note on numerical values. The prof displayed exact feature values visually during the lecture. The lecture covers the complete procedure — forward pass, softmax, error, gradient. And weight update — but the specific feature numbers shown on screen were noted verbally rather than written down. The procedure below is correct and general. The same steps apply regardless of the exact feature values. The sample values used below illustrate the matrix operations taught by the instructor.

5.10.3 Step 1: Take the First Batch (Instances 1 and 2)

Batch assembly. The first batch contains instances 1 and 2.

Input matrix $X_{batch}$ ( $B \times (d + 1) = 2 \times 3$ ):

X_{batch} = [x_{0}^{(1)} x_{0}^{(2)} x_{1}^{(1)} x_{1}^{(2)} x_{2}^{(1)} x_{2}^{(2)}] = [11 x_{11} x_{21} x_{12} x_{22}]

Each row is one instance. Column 0 is the bias term (always 1). Columns 1 and 2 are the two features.

One-hot label matrix $Y$ ( $B \times K = 2 \times 3$ ):

Y = [100100]

Row 1: $[1, 0, 0]$ — instance 1 is class 1 (cat). Row 2: $[0, 1, 0]$ — instance 2 is class 2 (dog).

5.10.4 Step 2: Forward Pass

Matrix multiplication: $Z = X_{batch} \cdot W$

This produces a $2 \times 3$ matrix. Each row contains the raw scores $[z_{1}, z_{2}, z_{3}]$ for one instance:

Z = [z_{1}^{(1)} z_{1}^{(2)} z_{2}^{(1)} z_{2}^{(2)} z_{3}^{(1)} z_{3}^{(2)}]

The computation for instance $i$ , class $k$ : $z_{k}^{(i)} = w_{0 k} + w_{1 k} \cdot x_{1}^{(i)} + w_{2 k} \cdot x_{2}^{(i)}$ .

Dimensional check: $X_{batch} \in R^{2 \times 3}$ , $W \in R^{3 \times 3}$ , so $Z \in R^{2 \times 3}$ . Correct.

5.10.5 Step 3: Apply Softmax (Row-wise)

Apply softmax independently to each row of $Z$ :

y^_{k}^{(i)} = \frac{e ^{z_{k}^{(i)}}}{\sum _{j = 1}^{3} e ^{z_{j}^{(i)}}}

This gives $Y^\in R^{2 \times 3}$ . Each row sums to 1 — a valid probability distribution over the 3 classes. The predicted class for instance $i$ is $ar g max_{k} y^_{k}^{(i)}$ .

5.10.6 Step 4: Compute Error

Error = Y^- Y

This is a $2 \times 3$ matrix. Each entry $y^_{k}^{(i)} - y_{k}^{(i)}$ is the signed error for class $k$ on instance $i$ . For the correct class, the error is negative (predicted probability < 1). For incorrect classes, the error is positive (predicted probability > 0).

5.10.7 Step 5: Compute Gradient

\nabla_{W} L = \frac{1}{B} \cdot X_{batch}^{⊤} \cdot (Y^- Y)

$X_{batch}^{⊤} \in R^{3 \times 2}$ : transpose of the input matrix.
$(Y^- Y) \in R^{2 \times 3}$ : error matrix.
Product: $R^{3 \times 2} \cdot R^{2 \times 3} = R^{3 \times 3}$ — same shape as $W$ .

Dimensional check: The gradient has the same dimensions as the weight matrix ( $3 \times 3$ ). Each entry tells you how to adjust one weight. Division by $B = 2$ averages the gradient over the batch.

Rationale: The prof explained: "Multiply the error with the features. Find the mean — here mean corresponds to 2 because we have a batch of size 2. Divide by 2 to get the gradient." Each weight's gradient is the average of (feature × error) across the $B$ instances in the batch.

5.10.8 Step 6: Update Weights

W^{new} = W^{old} - η \cdot \nabla_{W} L

With $η = 0.1$ :

W^{new} = W^{old} - 0.1 \cdot \frac{1}{2} \cdot X_{batch}^{⊤} \cdot (Y^- Y)

This is one batch complete. All 9 weights are updated simultaneously in matrix form.

5.10.9 Step 7: Second Batch and Epoch Completion

Take the second batch (instances 3 and 4 — classes 3 and 2 respectively). Repeat steps 2–6 with the updated weights. Once both batches are processed, one epoch is complete. The loss should decrease across epochs as the weights converge.

5.10.10 Computational Graph Summary

The computational graph for mini-batch multi-class classification differs from linear regression in three ways:

Batch subscript $B$ . Only $B$ instances flow through each forward/backward pass, not all $N$ . Input $X_{B}$ , output $Y_{B}$ , and all intermediate activations carry the batch subscript.
Softmax activation. The output layer uses softmax (not identity or sigmoid) to produce class probabilities that sum to 1.
Averaging by $B$ . The gradient divides by $B$ (not $N$ ). The update rule: $W^{new} = W^{old} - η \cdot \nabla_{W} L$ .

Worked Example — Gradient shape check with concrete dimensions.

Input batch: $X_{batch} \in R^{2 \times 3}$ (2 instances, 3 features including bias).

Weight matrix: $W \in R^{3 \times 3}$ .

Forward pass: $Z = X_{batch} \cdot W \in R^{2 \times 3}$ (2 instances, 3 raw scores each).

Softmax: $Y^\in R^{2 \times 3}$ (row-wise, each row sums to 1).

Error: $Y^- Y \in R^{2 \times 3}$ .

Gradient: $\nabla_{W} L = \frac{1}{2} \cdot X_{batch}^{⊤} \cdot (Y^- Y) \in R^{3 \times 3}$ .

All matrix dimensions are consistent through every step. This is what "the math checks out" means in practice.

Recap + Bridge: The multi-class mini-batch training loop generalizes the binary SGD example from Section 5.7. Replace scalars with matrices. Replace sigmoid with softmax. Replace binary cross-entropy with categorical cross-entropy. The gradient form stays: error times input, averaged over the batch. Section 5.11 closes the loop with multi-class evaluation.

Real-World Connection. This matrix-form training loop is exactly what PyTorch executes. When you call loss.backward() and optimizer.step(), the six steps run: forward, softmax, error, gradient, average, update. Under the hood, nn.Linear(3, 3) defines a $3 \times 3$ weight matrix. nn.CrossEntropyLoss() combines softmax + categorical cross-entropy. It uses the numerical stability trick of subtracting max before exponentiation. optim.SGD(model.parameters(), lr=0.1) updates all 9 weights with the gradient. The same pattern scales to networks with millions of weights. The matrix dimensions grow, but the six steps (forward, softmax, error, gradient, average, update) are identical.

5.11 Multi-Class Evaluation and Confusion Matrix

Hook: Your multi-class model got 80% accuracy. Sounds decent. But class 2 has 0% recall — it missed every single instance of that class. Accuracy hid the disaster. When you have $K$ classes, you need to look at each class individually.

Intuition + Analogy — Three separate report cards. A confusion matrix for $K$ classes is like a report card with $K$ subjects. The overall GPA (accuracy) might look fine. But you could be failing math while acing English. Per-class precision and recall are the subject-specific grades. Precision asks: "When the teacher said you were good at math, was that correct?" Recall asks: "How many math problems did you solve?" You need both to diagnose where the model is weak.

5.11.1 Extending the Confusion Matrix

For $K$ classes, the confusion matrix is a $K \times K$ table. Rows are actual classes. Columns are predicted classes. The diagonal (top-left to bottom-right) contains correct predictions. Every off-diagonal cell is a misprediction — the row tells you what the class actually was, the column tells you what the model incorrectly predicted.

Example — 3-class confusion matrix:

	Predicted 1	Predicted 2	Predicted 3	Row Total
Actual 1	50	3	2	55
Actual 2	5	38	4	47
Actual 3	1	2	45	48
Column Total	56	43	51	150

Diagonal (50, 38, 45): 50 + 38 + 45 = 133 correct predictions.
Off-diagonal examples: 3 instances of class 1 were wrongly predicted as class 2. 5 instances of class 2 were wrongly predicted as class 1.
Row sums: total actual instances per class.
Column sums: total predicted instances per class.

Overall Accuracy:

Accuracy = \frac{50 + 38 + 45}{150} = \frac{133}{150} \approx 0.887

87.7% looks good — but let us check per-class.

5.11.2 Per-Class Precision and Recall

Precision and recall are computed per class. For class $k$ :

Precision for class $k$ (column view):

Precision_{k} = \frac{diagonal _{k}}{sum of column k}

"Of all instances predicted as class $k$ , what fraction were actually class $k$ ?"

Recall for class $k$ (row view):

Recall_{k} = \frac{diagonal _{k}}{sum of row k}

"Of all instances that truly belong to class $k$ , what fraction did the model correctly find?"

Worked Example — Per-class metrics from the 3×3 confusion matrix.

Class 1:

Precision: $\frac{50}{50 + 5 + 1} = \frac{50}{56} \approx 0.893$
Recall: $\frac{50}{50 + 3 + 2} = \frac{50}{55} \approx 0.909$

Class 1 is handled well — both precision and recall are ~90%.

Class 2:

Precision: $\frac{38}{3 + 38 + 2} = \frac{38}{43} \approx 0.884$
Recall: $\frac{38}{5 + 38 + 4} = \frac{38}{47} \approx 0.809$

Class 2 has good precision (88.4%) but lower recall (80.9%). The model misses about 1 in 5 class-2 instances — they get misclassified as class 1 (5 cases) or class 3 (4 cases).

Class 3:

Precision: $\frac{45}{2 + 4 + 45} = \frac{45}{51} \approx 0.882$
Recall: $\frac{45}{1 + 2 + 45} = \frac{45}{48} \approx 0.938$

Class 3 has the best recall. The model rarely misses it.

Sense check: All three per-class precisions are around 0.88–0.89. But recalls vary from 0.81 to 0.94. Class 2 is the model's weakest class. It gets confused with classes 1 and 3 more than the others do.

5.11.3 Per-Class F1 Score

For class $k$ :

F_{1, k} = 2 \cdot \frac{Precision _{k} \cdot Recall _{k}}{Precision _{k} + Recall _{k}}

Class-by-class F1 from the example:

Class 1: $F_{1} = 2 \cdot \frac{0.893 \cdot 0.909}{0.893 + 0.909} \approx 0.901$
Class 2: $F_{1} = 2 \cdot \frac{0.884 \cdot 0.809}{0.884 + 0.809} \approx 0.845$
Class 3: $F_{1} = 2 \cdot \frac{0.882 \cdot 0.938}{0.882 + 0.938} \approx 0.909$

Domain check: All F1 scores are between 0 and 1. Class 2 has the lowest F1 (0.845), confirming it is the model's weakest class. The harmonic mean catches this. It penalizes the low recall (0.809) more than the arithmetic mean would.

Overall F1: There are two common ways to aggregate per-class F1 into a single number:

Macro F1: Average the per-class F1 scores equally. $\frac{0.901 + 0.845 + 0.909}{3} \approx 0.885$ . Every class has equal weight regardless of size.
Weighted F1: Average F1 scores weighted by the number of true instances per class. Gives more weight to larger classes.

5.11.4 Decision Boundary Intuition

Visual Intuition. For $K = 3$ , picture the 2D feature space with three colored regions. The three perceptrons each learn one linear decision boundary. Where all three boundaries intersect, you get three wedge-shaped regions. Each region is where one class's softmax probability is highest. The boundaries meet at a point. At that point, all three scores are roughly equal. The model is most uncertain there.

The prof described it: "We need three distinct decision boundaries. When you put it together, you get a feature space beautifully separated into three regions." Each perceptron pushes its region outward. The boundaries shift with each weight update. They move until the regions align with the true class clusters.

Pitfall — Relying on accuracy alone. In the example, accuracy is 88.7%. But class 2 recall is only 80.9%. The model misses nearly 1 in 5 class-2 instances. If class 2 represents a rare but critical condition (e.g., a specific disease), a 19.1% miss rate is unacceptable even at 88.7% overall accuracy. Always check per-class metrics.

Pitfall — Confusing precision and recall for multi-class. In binary classification, there is one positive class and one negative class. In multi-class, every class is its own "positive" — precision and recall are defined per class by treating that class as positive and all others as negative. The column sum is TP + FP for that class. The row sum is TP + FN.

Exam note: The prof explicitly said to be able to compute per-class precision, recall. And F1 from a confusion matrix. Know that precision = diagonal / column sum, recall = diagonal / row sum. The diagonal contains correct predictions; everything off-diagonal is an error.

Recap + Bridge: A $K \times K$ confusion matrix tells you exactly where your model succeeds and fails. Per class. Precision answers "how trustworthy are class-k predictions?" Recall answers "how many class-k instances did you find?" F1 balances them. This is the final piece of the classification toolkit. The appendices consolidate exam guidance and real-world applications.

Real-World Connection. Multi-class confusion matrices are standard in medical imaging diagnostics. A skin lesion classifier might predict melanoma, benign nevus, or seborrheic keratosis from a dermoscopy image. The 3×3 matrix reveals whether the model confuses melanoma with benign nevi. That is a dangerous error — high FP for benign, high FN for melanoma. Regulatory approval requires reporting per-class sensitivity and specificity. The same matrix pattern appears in every production multi-class system. Product categorization, emotion detection, voice assistants — all use it.

Exam Guidance Summary

Exam note — Format. The exam tests analytical reasoning, numerical computation, plot interpretation. And code reading. No theoretical-only questions. No code writing required.

Numerical Questions

Expect a problem like the hours-of-study → pass/fail worked example (Section 5.7). Given an instance or batch:

Compute the forward pass ( $z = w^{⊤} x + b$ , then sigmoid or softmax).
Compute the error ( $y^- y$ ).
Compute gradients ( $(y^- y) \cdot x_{j}$ ).
Compute weight updates ( $θ_{j} := θ_{j} - η \cdot gradient$ ).

Also: batch size calculations. Given $N$ instances and batch size $B$ , find:

Number of batches per epoch: $⌈ N / B ⌉$ .
Weight updates per epoch: same as number of batches.
Weight updates for $T$ epochs: $⌈ N / B ⌉ \times T$ .

Analytical (Scenario-Based) Questions

Given a use case, pick the right gradient descent variant and justify:

Accuracy priority → Batch GD.
Speed priority → SGD or mini-batch.
Limited memory → SGD or small mini-batch.
Large dataset → Mini-batch or SGD.
GPU available → Mini-batch (choose $B$ to fit GPU memory).

Justify across dimensions: accuracy, speed, stability, computational resources, number of instances. And data dimensionality.

Plot Interpretation

Given cost-curve plots for different GD trajectories:

Smoothest curve with steady descent → Batch GD.
Slight wiggles, generally downward → Mini-batch GD.
Large oscillations, bouncing around → SGD.
Extreme oscillations that diverge → Learning rate too high.
Flat but noisy, never converging → Learning rate too low, or no schedule.

Code Interpretation

Given a code snippet, identify:

Hyperparameters (learning rate, batch size, number of epochs/iterations).
Convergence criterion (fixed iterations, loss threshold, gradient threshold, early stopping).
Gradient descent variant (batch, mini-batch, or stochastic).

Confusion Matrix Questions

Given a confusion matrix (binary or $K \times K$ ), compute:

Accuracy: sum of diagonal / total.
Precision: TP / (TP + FP). For multi-class: diagonal / column sum.
Recall: TP / (TP + FN). For multi-class: diagonal / row sum.
F1 score: harmonic mean of precision and recall. Per-class for multi-class.

Key Formulas to Memorize

Formula	Usage
$σ (z) = \frac{1}{1 + e ^{- z}}$	Sigmoid activation
$σ^{'} (z) = σ (z) (1 - σ (z))$	Sigmoid derivative
$lo g (\frac{y ^}{1 - y ^}) = w^{⊤} x + b$	Logit / log-odds
$L = - [y lo g (y^) + (1 - y) lo g (1 - y^)]$	Binary cross-entropy
$θ := θ - η \cdot (y^- y) \cdot x$	SGD weight update
$θ := θ - η \cdot \frac{1}{B} \sum (y^- y) \cdot x$	Mini-batch weight update
$softmax (z_{k}) = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$	Softmax for multi-class
$Precision = \frac{TP}{TP + FP}$	Precision
$Recall = \frac{TP}{TP + FN}$	Recall
$F_{1} = 2 \cdot \frac{P \cdot R}{P + R}$	F1 Score

Key Industry Applications

Game-Playing AI

Deep neural networks process raw screen pixels as input. They output game-move recommendations. Training needs learning rate schedules. Start at $η = 0.1$ for the first ~100K frames. Then drop to $0.05$ , then $0.01$ . The same GD variants and debugging principles apply. Just at millions of parameters and billions of frames. DeepMind (AlphaGo, AlphaStar) and OpenAI (OpenAI Five) used these techniques for superhuman performance.

Mini-Batch Gradient Descent — The Industry Default

Every major framework defaults to mini-batch SGD. PyTorch's DataLoader, TensorFlow's tf.data. And JAX's data pipelines all implement randomized batching. Batch size is chosen to fill GPU memory. Typically 32–256 for images, 8–64 for large language models. Mini-batch gives many updates per epoch and stable gradients. That dual benefit makes it the practical standard.

Spam Detection

Real-world email systems classify every message as spam or not-spam. Features include keyword frequency, sender reputation, header metadata. And attachment characteristics. They also use behavioral signals — how quickly users delete or mark as spam. Systems like Gmail process hundreds of billions of emails daily. Latency is under 100ms per message.

Medical Diagnosis

Binary and multi-class models assist clinical decision-making. Features: symptoms, lab results, patient history. The model flags potential disease (present vs. absent). Multi-class variants grade severity (mild / moderate / severe). Recall is prioritized in deployment. Missing a diagnosis costs more than a false alarm that a doctor can review.

Convolutional Neural Networks for Image Features

CNNs learn hierarchical feature extractors automatically. Early layers detect edges. Middle layers detect shapes. Deep layers detect object parts. No manual feature engineering needed for images. CNNs like ResNet and EfficientNet are the backbone of modern computer vision. They power face recognition and medical image analysis.

Sentiment Analysis

Multi-class classification of text: positive, negative, neutral. Used on product reviews, social media. And customer feedback. Features come from NLP techniques. The same softmax + categorical cross-entropy setup powers these systems. Amazon and Yelp use them to surface the most helpful reviews.

Weight Initialization in Practice

Libraries like scikit-learn, PyTorch (nn.init). And TensorFlow provide built-in initializers. He, Xavier (Glorot). And LeCun are standard. He initialization is the default for layers followed by ReLU. Xavier is the default for tanh or sigmoid. These are one-line function calls. Understanding how they preserve variance helps debug training failures.

DNN Lecture 05 notes · Gradient Descent Variants, Classification, and Evaluation

Deep Neural Networks· postgraduate· 2026-07-15

Sections Breakdown

15.1 Recap of Linear Regression and Debugging Tips

Learning rate, stopping criteria, weight initialization, feature scaling, and over/underfitting for a single-perceptron linear regressor.

25.2 Types of Gradient Descent

Batch, mini-batch, and stochastic gradient descent: trade-offs in smoothness, speed, memory, and the ability to escape local minima.

35.3 Mini-Batch Gradient Descent Formulation

The shuffle-split-update loop, its math, and how batch size controls the speed-stability trade-off.

45.4 Classification: Concepts and Types

Binary, multi-class, and multi-label classification and how to choose the right framing.

55.5 Sigmoid Activation and Logistic Regression

The sigmoid function, its derivative, the logit, and why linear regression fails at classification.

65.6 Binary Cross-Entropy Loss

Why cross-entropy punishes confident mistakes and how its gradient collapses to (y-hat - y) x feature with sigmoid.

75.7 Binary Classification: Worked Numerical Example with SGD

A full hand-worked SGD training of a sigmoid perceptron on a pass/fail dataset.

85.8 Evaluation Metrics for Classification

Confusion matrix, accuracy, precision, recall, F1, train/validation/test splits, and class imbalance.

95.9 Multi-Class Classification

K-output perceptrons, softmax activation, one-hot encoding, and categorical cross-entropy.

105.10 Multi-Class Classification: Worked Numerical Example with Mini-Batch GD

A hand-worked mini-batch softmax training on a 3-class problem.

115.11 Multi-Class Evaluation and Confusion Matrix

Per-class precision, recall, F1 and decision-boundary intuition for K classes.

12Exam Guidance Summary

The professor's exam strategy: numerical, scenario, plot-interpretation, code-reading, and confusion-matrix questions, plus key formulas.

13Key Industry Applications

Where these ideas appear in production: game-playing AI, spam filters, medical diagnosis, CNNs, sentiment analysis, and initialization.

Postgraduate students studying Deep Neural Networks

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Linear Regression Recap and Debugging

Must-know: The most common training failure is a bad learning rate eta. If the loss oscillates or goes NaN, lower eta; if it is flat, raise it slightly. Four stopping criteria exist: fixed iterations, a loss threshold, a gradient threshold, and early stopping. Always check eta before changing the model architecture. Weight initialization and feature scaling also affect convergence.

$J (w, b) = \frac{1}{N} i = 1 \sum N (y^{(i)} - y^^{(i)})^{2}$

Top pitfall: Using eta = 1.0 or higher usually signals broken data scaling; in practice eta rarely exceeds 0.2.

Self-check: If your training loss keeps climbing instead of falling, which control knob should you turn first?

Connects to: Mini-batch gradient descent (5.3); Gradient descent variants (5.2); Feature engineering (5.1.4)

Gradient Descent Variants

Must-know: Batch GD uses all N examples for one smooth update per epoch. Stochastic GD (SGD) uses a single example, the noisiest but with the most updates. Mini-batch GD uses B examples and is the practical default. Noisier gradients can help escape shallow local minima.

$θ := θ - η \cdot \frac{1}{B} i \in batch \sum \nabla_{θ} L^{(i)}$

Top pitfall: Assuming SGD always converges fastest - its noisy updates may never settle near the minimum; mini-batch is the best trade-off.

Self-check: With N = 60,000 instances and batch size B = 128, how many weight updates happen in one epoch?

Connects to: Mini-batch formulation (5.3); Learning rate debugging (5.1.2); Batch-size math (5.2.5)

Mini-Batch Gradient Descent

Must-know: Mini-batch GD splits the data into shuffled chunks of size B, computes the gradient on each chunk, and updates after each chunk. The loss is averaged over B, not N. Larger B means more stable but slower updates; smaller B means noisier but faster.

$θ_{j} := θ_{j} - η \cdot \frac{1}{B} i = 1 \sum B \frac{\partial L ^{(i)}}{\partial θ _{j}}$

Top pitfall: Forgetting to shuffle the data before each epoch - a fixed order lets the model learn spurious patterns from the sequence.

Self-check: Why does the mini-batch gradient have variance that scales as 1/B?

Connects to: Gradient descent variants (5.2); SGD worked example (5.7); Learning rate (5.1)

Classification: Binary, Multi-Class, Multi-Label

Must-know: Classification predicts a categorical label. Binary has two classes (one sigmoid output). Multi-class has K mutually exclusive classes (K softmax outputs, pick the highest). Multi-label makes K independent yes/no decisions. Never use a multi-class setup when the answer can be multiple labels.

$y^= σ (z) (binary); y^_{k} = softmax (z)_{k} (multi-class)$

Top pitfall: Confusing multi-class (one label from K) with multi-label (multiple labels) - a multi-class model can only pick one class and cannot output 'both'.

Self-check: An image can be both 'cat' and 'AI-generated'. Which classification framing fits, and why?

Connects to: Sigmoid (5.5); Softmax (5.9); Multi-label warning (5.4.4)

Sigmoid Activation and Logistic Regression

Must-know: The sigmoid sigma(z) = 1/(1+e^{-z}) squashes any real number into (0,1), turning a linear perceptron into a binary classifier. Its derivative is self-contained: sigma'(z) = sigma(z)(1-sigma(z)), so backprop is cheap. The logit is the inverse mapping.

$σ (z) = \frac{1}{1 + e ^{- z}}, σ^{'} (z) = σ (z) (1 - σ (z))$

Top pitfall: Trying to use min-max normalization instead of sigmoid - normalization needs a finite domain, but the perceptron output z is unbounded (-infinity, +infinity).

Self-check: What is sigma(0), and what does it mean for the model's prediction?

Connects to: Binary cross-entropy (5.6); Logit (5.5.6); Logistic regression (5.5.2)

Binary Cross-Entropy Loss

Must-know: Cross-entropy punishes confident wrong predictions far more than hesitant ones. It combines two log terms switched by the label y. With sigmoid, its gradient collapses to the same form as linear regression: (predicted - actual) x feature.

$L (y^, y) = - [y lo g (y^) + (1 - y) lo g (1 - y^)]$

Top pitfall: Using MSE for classification - it is not convex for binary targets and does not exponentially punish confident mistakes.

Self-check: Why is the penalty for a confident wrong prediction (about 4.6) so much larger than for a confident right one (about 0.01)?

Connects to: Sigmoid derivative (5.5.5); SGD update (5.7); Categorical cross-entropy (5.9.8)

Binary Classification with SGD (Worked Example)

Must-know: Train a single sigmoid perceptron one instance at a time. Compute the forward pass z = w transpose x + b, then y-hat = sigma(z). Find the error (y-hat - y) and update each weight by w_j := w_j - eta (y-hat - y) x_j. After training, apply a threshold to pick the class. The default is 0.5, but raise it if predictions cluster near 1.

$w_{j} := w_{j} - η \cdot (y^- y) \cdot x_{j}$

Top pitfall: Using eta = 0.5 in real training. The classroom demo uses it only to make weight changes visible. A rate that large makes weights oscillate or diverge. Use 0.01 or smaller.

Self-check: After initializing weights to zero, what is the first prediction for every input, and why?

Connects to: Binary cross-entropy (5.6); Gradient descent variants (5.2); Threshold tuning (5.8)

Evaluation Metrics for Classification

Must-know: Accuracy can mislead on imbalanced data. Build a 2x2 confusion matrix (TP, TN, FP, FN) and derive precision = TP/(TP+FP), recall = TP/(TP+FN), and F1 = harmonic mean of the two. Prefer F1 when classes are imbalanced.

$Accuracy = \frac{T P + T N}{T P + T N + F P + F N}, F_{1} = 2 \frac{P \cdot R}{P + R}$

Top pitfall: Reporting only accuracy on a 99% 'not-spam' dataset - a model that always says 'not-spam' scores 99% yet catches zero spam.

Self-check: Given TP=1, FN=1, FP=2, TN=2, compute accuracy, precision, recall, and F1.

Connects to: Confusion matrix (5.8.3); Class imbalance (5.8.8); Multi-class evaluation (5.11)

Multi-Class Classification and Softmax

Must-know: For K mutually exclusive classes use K output neurons with a softmax activation that turns the scores into a probability distribution. One-hot encode labels and minimize categorical cross-entropy. A single neuron cannot separate more than two regions, so you need K perceptrons.

$softmax (z_{k}) = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}$

Top pitfall: Trying to solve multi-class with one neuron - it draws at most one linear decision boundary (two regions); K classes need K perceptrons.

Self-check: After softmax, what property do the K outputs always satisfy, and why does that matter?

Connects to: One-hot encoding (5.9.7); Categorical cross-entropy (5.9.8); Multi-class example (5.10)

Multi-Class Worked Example (Mini-Batch)

Must-know: Train a 3-class softmax network in batches. Do the forward pass, apply softmax row-wise, and compute the error as prediction minus the one-hot label. Average the gradient over the batch (divide by B), then update. Shuffle between epochs. The same (y-hat - y) times feature gradient carries over from binary classification.

$\frac{\partial L}{\partial w _{j}} = \frac{1}{B} i = 1 \sum B (y^^{(i)} - y^{(i)}) \cdot x_{j}^{(i)}$

Top pitfall: Forgetting to one-hot encode the labels - categorical cross-entropy needs a probability target vector, not a raw class integer.

Self-check: In a batch of size B, how do you combine the per-instance errors into one weight update?

Connects to: Softmax (5.9); Mini-batch GD (5.3); Categorical cross-entropy (5.9.8)

Multi-Class Evaluation and Confusion Matrix

Must-know: Extend the confusion matrix to a KxK grid; the diagonal holds correct predictions. Compute precision, recall, and F1 per class (precision uses the column sum, recall uses the row sum), then average for an overall score. Decision boundaries partition the feature space into K regions.

$Precision_{k} = \frac{T P _{k}}{\sum _{j} C _{j k}}, Recall_{k} = \frac{T P _{k}}{\sum _{j} C _{k j}}$

Top pitfall: Reporting only the macro-average and missing that one class may have near-zero recall while the average still looks fine.

Self-check: In a 3-class matrix, what do the off-diagonal entries in row 2 represent?

Connects to: Confusion matrix (5.8.3); Per-class metrics (5.11.2); Softmax (5.9)

Previous LectureLecture 04 · Linear Neural Networks for Regression

Next LectureLecture 06 · Deep Feedforward Neural Networks

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.