Deep Neural Networks

Linear Neural Networks for Regression

📅 Published: 2026-07-13

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Deep Neural Networks and Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

The Perceptron and its weighted-sum operation — covered in Lecture 2 (§2.4)
Thresholding and activation functions — covered in Lecture 2 (§2.4.5)
Hyperparameters — covered in Lecture 2 (§2.5.4)

Linear Neural Networks for Regression

4.1 Linear Regression — Supervised Learning Recap

Hook: Can you guess the price of a house just by knowing its size, age, and number of rooms? What if you could do it automatically, with a single equation that learns from past sales data? That is what linear regression does — and it is where all of deep learning begins.

4.1.1 Definition and Explanation

Intuition + Analogy: Think of a recipe. You want to make a cake. The recipe says: 2 cups of flour, 1 cup of sugar, 3 eggs. The cake's sweetness depends on how much of each ingredient you add. Linear regression works the same way. Each input feature (size, rooms, location) is an ingredient. Each has a weight — how much that ingredient matters. The output (price) is the weighted sum of all ingredients plus a baseline. More size? Higher price. Worse location? Lower price. The model learns exactly how many "units of price" each feature contributes.

But the recipe analogy breaks here: in baking, the recipe is fixed. In regression, the machine figures out the recipe by looking at hundreds of past cakes (houses) and their results (prices).

A linear regression is a supervised learning activity that predicts a numerical value. The output is a continuous-valued attribute — a real number on an unbroken scale. Unlike classification, the output is not a binary zero or one. It is not a class label like "high price" or "low price."

Common examples:

House price prediction: the output is a price in currency units.
Temperature forecasting: the output is a temperature reading.
Stock price prediction: the output is a price.
Crop yield prediction: the output is a yield quantity as a function of rainfall, fertilizers, temperature, and other environmental parameters.

The key takeaway: no matter what input features you feed in, the required output is a single continuous value.

Formalize: The linear regression model has a simple mathematical form. You take each input feature $x_{j}$ , multiply it by a learned weight $w_{j}$ , sum them all up, and add a bias $b$ :

$y^= w_{1} x_{1} + w_{2} x_{2} + \dots + w_{d} x_{d} + b$

Every symbol: $y^$ ("y-hat") is the predicted value. Each $x_{j}$ is a feature (e.g., size in square feet). Each $w_{j}$ is a weight — how much feature $j$ influences the prediction. $b$ is the bias — the base value when all features are zero. Even if a house with zero square feet makes no physical sense, the bias lets the line fit data that does not pass through the origin.

In compact vector form, with weight vector $w \in R^{d}$ and feature vector $x \in R^{d}$ :

$y^= w^{⊤} x + b$

Where $w^{⊤} x = \sum_{j = 1}^{d} w_{j} x_{j}$ is the dot product of weights and features.

The professor's example with numbers: a house price model might look like:

$Y = 50 + 0.5 \times size + 1.5 \times location + 0.25 \times rooms$

Here, 50 is the bias (base price). Each square foot adds 0.5 currency units. A one-point better location adds 1.5. Each extra room adds 0.25. These weights are what the machine learns from training data. (The enrichment docs confirm this same weighted-sum form — see Eq. 3.1.1 in the reference text: $price = w_{area} \cdot area + w_{age} \cdot age + b$ .)

What if you need two outputs? Suppose you want to predict both the sale price of a house and its rental income. In traditional machine learning, you build two separate regression models — one for each target. Each model takes all input features but emits a different output. This works but means building and maintaining multiple models.

What if the underlying pattern is nonlinear? A linear model has limited expressive power. If the true relationship needs polynomial terms (e.g., $size^{2}$ ), a simple linear model will underfit. In traditional ML, you would manually engineer polynomial features and retrain. But with a multilayer perceptron, the network learns these nonlinearities automatically — no manual feature engineering needed.

Worked Example: You want to predict the rent of an apartment using two features: area (in m²) and distance to city center (in km). After training on 100 apartment listings, the model learns:

$y^= 200 + 15 \times area - 30 \times distance$

A 50 m² apartment 3 km from the center: $y^= 200 + 15 \times 50 - 30 \times 3 = 200 + 750 - 90 = 860$ . The predicted rent is 860 currency units. Sense-check: bigger apartments cost more (positive weight on area), and apartments farther from the center cost less (negative weight on distance). Both make intuitive sense.

Assumptions & Scope: Linear regression assumes the relationship between features and the target is roughly linear — that the expected value of $Y$ given $x$ can be written as a weighted sum. When this holds, linear regression is fast, interpretable, and needs little data. When it does not hold (e.g., the true pattern is a sine wave or involves feature interactions like "size × location"), the model will underfit no matter how much data you give it. It also assumes that observation noise is well-behaved (usually Gaussian). If your data has extreme outliers, predictions can be pulled off course.

Visual Intuition: Imagine a scatter plot with house size on the x-axis and price on the y-axis. Each dot is one sold house. Linear regression draws one straight line through the dots. The line's y-intercept is the bias $b$ . The line's slope is the weight $w$ . The vertical distance from each dot to the line is the prediction error. The goal of training is to find the slope and intercept that make the average squared distance as small as possible. With two features, the line becomes a flat plane in 3D. With more features, it becomes a hyperplane in high-dimensional space — impossible to draw, but the same idea.

Pitfalls:

Forgetting the bias term. Without $b$ , the line is forced through the origin. If the true relationship has a nonzero intercept, the model will be systematically wrong — every prediction will be too low or too high.
Thinking linear models can learn curves. They cannot. A single linear neuron draws a straight line (or plane). If your data forms a U-shape, the model will miss the bend. You need hidden layers and nonlinear activations to fix this.
Ignoring feature scale. If one feature is in thousands (square feet) and another in single digits (bedrooms), the large-scale feature can dominate. Always standardize your features before training.

4.1.2 Student Questions and Answers

Q: Can a single machine learning model predict two outputs (e.g., both house price and rental income)? A: In traditional machine learning, you would build two separate regression models — one for each target. But as you move toward deeper architectures, a single network can have multiple output nodes. Each output node predicts a different continuous value. The network learns shared features in its hidden layers and branches at the output, so you get two predictions from one model.

Q: Is a linear model enough for complex patterns? A: No. If the data is not linearly separable, a linear model fails. In traditional ML, you increase polynomial degree until the bias-variance trade-off is balanced. With multilayer perceptrons, the network learns nonlinearity from data without re-engineering features. Several students asked variations of this question — it is a fundamental point worth remembering.

Exam note: Single-layer perceptrons can only model linear relationships. Multilayer perceptrons bring nonlinearity. This limitation is what motivates the entire deep learning field.

Recap + Bridge: Linear regression predicts a continuous number as a weighted sum of input features plus a bias. It is the simplest neural network — one neuron, no hidden layers. Next, we zoom into that single neuron and look at exactly how it computes its output.

Real-World & Domain Connection: Linear regression powers home valuation engines like Zillow's Zestimate, which uses hundreds of features (square footage, bedrooms, location scores, recent sales) to compute a weighted-sum estimate of market value. It is also the backbone of hedonic pricing models in economics, where the price of a good is decomposed into the implicit prices of its attributes. In finance, the Capital Asset Pricing Model (CAPM) uses linear regression to relate a stock's returns to market returns. Because the model is transparent — you can point to each weight and say "this is how much this feature matters" — it remains the first tool any analyst reaches for when they need a prediction they can explain to a non-technical stakeholder.

4.2 Single Perceptron — Core Operation Review

Hook: Imagine a tiny decision-maker inside your computer. You hand it a list of numbers. It weighs each one differently, adds them up, and decides what single number to shout back. That decision-maker is a perceptron — the atom of every neural network you will ever build.

4.2.1 Definition and Explanation

Intuition + Analogy: Picture a hiring manager reading a resume. They see GPA (3.8), years of experience (5), and number of programming languages (4). Each factor matters differently — maybe GPA counts for 20%, experience for 50%, and languages for 30%. The manager mentally computes a weighted score: $(0.2 \times 3.8) + (0.5 \times 5) + (0.3 \times 4)$ . That score is the weighted sum. Then the manager asks: "Is the score above 70? If yes, call for interview." That threshold decision is classification.

A perceptron works exactly the same way. Step 1: compute the weighted sum of inputs. Step 2: pass it through a decision function. For a hiring decision, the function is binary (yes/no). For a salary offer, the function just passes the number through — because salary is a continuous value.

Where the analogy breaks: the hiring manager uses fixed weights from a company policy. A perceptron learns its weights from past data, adjusting them every time it sees a new example and compares its prediction to reality.

A single perceptron (artificial neuron) performs two sequential steps:

Weighted sum: It computes the sum of all input features multiplied by their associated weights. With a bias term $W_{0}$ and feature $X_{0} = 1$ (the bias "input"), the expression is:

$Z = W_{0} + W_{1} X_{1} + W_{2} X_{2} + \dots + W_{d} X_{d} = i = 0 \sum d W_{i} X_{i}$

Every symbol: $Z$ is the net input (a scalar). $X_{i}$ is feature $i$ (for $i \geq 1$ ) or the constant 1 (for $i = 0$ , the bias slot). $W_{i}$ is the weight for feature $i$ . $W_{0}$ is the bias — it shifts the decision boundary. $d$ is the number of actual features.

Formalize: In vector form with $W \in R^{d + 1}$ (weights including bias) and $X \in R^{d + 1}$ (features with a prepended 1 for bias):

$Z = W^{⊤} X$

The enrichment reference text (Eq. 3.1.2) uses the same structure: $y^= w_{1} x_{1} + \dots + w_{d} x_{d} + b$ . The notation differs — the lecture uses $W_{0}$ for bias while the text uses $b$ — but the math is identical. In both representations, this is a dot product between a weight vector and a feature vector.

Transformation: The sum $Z$ passes through a function that decides what signal to emit. This function is called an activation function.

Classification vs. Regression — the key fork:

In a classification context, step 2 was thresholding. If the sum exceeds a threshold, emit 1 (fire). Otherwise, emit 0 (don't fire). This produces binary output — a discrete yes/no decision.

For regression, thresholding is not needed. The output $Y$ must be a continuous real number. A house price can be 105.5 — not just 0 or 1. If you threshold, you squash the continuous nature of the output into a binary choice, which is exactly wrong for regression.

But you may still need a transformation. House prices cannot be negative. If the weighted sum produces $- 105.5$ , that output is meaningless in the real world. So you clip negative values to zero while keeping positive values unchanged. This clipping is one form of transformation — the activation function shapes the output into a usable range.

Key distinction — commit this to memory:

Thresholding compares the sum against a cutoff (e.g., 0.5) and emits a binary signal. Used for classification.
Activation functions reshape values continuously — clipping, scaling, bounding — without forcing a discrete decision. Used for regression.

This distinction mirrors the biology the neuron model was inspired by. Real neurons in your brain receive chemical signals at dendrites (the inputs $X_{i}$ ), weight them by synaptic strengths (the weights $W_{i}$ ), aggregate them in the cell body (the sum $Z$ ), and fire an electrical spike down the axon only if the total exceeds a threshold. The artificial perceptron abstracts this: the weighted sum is the aggregation, and the activation function is the firing rule.

Worked Example: A single perceptron has two features: square footage ( $X_{1} = 1500$ ) and number of bedrooms ( $X_{2} = 3$ ). Learned weights: $W_{0} = 50$ (bias), $W_{1} = 0.2$ , $W_{2} = 10$ .

Step 1 — weighted sum: $Z = 50 + 0.2 \times 1500 + 10 \times 3 = 50 + 300 + 30 = 380$ .

Step 2 — for regression (identity activation): output = 380. Predicted price = 380,000 currency units.

If this were classification (threshold at 300): 380 > 300, so output = 1. Decision: "expensive house." Same perceptron, different activation function — completely different task.

Assumptions & Scope: The perceptron model assumes each feature contributes independently and linearly to the output. There are no interaction terms (like $X_{1} \times X_{2}$ ) built in. It assumes the bias term captures any constant offset. This works well when the data is roughly linear. It breaks when features interact (e.g., an extra bedroom matters more in a large house than a small one) or when the relationship curves. It also assumes all input features are numeric — categorical features must be encoded first.

Visual Intuition: Picture a neuron as a funnel. Many input streams (features $X_{1}, X_{2}, \dots, X_{d}$ ) pour in from the left. Each stream passes through a valve (weight $W_{i}$ ) that controls its flow rate — a strong weight opens the valve wide, a near-zero weight nearly shuts it. All streams collect in a central chamber (the sum $Z$ ). At the chamber's exit sits a gate (the activation function). For classification, the gate is a rigid barrier that only opens when the water level exceeds a mark. For regression, the gate stays fully open — whatever flows in flows out. For non-negative regression (like house prices), a one-way flap prevents backflow — negative values are blocked, positives pass freely. This one-way flap is the ReLU function.

Pitfalls:

Using thresholding for regression. This is the most common conceptual mistake. If your output needs to be a continuous number, do not threshold — it destroys the very information you are trying to predict.
Forgetting that bias is just another weight. $W_{0}$ is not special — it is a weight connected to a constant input of 1. Treating it the same as other weights simplifies both the math and the code.
Confusing the weighted sum with the final output. $Z$ is the raw internal signal. The output $Y^$ is $Z$ after the activation function. For the identity activation they are equal, but conceptually they are distinct steps.

Q: What is the difference between thresholding and transformation? A: Thresholding compares the sum against a cutoff (e.g., 0.5) and emits a binary signal — it is used for classification. Transformation via an activation function produces a continuous output appropriate for regression. Thresholding gives you zeros or ones; activation gives you a continuous range. Think of a dimmer switch (transformation — any brightness) versus an on/off switch (thresholding — only full light or complete darkness).

Recap + Bridge: A perceptron does two things: compute a weighted sum, then pass it through an activation function. For classification, the activation is a threshold. For regression, it is continuous — often just the identity (pass-through). Next, we compare this neural framing of linear regression against the traditional statistical approach and see why the neural perspective wins when you want to scale up.

Real-World & Domain Connection: The perceptron was invented in 1958 by Frank Rosenblatt at the Cornell Aeronautical Laboratory. The Mark I Perceptron was a physical machine — a room-sized array of 400 photocells connected to potentiometers (adjustable resistors representing weights) driving electric motors that physically turned the dials to adjust weights during learning. Today, that same weighted-sum-plus-activation pattern runs trillions of times per second in every GPU training a modern neural network. The perceptron's mathematical simplicity — a dot product followed by a nonlinearity — is exactly why it scales so well: you can compute millions of them in parallel on a single chip. In industry, every "fully connected layer" in a production model (recommendation systems at YouTube, ranking at Google Search, feed ranking at Instagram) is just a collection of perceptrons, each running the same two-step operation you just learned.

4.3 Linear Neural Networks vs Traditional Linear Regression

Hook: Same equation, two different worlds. A statistician and a deep learning engineer both write $Y = b + \sum w_{i} x_{i}$ . The statistician solves it with a closed-form formula and checks p-values. The engineer runs it through gradient descent and stacks more layers on top. Which approach is better? It depends on what you want to do next.

4.3.1 Definition and Explanation

Intuition + Analogy: Think of building a house. Traditional linear regression is like a master carpenter who carefully measures, cuts, and fits every beam using precise geometric formulas. The result is a perfect one-room cabin — optimal, provably correct, no wasted material. A linear neural network is the same one-room cabin, but built by a general contractor who uses adjustable scaffolding (gradient descent) instead of a formula. The cabin is identical. But when you want to add a second story (hidden layers), the carpenter needs a completely new blueprint. The contractor just extends the scaffolding upward.

The analogy breaks here: in construction, the contractor's approach is messier and potentially less precise. In machine learning, gradient descent converges to the same optimal weights (for convex problems like linear regression), so you lose nothing in quality while gaining everything in extensibility.

The mathematical model is identical in both paradigms:

$Y = W_{0} + W_{1} X_{1} + W_{2} X_{2} + \dots + W_{d} X_{d} = W_{0} + i = 1 \sum d W_{i} X_{i}$

The difference is in how the model is framed, trained, and extended.

Formalize — Two framings of the same model:

Traditional statistical framing:

Solve using the normal equation (closed form): $W^{*} = (X^{⊤} X)^{- 1} X^{⊤} Y$
This requires $X^{⊤} X$ to be invertible (features must be linearly independent)
Training is a one-step matrix computation — no iteration
Evaluation uses p-values, confidence intervals, $R^{2}$ , F-tests
Adding nonlinearity means choosing a new model class (polynomial regression, splines, GAMs)

Neural network framing:

Solve using gradient descent: $W^{new} = W^{old} - η \nabla_{W} J$
No invertibility requirement — works even with redundant features
Training is iterative — weights improve step by step
Evaluation uses loss curves, train/test splits
Adding nonlinearity means adding a layer — the training loop stays the same

The enrichment reference text (Section 3.1.4) shows linear regression explicitly as a neural network diagram (Fig. 3.1.2): input neurons $x_{1}, \dots, x_{d}$ all connected directly to a single output neuron $o_{1}$ . The text calls it "a single-layer fully connected neural network." Both framings produce the same prediction function. The strategic difference is architectural: the neural network framing is designed to be extended.

Comparison — traditional vs. neural network framing:

Dimension	Traditional ML (Statistics)	Neural Network
Training method	Closed-form normal equation	Iterative gradient descent
Requires invertible matrix?	Yes ( $X^{⊤} X$ full rank)	No
Handles redundancy?	Fails or needs regularization	Learns fine through SGD
Adding nonlinearity	New model class, new math	Add a hidden layer
Interpretability	High (p-values, confidence intervals)	Lower (weights only)
Scales to deep architectures?	No	Yes (the whole point)
Best for	Small datasets, inference, explanation	Large datasets, prediction, stacking

When to pick which: Use traditional linear regression when you need statistical inference (p-values, confidence intervals, hypothesis tests) and your dataset is small. Use the neural network framing when you plan to extend to deep architectures or when your dataset is too large for matrix inversion.

Worked Example: A researcher studies the effect of fertilizer on crop yield with 30 data points. She uses traditional linear regression — computes p-values, reports that fertilizer is significant (p < 0.01), and publishes the regression table. The same researcher later builds a crop yield prediction system for a farming app with 500,000 data points and 200 features. She frames it as a neural network, trains via minibatch SGD, and later adds two hidden layers to capture interaction effects — all without rewriting the training loop. Same math at the core, but the framing determines the ceiling.

Assumptions & Scope: The equivalence between the two framings holds exactly only for linear models with convex loss functions (like MSE). When you stack layers with nonlinear activations, the loss surface becomes non-convex and no closed-form solution exists — gradient descent is then your only option. The traditional framing's advantage (closed-form solution) disappears the moment you go beyond one layer.

Visual Intuition: Picture two roads leading to the same village. The traditional road is a short, straight highway — one fast trip, but it dead-ends at the village. The neural network road is a winding path that also reaches the village, but it continues onward into the mountains. Both roads get you to linear regression. Only one road takes you to deep learning.

Pitfalls:

Confusing "linear neural network" with "deep neural network." A linear neural network has zero hidden layers and a linear (identity) activation. It is linear regression in neural clothing. Do not expect it to learn curves.
Thinking traditional ML is obsolete. For small datasets and inference-heavy problems, the closed-form solution is faster, more interpretable, and gives you uncertainty estimates that gradient descent does not.
Underestimating the value of the neural framing. The entire deep learning revolution happened because researchers kept the same training paradigm — forward pass, loss, backward pass, weight update — and just added more layers. The framing unlocked the architecture.

Recap + Bridge: Linear neural networks and traditional linear regression use the same equation. The difference is the training method and the extensibility roadmap. With the neural framing in place, we can now turn to the component that makes a single neuron produce a regression output instead of a classification decision: the activation function.

Real-World & Domain Connection: In industry, data scientists frequently start with scikit-learn's LinearRegression for quick baselines and inference, then migrate to PyTorch/TensorFlow linear layers when the project grows to need neural architectures. This two-phase approach — classical baseline, then neural extension — is standard practice at companies like Stripe (fraud detection starting from logistic regression before moving to deep models) and Spotify (recommendations starting from matrix factorization before adding neural collaborative filtering). The fact that you can express a linear model as nn.Linear(d, 1) in PyTorch and then stack more layers on top without changing your data pipeline or loss function is what makes the neural framing so practical.

4.4 Activation Functions

Hook: A perceptron without an activation function is just a fancy calculator doing weighted sums. The activation function is what gives it a purpose. It decides: should the output be any number, only positive numbers, or a probability between zero and one? Choose the wrong activation, and your model produces nonsense. Choose the right one, and everything just works.

4.4.1 Identity Function (Linear Activation)

#### Definition

The identity function does nothing to its input — it emits exactly what it receives: $f (z) = z$ . Plotting $f (z)$ against $z$ gives a straight diagonal line through the origin with slope 1.

Intuition + Analogy: Imagine a transparent glass tube. You pour water in at one end — exactly the same water comes out the other end. No change, no filtering, no restriction. That is the identity function: whatever goes in comes out unchanged. It is the "do nothing" function. For regression, this is exactly what you want — the weighted sum is already the number you want to predict. You do not need to bend it or squeeze it into a range.

Where the analogy breaks: water can flow both ways through a tube. The identity function handles negative inputs just fine — it passes them through unchanged. That can be a problem if your output should never be negative (like a price). Then you need a one-way valve — and that is ReLU.

Formal definition:

$f (z) = z$

The identity function emits exactly what it receives. If $z = 42$ , $f (z) = 42$ . If $z = - 15.3$ , $f (z) = - 15.3$ . Plotting $f (z)$ against $z$ gives a straight diagonal line through the origin with slope 1.

#### Why identity is enough for regression

Formalize: In regression, the raw weighted sum $Z = \sum W_{i} X_{i}$ can be any real value in $(- \infty, \infty)$ . The target output $Y$ is also any real value. There is no mismatch. The identity function preserves the full real-valued range. After computing $Y^= W^{⊤} X$ , you take $Y^$ as the final output directly — no transformation needed.

If the model learns that $W^{⊤} X = 105.5$ , then that is the prediction. If it learns $- 30.2$ , that is the prediction too. The model is free to output any real number because the identity function imposes no restriction.

#### Derivative of the identity function

The gradient (derivative) of $f (z) = z$ is:

$\frac{df}{d z} = 1$

This single fact has enormous practical significance during backpropagation. The gradient is 1 everywhere. It does not shrink, saturate, or die. When the gradient flows backward through an identity activation, it passes through unchanged — every downstream weight gets the full gradient signal. This is why identity is numerically well-behaved for gradient descent.

Contrast this with thresholding: the threshold function is non-differentiable at the threshold point and has a zero gradient everywhere else. You cannot do gradient descent through a threshold. The identity function has no such pathology.

#### When identity alone is not enough

If your application forbids negative outputs — house prices, counts, durations — then raw identity is too permissive. It will happily emit $- 50000$ for a house price prediction. That output is meaningless. You need a function that blocks negatives while passing positives unchanged.

4.4.2 ReLU — Rectified Linear Unit

Formalize: The function that clips negatives to zero and passes positives unchanged is the Rectified Linear Unit:

$ReLU (x) = max (0, x) = {0 x if x < 0 if x \geq 0$

When $x$ is negative, the output is exactly 0. When $x$ is zero or positive, the output equals $x$ .

Derivative of ReLU:

$\frac{d}{d x} ReLU (x) = {01 if x < 0 if x > 0$

At exactly $x = 0$ , the derivative is technically undefined, but in practice frameworks assign it 0 or 1 — the difference is negligible because the probability of hitting exactly zero with floating-point numbers is tiny.

ReLU is one of the most commonly used activation functions in all of deep learning — not just at output layers, but in hidden layers too. Three reasons it dominates:

Cheap to compute. $max (0, x)$ is a single comparison — no exponentials, no divisions.
Simple gradient. 0 or 1 — no vanishing gradient for positive activations.
Introduces nonlinearity. The kink at zero is enough to let stacked ReLU layers approximate any continuous function (this is the universal approximation theorem in action).

Worked Example: A perceptron computes $Z = - 42$ (weighted sum). Pass it through identity: output = $- 42$ . Pass it through ReLU: output = $0$ .

Another perceptron computes $Z = 38.7$ . Identity: output = $38.7$ . ReLU: output = $38.7$ .

Now consider a model predicting house prices where negative outputs make no sense. With identity, a badly-initialized model might predict a price of $- 1000$ . With ReLU at the output, the same model predicts 0 — still wrong, but at least not absurd. As training progresses and weights adjust to push the weighted sum positive, ReLU's output rises above zero and behaves identically to the identity function for the valid range. Final prediction after convergence: a positive price.

Q: What activation function clips negatives and passes positives?

A: ReLU — the Rectified Linear Unit. $ReLU (x) = max (0, x)$ . This was assigned as a homework identification task.

4.4.3 Symbol Registry — Activation Functions

$z$ — net input to the neuron (weighted sum). Defined as $z = \sum W_{i} X_{i}$ . Scalar in $(- \infty, \infty)$ .
$f (z)$ — activation function output. Scalar. The exact domain depends on which function is used.
Identity: $f (z) = z$ . Output in $(- \infty, \infty)$ .
ReLU: $f (x) = max (0, x)$ . Output in $[0, \infty)$ .

4.4.4 Choosing Activation Functions for Output Layers

The activation function at the output node must match the problem type. This is not a hyperparameter you tune by trial and error — it is determined by what your output needs to be:

Problem type	Output range needed	Activation	Why
Regression (unbounded)	$(- \infty, \infty)$	Identity ( $f (z) = z$ )	Any real value is valid
Regression (non-negative)	$[0, \infty)$	ReLU	Blocks negatives, passes positives
Binary classification	$[0, 1]$	Sigmoid	Output interpreted as probability
Multi-class classification	Probability distribution over $K$ classes	Softmax	Outputs sum to 1

By default, a linear neural network for regression uses identity. You modify it only if the application demands a bounded output range.

Assumptions & Scope: The activation function at the output layer assumes the model's internal computation (the weighted sum) produces values in a range the activation can map usefully. If your weighted sum is consistently $- 1000$ but you use ReLU, the model outputs 0 forever — no learning happens because the gradient for negative inputs is 0. This is the dying ReLU problem. For output layers with ReLU, initialize weights so the initial weighted sum lands in the positive region. For hidden layers, dying ReLU is mitigated by variants like Leaky ReLU, which you will encounter in later modules.

Scope — hidden vs. output activations: The discussion above focuses on the output layer's activation. Hidden layers use activations for a different purpose: introducing nonlinearity so the network can learn curved decision boundaries. Hidden layer activations (ReLU, tanh, sigmoid, GELU) are chosen for gradient flow and expressiveness, not for output range. A regression network might use ReLU hidden layers with an identity output — this is the standard architecture.

Visual Intuition: Draw a standard x-y coordinate plane. For identity, draw a straight diagonal line from bottom-left $(- 5, - 5)$ through the origin $(0, 0)$ to top-right $(5, 5)$ . Label this "Identity — anything goes."

For ReLU, draw the same line from $(0, 0)$ to $(5, 5)$ , but from $(- 5, 0)$ to $(0, 0)$ draw a flat horizontal line at y = 0. There is a sharp corner (the "kink") at $(0, 0)$ . Everything to the left of zero is flattened. Everything to the right rises normally. The takeaway: identity lets the full signal through, ReLU blocks the negative half.

Pitfalls:

Using sigmoid/tanh at the output for regression. These functions squash outputs into $(0, 1)$ or $(- 1, 1)$ . If your house prices range from 100K to 2M, a sigmoid output can never reach those values. The model will saturate and stop learning.
Forgetting that ReLU has a zero-gradient region. For $x < 0$ , the gradient of ReLU is exactly 0. If your weighted sum goes negative and stays negative, that neuron receives no gradient signal and its weights never update. This is especially dangerous at the output layer where there is no downstream neuron to provide an alternative gradient path.
Treating the activation as separate from the loss. The activation function and the loss function are a pair. Identity output + MSE loss is the standard for unbounded regression. If you change one, re-evaluate the other. Sigmoid output + MSE leads to slow convergence (the vanishing gradient problem) — sigmoid pairs better with cross-entropy.

Recap + Bridge: The identity function passes the weighted sum through unchanged — perfect for unbounded regression. ReLU clips negatives to zero — right for non-negative outputs. Choose the activation to match the output range your problem demands. Now that we know what the model emits, we need a way to measure how wrong it is. That is the job of the objective function.

Real-World & Domain Connection: ReLU was introduced by Nair and Hinton in 2010 and popularized by Krizhevsky et al. in the 2012 AlexNet paper that revolutionized computer vision. Before ReLU, deep networks used sigmoid or tanh activations and struggled with vanishing gradients — gradients would shrink exponentially as they propagated backward, making deep layers nearly untrainable. ReLU's constant gradient of 1 in the positive region solved this, enabling networks with dozens or hundreds of layers. Today, ReLU and its descendants (Leaky ReLU, Parametric ReLU, GELU, Swish) are the default hidden-layer activations in every major framework. GELU (Gaussian Error Linear Unit) is used in modern transformers like GPT and BERT. All trace their lineage back to this simple idea: clip the negatives, pass the positives, keep the gradient alive.

4.5 Objective Functions — Loss and Cost

Hook: You trained a model. It predicted a house price of 350K. The actual price was 400K. How wrong is the model? "By 50K" is one answer. "By 50K squared, then averaged over all houses" is another. These are loss functions — the tape measure that tells your model how badly it messed up. Pick the wrong tape measure, and your model learns the wrong thing.

4.5.1 Mean Square Error (MSE)

Intuition + Analogy: Imagine you are throwing darts at a dartboard. Each dart is a prediction, and the bullseye is the true value. MSE measures the average squared distance from each dart to the bullseye. If you miss by 2 cm, that counts as an error of 4 (2²). If you miss by 10 cm, that counts as 100 (10²). A single wild dart that hits the wall inflates the score massively. MSE screams at the model: "Fix your biggest mistakes first!"

Contrast this with MAE: it counts errors linearly. A 10 cm miss is simply 5 times worse than a 2 cm miss, not 25 times worse. MAE whispers: "All errors matter equally."

Where the analogy breaks: in darts, you care about your average performance. In machine learning, MSE's squaring has a mathematical superpower — it makes the derivative clean and the optimization convex (a single bowl-shaped surface). The 1/2 factor is there purely for that mathematical convenience.

The most common loss function for regression is the Mean Square Error (MSE). Given $N$ training instances, predicted values $Y^_{i}$ , and actual values $Y_{i}$ :

$MSE = \frac{1}{2 N} i = 1 \sum N (Y^_{i} - Y_{i})^{2}$

Formalize — step by step:

For each instance $i$ , compute the residual (error): $e_{i} = Y^_{i} - Y_{i}$ — the difference between predicted and actual.
Square it: $e_{i}^{2}$ — this makes all errors positive and penalizes large errors disproportionately.
Sum across all $N$ instances: $\sum e_{i}^{2}$ .
Divide by $2 N$ : $\frac{1}{2 N} \sum e_{i}^{2}$ .

Why the $\frac{1}{2}$ factor? The enrichment reference text (Eq. 3.1.5–3.1.6) confirms this convention. The $\frac{1}{2}$ serves a single purpose: mathematical elegance during differentiation. When you take the derivative of $(Y^_{i} - Y_{i})^{2}$ , the power rule brings down a factor of 2: $\frac{d}{d Y ^} (Y^- Y)^{2} = 2 (Y^- Y)$ . The $\frac{1}{2}$ cancels it, leaving a clean $\frac{1}{N} \sum (Y^_{i} - Y_{i})$ . Some textbooks omit the $\frac{1}{2}$ and write $\frac{1}{N} \sum (Y^_{i} - Y_{i})^{2}$ . Both are correct. The professor uses the $\frac{1}{2 N}$ form — use whichever your course specifies.

Why MSE is the preferred choice:

Smooth and differentiable everywhere. You can compute the gradient at any point.
Convex. The cost surface is a bowl with exactly one global minimum — no confusing local minima to trap the optimizer.
Emphasizes large errors. Squaring amplifies outliers. If a single instance has a large error, MSE pulls the model hard toward fixing it.
Simple gradient. The derivative is a linear function of the error, which makes weight updates computationally cheap.

Connection to maximum likelihood: The enrichment text (Section 3.1.3) shows that minimizing MSE is equivalent to maximum likelihood estimation under the assumption that the noise (the difference between the model and reality) follows a Gaussian (normal) distribution with mean zero. If you assume $Y = w^{⊤} x + b + ϵ$ where $ϵ \sim N (0, σ^{2})$ , then the negative log-likelihood simplifies to MSE (up to a constant). This probabilistic foundation justifies MSE beyond "it just works."

Worked Example: Three houses. Model predicts: $Y^= [200, 350, 500]$ . Actual prices: $Y = [210, 340, 510]$ .

Errors: $e = [200 - 210, 350 - 340, 500 - 510] = [- 10, 10, - 10]$ .

Squared errors: $e^{2} = [100, 100, 100]$ .

Sum = 300. $N = 3$ .

MSE (with 1/2): $\frac{1}{2 \times 3} \times 300 = \frac{300}{6} = 50$ .

MSE (without 1/2): $\frac{1}{3} \times 300 = 100$ .

The model is off by about 10 units per house on average. Sense-check: both forms decrease as predictions get closer to actuals. The choice of convention affects the numerical value but not the optimal weights.

4.5.2 Mean Absolute Error (MAE)

Formalize: MAE takes the absolute value of each residual and averages them:

$MAE = \frac{1}{N} i = 1 \sum N ∣ Y^_{i} - Y_{i} ∣$

Every symbol: $Y^_{i}$ is the predicted value. $Y_{i}$ is the actual value. $∣ \cdot ∣$ is the absolute value (distance from zero, always non-negative). $N$ is the number of instances.

MAE penalizes errors linearly. A residual of 10 contributes 10 to the sum. A residual of 100 contributes 100. There is no squaring, so large errors do not dominate the way they do in MSE.

When to prefer MAE: If your data has outliers — a house priced at 10 million in a neighborhood of 300K houses — MSE would square that enormous error and drag the model's line toward the outlier, degrading predictions for all normal houses. MAE treats the outlier's error proportionally, keeping the model focused on the typical cases.

4.5.3 MSE vs MAE — Comprehensive Comparison

Property	MSE	MAE
Formula	$\frac{1}{2 N} \sum (Y^_{i} - Y_{i})^{2}$	$\frac{1}{N} \sum ∣ Y^_{i} - Y_{i} ∣$
Penalizes large errors	Quadratically (strongly)	Linearly (mildly)
Outlier sensitivity	High — one outlier can dominate	Low — outliers have proportional effect
Differentiable everywhere	Yes — smooth everywhere	No — has a cusp at zero ( $\frac{\partial ∣ x ∣}{\partial x}$ is undefined at $x = 0$ )
Convex	Yes (strictly convex bowl)	Yes (convex but not strictly)
Gradient	$\frac{1}{N} (Y^_{i} - Y_{i})$ — linear in error	$\frac{1}{N} sign (Y^_{i} - Y_{i})$ — constant magnitude
Optimal prediction (theoretical)	Conditional mean $E [Y ∣ X]$	Conditional median
When to use	Standard regression, no extreme outliers	Robust regression, heavy-tailed data

When to pick which: Default to MSE for standard regression problems. Switch to MAE when your data contains outliers you do not want to overfit, or when you need a prediction of the median rather than the mean.

Scope — MAE's non-differentiability: The absolute value function has a sharp corner at zero. Its derivative is undefined at exactly that point. In practice, frameworks either use a subgradient (any value between -1 and 1 at zero) or the Huber loss — a hybrid that behaves like MSE near zero and like MAE for large errors. The Huber loss gives you the best of both worlds: differentiability everywhere and outlier resistance. It is available in PyTorch as nn.HuberLoss and in the enrichment text as Exercise 3.5.6.

4.5.4 Convexity and Why It Matters

Formalize: A function $J (w)$ is convex if, for any two points $w_{a}$ and $w_{b}$ and any $λ \in [0, 1]$ :

$J (λ w_{a} + (1 - λ) w_{b}) \leq λ J (w_{a}) + (1 - λ) J (w_{b})$

In plain language: the line segment connecting any two points on the function lies on or above the function itself. The function curves upward like a bowl — never like a saddle or a wavy surface.

Intuition + Analogy: Drop a marble into a cereal bowl. No matter where you drop it from — the rim, halfway down, near the center — it always rolls to the same lowest point at the bottom. A convex cost function is that bowl. Drop your weights anywhere (random initialization), follow the slope downhill, and you always end up at the global minimum. There are no fake bottoms (local minima) to trap the marble.

Now imagine a wavy potato chip. Drop the marble somewhere, and it might roll into a shallow dip and stop — convinced it found the bottom — while a much deeper dip exists elsewhere. That is a non-convex function. Your optimizer can get stuck.

Why convexity matters for reliable optimization:

Start at any arbitrary weight initialization.
Follow the negative gradient downhill.
You are guaranteed to reach the global minimum.
There is no risk of stopping at a fake bottom because no fake bottoms exist.

MSE is convex for linear regression. This is provable — the Hessian matrix (second derivative) is $X^{⊤} X$ , which is positive semi-definite, the mathematical condition for convexity. Cross-entropy loss for logistic regression is also convex. This is why these loss functions are the default choices.

Worked Example — Convex vs. Non-convex: Consider a model with one weight $w$ . The MSE cost: $J (w) = (w - 3)^{2}$ . This is a parabola opening upward. Its minimum is at $w = 3$ . Gradient descent from any starting point — $w = 100$ , $w = - 50$ — always finds $w = 3$ .

Now consider a non-convex cost: $J (w) = w^{4} - 4 w^{2} + w$ . This has two local minima. Start at $w = - 1$ and gradient descent rolls into the nearest dip at $w \approx - 1.47$ . But the global minimum is at $w \approx 1.45$ . The optimizer never finds it because it got stuck in the wrong bowl. MSE avoids this problem entirely.

Real-world: Deep neural networks have non-convex loss surfaces with many local minima and saddle points. Modern optimizers like Adam (Adaptive Moment Estimation) and AdaProp include momentum and adaptive learning rates to help escape shallow local minima and navigate saddle points. These are covered in a later optimization module (post-midterm).

Assumptions & Scope: Convexity guarantees global convergence only when combined with an appropriate learning rate. If the learning rate is too large, gradient descent can overshoot and diverge even on a convex function. Convexity also assumes the loss is defined as a function of the parameters alone, given fixed training data. If the data distribution shifts (non-IID), these guarantees weaken.

Visual Intuition: Plot cost ( $J$ ) on the y-axis against weight ( $w$ ) on the x-axis. For MSE, you get a single U-shaped parabola. The minimum is the unique bottom. Draw a vertical dashed line at the minimizer $w^{*}$ . Arrow annotations show gradient descent steps — big steps when far from the minimum (steep slope), small steps when close (shallow slope). For a non-convex function, draw a wavy line with two valleys. Point to the shallower valley and label it "local minimum — trap." Point to the deeper valley and label it "global minimum — goal."

Pitfalls:

Assuming all loss functions are convex. Most interesting loss surfaces in deep learning are not. Only linear models with MSE or logistic regression with cross-entropy enjoy this guarantee.
Confusing convexity of the loss with convexity of the model. A neural network can be highly non-convex in its parameters even if the loss function (like MSE) is convex in the predictions.
Setting the learning rate too high on a convex function. A convex bowl still has steep walls. Too large a step and you bounce from one side of the bowl to the other, never settling at the bottom. This is called oscillation or divergence.
Forgetting that "global minimum on training data" does not mean "best model." A model that perfectly minimizes training MSE may overfit and generalize poorly. The global minimum of the training loss is not always the global minimum of the test loss.

Q: Is there a chance of converging to a local minimum instead of the global minimum?

A: Yes, if your cost function is not convex. MSE is convex, so with MSE and a linear model there is no local minimum problem — you are guaranteed to reach the global minimum. But if you design your own non-convex cost function, or if you use a deep network, you can get stuck. Several students asked this question. Advanced optimization techniques like Adam and AdaProp include momentum mechanisms that help the optimizer roll through shallow local minima and escape them. These are covered in the post-midterm optimization module.

Recap + Bridge: MSE squares the errors and averages them — it is convex, differentiable, and the default for regression. MAE uses absolute errors — it is less sensitive to outliers but not differentiable at zero. Both define the "unfitness" of the model. Now that we can measure how wrong our model is, we need a procedure to adjust the weights and reduce that error. That procedure is gradient descent.

Real-World & Domain Connection: In finance, choosing between MSE and MAE has real dollar consequences. A hedge fund building a stock price predictor might use MSE to aggressively penalize large prediction errors — a big miss on a volatile stock could cost millions. A retail chain forecasting daily sales for 10,000 stores might use MAE because a single store's anomalous day (a local festival doubling sales) should not distort the forecast for all other stores. The Huber loss — MSE near zero, MAE for large residuals — is the industry standard in outlier-resistant statistics and is the default in many scikit-learn regressors (SGDRegressor with loss='huber'). In deep learning, PyTorch's SmoothL1Loss is a Huber variant used in object detection (Faster R-CNN) to prevent outlier bounding box errors from dominating the training signal.

4.6 Gradient Descent for Linear Regression

Hook: You know how wrong your model is (the loss). You know which direction reduces the error (the gradient). You could take one giant leap to the mathematically "correct" weights. Why don't you? Because your model has many weights, and each one is shouting a different correction. Take all their advice at full volume, and you overshoot into chaos. Take just a whisper of each, and you glide smoothly to the answer.

4.6.1 Why Gradients? Why Not Direct Adjustment?

Intuition + Analogy: You are steering a ship. Ten crew members surround you, each shouting a different direction based on what they see from their side of the deck. One shouts "two degrees port!" Another shouts "four degrees starboard!" A third yells "one degree port!" Each one's suggestion is correct from their limited viewpoint — their feature's gradient. If you jerk the wheel fully toward each suggestion, the ship zigzags violently and never reaches the harbor. Instead, you turn the wheel just a tiny bit in each suggested direction. Iterate. The crew gradually starts agreeing as you approach the true heading. That is gradient descent with a learning rate.

The learning rate $η$ (eta) controls what fraction of each suggested adjustment you actually apply. A full step would be $η = 1$ — apply the entire gradient. A cautious step might be $η = 0.01$ — apply only 1% of what the gradient demands.

The gradient says "move weight $W_{j}$ by this amount to reduce the error." But you have many weights. $W_{1}$ wants to shift +2. $W_{2}$ wants to shift −4. $W_{3}$ wants to shift +1. Each weight pulls the model in a different direction. If you apply all adjustments fully and simultaneously, the combined effect overshoots or diverges. Taking small, controlled steps — a fraction of each gradient — lets the algorithm converge by balancing competing feature pulls.

4.6.2 The Weight Update Rule

Formalize — step-by-step derivation:

Start with the MSE cost function (using the $\frac{1}{2 N}$ convention from the lecture):

$J (W) = \frac{1}{2 N} i = 1 \sum N (Y^_{i} - Y_{i})^{2}$

where $Y^_{i} = W^{⊤} X_{i} = \sum_{j = 0}^{d} W_{j} X_{ij}$ (with $X_{i 0} = 1$ for the bias).

Step 1: Differentiate $J$ with respect to weight $W_{j}$ . Apply the chain rule:

$\frac{\partial J}{\partial W _{j}} = \frac{1}{2 N} i = 1 \sum N 2 (Y^_{i} - Y_{i}) \cdot \frac{\partial Y ^ _{i}}{\partial W _{j}}$

The factor of 2 from the power rule cancels with the $\frac{1}{2}$ :

$\frac{\partial J}{\partial W _{j}} = \frac{1}{N} i = 1 \sum N (Y^_{i} - Y_{i}) \cdot \frac{\partial Y ^ _{i}}{\partial W _{j}}$

Step 2: Since $Y^_{i} = W_{0} X_{i 0} + W_{1} X_{i 1} + \dots + W_{d} X_{i d}$ , we have $\frac{\partial Y ^ _{i}}{\partial W _{j}} = X_{ij}$ .

Step 3 — Gradient for bias ( $W_{0}$ , where $X_{i 0} = 1$ ):

$\frac{\partial J}{\partial W _{0}} = \frac{1}{N} i = 1 \sum N (Y^_{i} - Y_{i}) \cdot 1 = \frac{1}{N} i = 1 \sum N (Y^_{i} - Y_{i})$

This is the mean prediction error across all instances.

Step 4 — Gradient for feature weight $W_{j}$ ( $j \geq 1$ ):

$\frac{\partial J}{\partial W _{j}} = \frac{1}{N} i = 1 \sum N (Y^_{i} - Y_{i}) \cdot X_{ij}$

This is the mean of the prediction error multiplied by the feature value.

Step 5 — The update rule:

$W_{j}^{new} = W_{j}^{old} - η \cdot \frac{\partial J}{\partial W _{j}}$

Every symbol: $W_{j}^{old}$ is the current weight. $η$ (eta) is the learning rate — a small positive number like 0.01. $\frac{\partial J}{\partial W _{j}}$ is the gradient — how much the cost changes when you nudge $W_{j}$ . The minus sign means "move opposite to the gradient" — downhill.

Vector form (all weights updated simultaneously):

$W^{new} = W^{old} - η \cdot \frac{1}{N} i = 1 \sum N (Y^_{i} - Y_{i}) \cdot X_{i}$

Where $W \in R^{d + 1}$ and $X_{i} \in R^{d + 1}$ (with $X_{i 0} = 1$ ).

Reconciliation with enrichment docs: The reference text (Eq. 3.1.11) gives the minibatch form: $w \leftarrow w - \frac{η}{∣ B ∣} \sum_{i \in B_{t}} x^{(i)} (w^{⊤} x^{(i)} + b - y^{(i)})$ . Setting $∣ B ∣ = N$ (full batch) recovers exactly the professor's formula above. The notation differs — the lecture uses $W_{j}$ and $X_{ij}$ while the text uses $w$ and $x^{(i)}$ — but the gradient expressions are identical.

Q: How does this update formula relate to the perceptron weight update formula from the previous module?

A: It is the same formula. The previous module's delta rule — old weight minus learning rate times a delta component — is exactly what you get from gradient descent on MSE. You take the cost function, differentiate with respect to the weights, and apply gradient descent. The form looks slightly different because of notation, but the underlying derivation is identical.

4.6.3 Batch Gradient Descent

Purpose: Compute the exact gradient using the entire training dataset before each weight update. This gives the most accurate direction toward the minimum.

Inputs & Outputs:

Input: Feature matrix $X \in R^{N \times (d + 1)}$ , target vector $Y \in R^{N}$ , learning rate $η$ , convergence tolerance $ϵ$ .
Output: Learned weight vector $W \in R^{d + 1}$ .

Steps — with rationale:

Initialize all weights to some starting value (e.g., all zeros). Rationale: the algorithm needs a starting point — any will do for convex problems.
Forward pass: Feed all $N$ instances through the model to get predictions $Y^= XW$ . Rationale: you need predictions to compute errors.
Compute cost: MSE = $\frac{1}{2 N} \sum (Y^_{i} - Y_{i})^{2}$ . Rationale: measure how wrong the current weights are.
Check convergence: If cost < tolerance $ϵ$ , stop. Rationale: close enough to the minimum — further steps are wasted computation.
Compute gradient: $\frac{\partial J}{\partial W} = \frac{1}{N} \sum_{i = 1}^{N} (Y^_{i} - Y_{i}) X_{i}$ . Rationale: this tells each weight how much to change.
Update: $W \leftarrow W - η \cdot \frac{\partial J}{\partial W}$ . Rationale: take a small step downhill.
Repeat from step 2.

Complexity & Cost: Each update requires one matrix-vector multiplication over the entire dataset — $O (N d)$ operations. Memory must hold all $N$ examples. With millions of samples, one update can take seconds to minutes.

When to Use: Small-to-medium datasets where the entire dataset fits in memory and you want the most reliable convergence. Batch GD is rare in deep learning practice — minibatch dominates.

Trace — Batch GD on a tiny dataset: 3 houses. $X = 111123$ (size feature + bias column), $Y = [2, 4, 5]^{⊤}$ . $W = [0, 0]^{⊤}$ . $η = 0.1$ .

Iteration 1: $Y^= XW = [0, 0, 0]^{⊤}$ . Errors: $[- 2, - 4, - 5]^{⊤}$ . Gradient: $\frac{1}{3} [- 11, - 25]^{⊤} \approx [- 3.67, - 8.33]^{⊤}$ . Update: $W = [0, 0] - 0.1 \times [- 3.67, - 8.33] = [0.37, 0.83]$ . Loss drops from 7.5 to ~1.51 — an ~80% reduction in one step. This convergence speed is typical for well-conditioned linear regression.

4.6.4 Mini-Batch Gradient Descent

Purpose: Balance gradient accuracy against update speed by using a small random subset (mini-batch) of the data for each weight update.

Inputs & Outputs: Same as batch GD, plus batch size $B$ (a hyperparameter, typically 32–256).

Steps — with rationale:

Initialize weights (e.g., zeros or small random values).
Shuffle the dataset randomly. Rationale: prevent the model from learning ordering artifacts.
Partition into mini-batches of size $B$ . For $N = 1000, B = 500$ : 2 mini-batches.
For each mini-batch, compute the gradient using only those $B$ instances, then update weights.
Repeat for multiple epochs (full passes through the dataset), reshuffling each epoch.

Complexity & Cost: Each update costs $O (B d)$ — $B ≪ N$ , so updates are much faster than batch GD. Memory holds only $B$ examples at a time. The gradient is noisier (estimated from fewer samples), but the noise is beneficial — it helps escape shallow local minima and saddle points.

When to Use / Alternatives: The default choice for virtually all deep learning. Use batch GD when the dataset is tiny (< 1000 examples); use pure SGD only when features are extremely high-dimensional (e.g., NLP with vocabulary-sized features).

4.6.5 Stochastic Gradient Descent (SGD)

Purpose: Update weights after seeing every single training example individually. Maximizes update frequency at the cost of gradient accuracy.

Inputs & Outputs: Same as batch GD, with $B = 1$ effectively.

Steps — with rationale:

Initialize weights.
Randomly pick one instance from the dataset.
Compute the prediction and error for that single instance.
Compute the gradient from that one error. No averaging — the factor is 1, not $1/ B$ .
Update weights immediately.
Repeat, picking another random instance.

Complexity & Cost: Each update is $O (d)$ — extremely fast. But the gradient is a very noisy estimate of the true direction. The optimizer zigzags toward the minimum rather than following a smooth path. The noise can help explore the loss surface and escape poor minima.

When to Use / Alternatives: Use when the dataset is enormous (too large for even mini-batches to iterate quickly) or when features are vocabulary-based with millions of dimensions. In practice, pure SGD is rare — mini-batch SGD dominates because modern hardware (GPUs) is optimized for batch matrix multiplications, and processing one sample at a time underutilizes the hardware.

Comparison of the three variants:

Variant	Instances per update	Gradient quality	Speed per update	Memory	Best for
Batch	All $N$	Smooth, accurate	Slowest	High (all data)	Small datasets, convex problems
Mini-Batch	$B$ (32–256)	Moderate noise	Moderate	Low (batch size)	Default for deep learning
Stochastic	1	Very noisy	Fastest	Minimal (1 sample)	Massive features, streaming data

> Real-world: For NLP tasks with vocabulary-based features (millions of dimensions), batch processing is infeasible. The enrichment text notes that vectorized matrix-vector operations can be ~500× faster than scalar operations (0.00036s vs 0.178s for 10,000-element vectors) — this is why mini-batch dominates over pure SGD: it uses the hardware while keeping updates frequent.

Assumptions & Scope: Gradient descent assumes the loss function is differentiable with respect to every parameter. This holds for MSE + linear regression but breaks for models with discrete operations (e.g., thresholding). It assumes the learning rate is small enough that linear approximation of the loss surface (a first-order Taylor expansion) is reasonable at each step. For non-convex problems, gradient descent is not guaranteed to find the global minimum — it converges to whatever stationary point (minimum, maximum, or saddle) lies downhill from the starting point.

Visual Intuition: Picture a foggy hillside at dawn. You are blindfolded and need to find the valley bottom. You feel the ground under your feet — that is the gradient. Steep uphill behind you means you face downhill. You take one small step forward (the learning rate). You feel again. Another small step. This is gradient descent. If you take giant leaps, you might jump clear over the valley and land on the opposite slope — overshooting. If you take tiny shuffles, you make progress but it takes all day. The learning rate is your stride length. With batch GD, you feel the entire hillside at once (accurate but slow). With SGD, you only feel the one pebble under your toe (noisy but fast). With mini-batch, you feel a small patch of ground — enough to know the general slope without surveying the whole mountain.

Pitfalls:

Setting the learning rate too high. The optimizer jumps across the minimum, bounces between sides of the bowl, and the loss oscillates or diverges. Signs: the loss graph looks like a zigzag that never settles.
Setting the learning rate too low. Convergence takes forever. Signs: the loss decreases steadily but painfully slowly. After 10,000 iterations you are still far from the minimum.
Not shuffling data for mini-batch SGD. If the data is sorted (e.g., all class-A examples first, then all class-B), the gradient from each mini-batch is biased and convergence suffers. Always shuffle.
Confusing an epoch with an iteration. An epoch is one full pass through the dataset. An iteration is one weight update. With batch GD: 1 iteration = 1 epoch. With mini-batch GD: many iterations per epoch. With SGD: $N$ iterations per epoch. Do not report "converged after 100 iterations" when you mean 100 epochs — the scale is completely different.
Forgetting that SGD noise can be a feature, not a bug. The noise in SGD helps the optimizer explore the loss surface and can lead to better generalization (flatter minima). Pure batch GD, with its exact gradients, can get stuck in sharp minima that do not generalize well.

Q: Does the update step update all weights at once?

A: Yes. The update is a vector operation. $W$ is not a single weight but a vector $[W_{0}, W_{1}, \dots, W_{d}]$ . The gradient is also a vector — each weight has its own gradient component. One step updates the entire vector simultaneously. In code: W = W - lr * gradient operates element-wise on all weights at once.

Recap + Bridge: Gradient descent adjusts weights by taking small steps opposite to the gradient. The learning rate controls step size. Batch, mini-batch, and stochastic variants trade gradient accuracy for speed. All three use the same weight update rule. The question now: when do you stop? That is the convergence problem.

Real-World & Domain Connection: The mini-batch sizes used in production are often powers of 2 (32, 64, 128, 256) because GPU memory architectures are optimized for these dimensions — memory transactions are coalesced efficiently when data dimensions align with warp/wavefront sizes. Google's original word2vec paper (Mikolov et al., 2013) used SGD on a vocabulary of millions of words, demonstrating that stochastic updates on NLP-scale feature spaces could produce state-of-the-art word embeddings. Today, large language model training (GPT, Llama) uses distributed minibatch SGD across thousands of GPUs, with each GPU computing gradients on its local mini-batch and synchronizing via all-reduce operations. The gradient descent algorithm you just learned scales from a single neuron to trillion-parameter models without changing its fundamental structure.

4.7 Convergence Criteria

Hook: You have been training for what feels like forever. The loss keeps dropping — 7.5, 3.2, 1.1, 0.8, 0.79, 0.78... At what point do you stop and say "good enough"? Keep training too long and you might overshoot the minimum and the loss starts climbing again. Stop too early and you leave performance on the table.

4.7.1 Definition and Explanation

Intuition + Analogy: You are tuning a radio to find the clearest signal. You turn the dial. The static fades. The music gets clearer. You keep turning. It gets clearer still. Eventually you hit the sweet spot — any further turning and the clarity worsens. You nudge back to the clearest point and stop. That is convergence. The three criteria below are three ways of deciding "I am at the clearest point."

Now imagine the radio has no static at all — pure silence. That is a loss of exactly zero: the model predicts every training value perfectly. This almost never happens with real data. There is always some irreducible noise — measurement error, missing features, inherent randomness. Chasing zero loss on real data is chasing ghosts.

Formalize — three convergence criteria:

Criterion 1 — Zero loss: If the cost function $J (W)$ reaches exactly zero, stop. The model predicts the training data perfectly. Every $Y^_{i} = Y_{i}$ . In practice, this is rare. It only happens when the data is perfectly linear with no noise, or when the model is massively overfit.

Criterion 2 — Monitor loss progression (early stopping with rollback): Track the loss value at each iteration. If the loss increases instead of decreasing, you have passed the minimum. Stop. Go back to the iteration that had the lowest loss and take those weights. The increasing loss is a clear signal: the optimizer has overshot the valley bottom and is now climbing the opposite slope.

Criterion 3 — Tolerance threshold: Set a small threshold $ϵ$ (epsilon, e.g., 0.001 or 0.5). If the loss drops below $ϵ$ , stop. The reasoning: perfect zero is unreachable for real data. There is always some irreducible error. Rather than risk divergence by chasing zero, accept "good enough." If $∣Δ J ∣$ (the change in loss between iterations) is smaller than $ϵ$ , the optimizer is barely moving — further training is wasted computation.

Criterion 4 (simple, common in practice) — Fixed iteration count: Hard-code a maximum number of iterations (e.g., 1000). Stop after that many rounds. Accept the final weights. This is a hyperparameter-based approach — crude but effective when you know roughly how many iterations the problem needs.

Worked Example: You train a linear regression model on house price data. Losses per iteration: [7.5, 3.2, 1.1, 0.8, 0.79, 0.78, 0.77, 0.76, 0.77, 0.79, ...].

Criterion 1 (zero): Never triggers — loss is nowhere near zero.
Criterion 2 (increasing loss): Triggers at iteration 9 (0.77 → 0.79). Revert to iteration 8 (loss = 0.76).
Criterion 3 (tolerance = 0.001): Look at $Δ J$ between iteration 8 and 7: $∣0.76 - 0.77∣ = 0.01 > 0.001$ . Not converged yet. Between iteration 8 and 9: loss increased — already stopped by criterion 2.
Criterion 4 (max 1000 iterations): Would have kept training, potentially oscillating, wasting 992 iterations.

Best strategy: combine criteria. Use fixed iterations as a safety cap, tolerance for early stopping when progress stalls, and loss increase detection as a hard stop with rollback.

Assumptions & Scope: Convergence criteria assume the loss decreases monotonically (or nearly so) with a well-chosen learning rate. If the learning rate is too high, the loss may oscillate from the very first iteration — none of these criteria will work correctly because the optimizer never settles. Convergence detection also assumes the loss surface is smooth enough that local increases reliably signal overshooting. On very noisy surfaces (e.g., SGD on a non-convex problem), the loss may fluctuate naturally — a single increase does not always mean you passed the minimum.

Visual Intuition: Draw a U-shaped curve (loss vs. iteration). Starting from the top left, the curve slopes down toward the bottom right. Three horizontal dashed lines at different loss values: $J = 0$ near the x-axis (labeled "Criterion 1 — zero loss, rare"), $J = ϵ$ slightly above zero (labeled "Criterion 3 — tolerance"), and a vertical dashed line labeled "Criterion 4 — max iterations." An arrow traces the curve downward, then dips below the minimum and rises — at the lowest point, a label "Criterion 2 — stop here, minimum loss."

Pitfalls:

Setting $ϵ$ too small. If the irreducible noise in your data means the best possible loss is 0.5, and you set $ϵ = 0.001$ , training will run forever (timed out only by max iterations). Match $ϵ$ to the scale of your problem.
Interpreting a single noisy increase as divergence. In mini-batch or SGD, the loss can bounce around due to batch randomness. A single uptick does not mean you passed the minimum. Look at the trend over several iterations, not point-by-point.
Using only one criterion. In production training loops, you typically combine: max epochs as a hard cap, validation loss plateau as a soft stop (patience-based early stopping), and tolerance on gradient magnitude. Any single criterion alone can fail.

Q: When doing gradient descent with small step sizes, how do we know when to stop?

A: Several students asked this. There are multiple ways: (1) Stop when cost becomes exactly zero — this is rare in practice. (2) Monitor cost progression — if cost starts increasing instead of decreasing, stop and revert to the iteration with minimum cost. This is the "overshoot detected" signal. (3) Set a tolerance threshold — if cost drops below a small value like 0.001, stop rather than risk divergence by chasing perfect zero. The course covers all three; the exam may ask you to list and explain them.

Recap + Bridge: You stop training when the loss is zero (rare), when the loss starts increasing (overshoot detected), when it falls below a tolerance (good enough), or after a fixed number of iterations (hard cap). Now, to understand why the loss increases if you overshoot and why a small learning rate helps, we need to visualize the error surface itself.

Real-World & Domain Connection: In production deep learning, the dominant convergence criterion is early stopping based on a held-out validation set. Keras and PyTorch Lightning implement this as a callback: monitor validation loss, and if it does not improve for $k$ consecutive epochs (the patience parameter), stop training and restore the best weights. This is criterion 2, automated. Google's training infrastructure for models like BERT and T5 uses validation perplexity as the monitored metric, with typical patience values of 3–5 epochs on massive datasets. The reason validation loss is preferred over training loss: a model can keep improving on training data while getting worse on unseen data (overfitting). Validation-based stopping catches this before it hurts generalization.

4.8 Cost Function Visualization

Hook: The cost function is a landscape. Each point on the map is a different set of weights. The height at that point is the error — higher is worse. Gradient descent is a hiker trying to walk downhill to the lowest valley. To understand why it works (and when it fails), you need to see the landscape.

4.8.1 1D Error Surface (One Weight)

Intuition + Analogy: Imagine you are standing on a curved path that runs through a valley. You can only walk forward (increase the weight) or backward (decrease it). You feel the slope under your feet — that is the gradient. If the ground slopes downward to your left, you step left. If it slopes downward to your right, you step right. You keep stepping until you reach the lowest point, where the ground is flat — zero gradient. That flat spot is the minimum.

The path is the 1D error surface. It is a single convex curve — a U-shape. The y-axis is cost (how wrong the model is). The x-axis is the weight value. Your starting position is your initial weight. Your destination is the bottom of the U: the optimal weight.

To visualize, consider a model with only one feature (one weight $w$ ). Plot cost (MSE) on the y-axis against $w$ on the x-axis:

At $w = 0.4$ , the cost might be around 40 — far from optimal.
Increase to $w = 0.8$ , cost drops to about 10 — getting closer.
At about $w = 1.4$ , cost reaches its minimum — the bottom of the bowl.
Increase further: cost rises again — you have gone past the minimum.

Gradient descent walks downhill along this curve. At each step, it computes the slope (the gradient $\frac{dJ}{d w}$ ). If the slope is negative (downhill to the right), it increases $w$ . If positive, it decreases $w$ . The step size is the learning rate. Too large a step, and the algorithm might jump from one side of the valley to the other — overshooting without ever settling at the bottom.

4.8.2 3D Bowl and Contours (Two Weights)

Formalize: With two weights ( $W_{0}$ and $W_{1}$ ), the cost surface lives in 3D. Place $W_{0}$ and $W_{1}$ on the horizontal plane (the floor) and cost $J (W_{0}, W_{1})$ on the vertical axis. The surface is a bowl — it curves upward in all directions from a single lowest point.

Viewing this bowl from directly above gives a contour plot. Each contour is a ring connecting all $(W_{0}, W_{1})$ pairs that produce the same cost. Concentric rings radiate outward from the minimum:

The innermost ring contains weights near the optimal values — lowest cost.
Outer rings contain increasingly worse weight configurations — higher cost.
Elliptical rings mean one weight influences the cost more than the other — the bowl is steeper along one axis.

Gradient descent on a contour plot moves perpendicular to the contour lines, taking the steepest downhill path. The path traces a smooth spiral from the outer ring to the center.

Worked Example — reading contours: A contour plot shows concentric ellipses. The center is at $(W_{0}, W_{1}) = (1.5, - 0.8)$ . The innermost ellipse is labeled $J = 5$ . The next: $J = 10$ . The next: $J = 20$ . Starting from $(W_{0}, W_{1}) = (5, 3)$ (on the $J = 80$ ring), gradient descent follows a path that cuts diagonally toward $(1.5, - 0.8)$ , crossing each ring at roughly right angles. The final weights converge to around the center: $(W_{0}, W_{1}) \approx (1.5, - 0.8)$ with minimum cost ≈ 5. Sense-check: this is exactly the behavior you expect from convex optimization — any starting point converges to the same global minimum.

4.8.3 Beyond Two Dimensions

With more than two weights, the surface lives in $D + 1$ dimensions — impossible to draw directly. But the concept holds. A convex cost function in any number of dimensions has exactly one global minimum. There are no local minima, no saddle points that trap the optimizer (for linear regression with MSE). The multidimensional bowl has a single lowest point. Gradient descent, following the steepest downhill path, is guaranteed to find it.

All the intuition from 1D and 2D generalizes:

The learning rate still controls step size — too large and you overshoot, even in 100 dimensions.
The gradient still points downhill — the direction of steepest descent.
Convergence is still guaranteed for convex problems.

Assumptions & Scope: The bowl visualization assumes a convex cost function — MSE with a linear model. For neural networks with hidden layers and nonlinear activations, the loss surface is no longer a simple bowl. It has ridges, saddle points, plateaus, and many local minima. The 1D/2D visualizations apply only to the linear regression case studied in this module. For deep networks, the cost surface is far more complex — but gradient descent, with careful learning rate tuning and momentum, still works remarkably well in practice.

Visual Intuition — 3D bowl: Name the axes: x-axis = $W_{0}$ (bias), y-axis = $W_{1}$ (feature weight), z-axis = $J (W_{0}, W_{1})$ (cost). The surface is smooth and U-shaped in every vertical slice. A red dot at the initial random weights sits high on the bowl wall. A sequence of connected blue dots traces a spiraling path down the inner surface, converging to a green dot at the very bottom — the global minimum. The takeaway: one bottom, one path (downhill), guaranteed arrival.

Visual Intuition — contour top-down view: Each contour is labeled with its cost value. The gradient descent path is a curved arrow that always crosses contours at right angles — the steepest descent direction. The arrow spirals inward toward the center.

Pitfalls:

Thinking all loss surfaces are bowls. Only linear models with convex losses (MSE, cross-entropy for logistic regression) give you a single bowl. Deep networks produce non-convex surfaces that look more like mountain ranges. Do not assume the 1D/2D intuition transfers directly.
Misreading contour plots. Contours that are far apart mean the slope is gentle — the cost changes slowly. Contours bunched close together mean a steep slope. Gradient descent takes larger steps in the steep direction and smaller steps in the shallow direction. This can create zigzagging paths if the bowl is skewed (elongated ellipses).
Forgetting the bias dimension. A 1-weight model is really a 2-weight model ( $W_{0}$ and $W_{1}$ ). The 1D curve you plot assumes $W_{0}$ is fixed or already optimized. The true surface is higher-dimensional than it first appears.

Q: How do we interpret cost function plots with more than one weight?

A: For two weights, visualize a 3D bowl. The cost is the vertical height. The two weights are the horizontal floor. Viewed from above, you see concentric contour rings — the innermost ring is the lowest cost. For more than two weights, you cannot draw it. But the idea is the same: a multidimensional bowl with a single lowest point. The gradient descent path winds downhill to that point. Several students asked this — it is a conceptual leap worth revisiting.

Q: Is the 1D error surface curve only for a single-feature model?

A: Yes. You can only visualize cost against one or at most two weights at a time. A single-feature model (plus bias) actually has two weights ( $W_{0}$ and $W_{1}$ ), so a true 1D plot holds one weight fixed. The 1D curve is a teaching tool: it shows you what the learning rate does, why small steps matter, and what overshooting looks like. With many weights, the intuition transfers but the plot does not.

Recap + Bridge: The cost surface for linear regression with MSE is a convex bowl — one global minimum, no traps. Gradient descent is guaranteed to reach it with a suitable learning rate. In 1D it is a U-curve. In 2D it is a 3D bowl with concentric contours. In higher dimensions, the same geometry holds — you just cannot draw it. Next, we put gradient descent to work on a concrete numerical example with three houses.

Real-World & Domain Connection: Loss landscape visualization is an active research area in deep learning. Li et al. (2018) showed that the loss surfaces of deep residual networks (ResNets) are surprisingly smooth and nearly convex in the region around the optimum — explaining why SGD works so well despite theoretical non-convexity. Tools like TensorBoard's embedding projector and the loss-landscape Python library let practitioners plot 2D slices of high-dimensional loss surfaces, revealing the path their optimizer took during training. In production, these visualizations help debug training failures: a jagged, oscillating loss curve suggests the learning rate is too high; a flat, unchanging curve suggests vanishing gradients or dead ReLUs. What you learn about the 1D and 2D error surface now is the foundation for diagnosing every training problem you will encounter later.

4.9 Worked Example: Batch Gradient Descent with Three Instances

Hook: Enough theory. You have the weight update rule, the loss function, and the gradient formulas. Can you actually run gradient descent by hand, with real numbers, and watch the loss drop? Let's do it — three houses, two weights, three iterations, and a calculator.

4.9.1 Problem Setup

Intuition: Before we compute anything, set the scene. We have three houses. The only feature is size (how big the house is). The target is price. We want a single line — $Y^= W_{0} + W_{1} \times size$ — that passes as close as possible to all three data points. We start with both weights at zero. The model predicts 0 for everything. It is about as wrong as possible. Gradient descent will fix it.

Model equation:

$Y^= W_{0} + W_{1} \cdot size$

Where $W_{0}$ is the bias (base price when size = 0) and $W_{1}$ is the weight for size (how much each unit of size changes the predicted price).

Given training data (3 instances):

Instance	Size ( $X_{1}$ )	Price ( $Y$ )
1	1	2
2	2	4
3	3	5

Hyperparameters (set before training):

Learning rate $η = 0.1$ . Note on the learning rate: the lecture states $η = 0.01$ at the outset, but the numerical walkthrough uses $η = 0.1$ . With $η = 0.01$ , the first update would produce $W_{0} = 0.0367, W_{1} = 0.0833$ . The professor's computed weights of $W_{0} = 0.36, W_{1} = 0.83$ confirm $η = 0.1$ was actually used. For this worked example, $η = 0.1$ is the value consistent with the computed results.
Number of iterations = 3 (hard-coded — we run exactly 3 iterations and stop).
No loss threshold — fixed iteration count for simplicity.

Initial weights: $W_{0} = 0$ , $W_{1} = 0$ .

4.9.2 Matrix Representation

Formalize — setting up the computation: Express the data in matrix form so the algebra is compact. The feature matrix $X \in R^{3 \times 2}$ includes a bias column of ones:

$X = 111123$

The first column is the bias feature ( $X_{0} = 1$ for every instance). The second column is size ( $X_{1}$ ). The target vector $Y \in R^{3}$ :

$Y = 245$

The weight vector $W \in R^{2}$ (to be learned):

$W = [W_{0} W_{1}]$

The prediction for all three instances in one operation: $Y^= XW$ .

4.9.3 Iteration 1 — Full Walkthrough

Step 1 — Forward pass (predictions):

$Y^= X \cdot W = 111123 \cdot [00] = 000$

All predictions are 0 because both weights are initialized to zero. The model has no information yet.

Step 2 — Compute errors: $e = Y^_{i} - Y_{i}$ for each instance:

$e = 0 - 2 0 - 4 0 - 5 = - 2 - 4 - 5$

All errors are negative — the model underpredicts every house. This makes sense: the actual prices are 2, 4, 5 and the model guessed 0 for all.

Step 3 — Compute MSE loss:

$MSE = \frac{1}{2 \times 3} [(- 2)^{2} + (- 4)^{2} + (- 5)^{2}] = \frac{1}{6} (4 + 16 + 25) = \frac{45}{6} = 7.5$

The lecture uses the $\frac{1}{2 N}$ convention for MSE, which matches the enrichment reference text (Eq. 3.1.5–3.1.6). Under this convention, the initial MSE is 7.5. If using $\frac{1}{N}$ instead, the value would be $\frac{45}{3} = 15$ . The convention affects the numerical loss value but not the optimal weights — both conventions lead to the same gradient direction (up to a constant factor absorbed by the learning rate).

Step 4 — Compute gradients:

Gradient for bias $W_{0}$ (where $X_{i 0} = 1$ for all instances):

$\frac{\partial J}{\partial W _{0}} = \frac{1}{3} i = 1 \sum 3 (Y^_{i} - Y_{i}) = \frac{1}{3} ((- 2) + (- 4) + (- 5)) = \frac{- 11}{3} \approx - 3.667$

Gradient for $W_{1}$ (multiply each error by its feature value $X_{i 1}$ ):

$\frac{\partial J}{\partial W _{1}} = \frac{1}{3} i = 1 \sum 3 (Y^_{i} - Y_{i}) \cdot X_{i 1} = \frac{1}{3} [(- 2 \times 1) + (- 4 \times 2) + (- 5 \times 3)] = \frac{1}{3} [- 2 + (- 8) + (- 15)] = \frac{- 25}{3} \approx - 8.333$

Step 5 — Weight update (with $η = 0.1$ ):

$W_{0}^{new} = 0 - 0.1 \times (- 3.667) = 0 + 0.3667 \approx 0.36$

$W_{1}^{new} = 0 - 0.1 \times (- 8.333) = 0 + 0.8333 \approx 0.83$

Both gradients were negative, so both weights increase — the model learns that higher weights reduce the error (since predictions were too low).

New weights after iteration 1: $W_{0} = 0.36$ , $W_{1} = 0.83$ .

Step 6 — Check loss after update (forward pass with new weights):

$Y^= 111123 \cdot [0.36 0.83] = 0.36 + 0.83 0.36 + 1.66 0.36 + 2.49 = 1.19 2.02 2.85$

New errors: $1.19 - 2 = - 0.81$ , $2.02 - 4 = - 1.98$ , $2.85 - 5 = - 2.15$ .

New MSE: $\frac{1}{6} [(- 0.81)^{2} + (- 1.98)^{2} + (- 2.15)^{2}] = \frac{1}{6} (0.656 + 3.920 + 4.623) = \frac{9.199}{6} \approx 1.53$ .

The loss dropped from 7.5 to about 1.53 — about an 80% reduction after just one iteration. This is the power of gradient descent when the problem is well-conditioned.

Key calculation walkthrough:

Initial loss: 7.5 → After 1 iteration: ~1.53
$W_{0}$ moved from 0 to 0.36
$W_{1}$ moved from 0 to 0.83
The model went from predicting 0 for everything to predicting values roughly in the ballpark of the actual prices.

4.9.4 Iterations 2 and 3

The process repeats using the updated weights from iteration 1:

Forward pass: compute new predictions with $W_{0} = 0.36, W_{1} = 0.83$ .
Compute errors: compare predictions against actual prices.
Compute gradients: average the error (for $W_{0}$ ) and error × feature (for $W_{1}$ ).
Update weights: $W_{j} \leftarrow W_{j} - 0.1 \times gradient$ .
After 3 iterations, stop (hard-coded limit). Return the final weights.

Key observation: The error dropped by about 80% after just one iteration. This is typical for simple linear regression with well-scaled data — gradient descent makes rapid initial progress, and later iterations refine the weights more slowly as you approach the minimum.

4.9.5 What "Learned Weights" Mean — Interpretation

Interpreting the final model: If the final learned weights after training are (for illustration) $W_{0} = 3$ and $W_{1} = 1.39$ , the trained model is:

$Y^= 3 + 1.39 \times size$

Interpretation: the base price (when size = 0) is 3 units. This is a mathematical intercept — it may not have physical meaning for a house of zero size. Each extra unit of size adds 1.39 units to the predicted price. If you double the size, the predicted price increases by $1.39 \times size$ .

The fitted line passes through or near the training data points. The vertical distance between each data point and the line is the residual (prediction error). Some residuals remain — the model is not perfect. But given the constraints (single feature, linear relationship), it is the best fit possible. This is the limit of a single-perceptron linear model on this data.

Assumptions & Scope: This worked example uses batch gradient descent (all 3 instances per update) on a tiny, noise-free dataset with a single feature. The data is nearly perfectly linear (prices go 2, 4, 5 as size goes 1, 2, 3 — almost a straight line). This is a teaching example to build intuition. Real datasets have noise, more features, and non-linear relationships. Do not expect 80% loss reduction in one iteration on real data.

Visual Intuition: Plot size on the x-axis (1, 2, 3) and price on the y-axis (2, 4, 5). Three blue dots mark the training data. The line $Y^= 0 + 0 \times size$ (initial weights) is a flat line along the x-axis — terrible fit. After iteration 1: the line tilts upward, passing roughly through the cluster. After iteration 3: the line angles through the three points, balancing the residuals. Vertical dashed lines from each dot to the line show the errors shrinking across iterations. The takeaway: gradient descent rotates and shifts the line until the errors are minimized — with big moves early and small refinements later.

Pitfalls:

Using the wrong learning rate convention in exams. The professor states $η = 0.01$ but computes with $η = 0.1$ . In an exam, use the learning rate value provided in the problem statement. If your calculated weights do not match expected values, double-check whether the learning rate was the source of discrepancy.
Forgetting to include the bias column of ones in $X$ . Without the column of ones, $X$ would be just a column vector of sizes, and $W$ would be a scalar $W_{1}$ . The model would have no bias term and would be forced through the origin — every prediction for size = 0 would be 0. Always prepend a column of ones.
Mixing up $Y^_{i} - Y_{i}$ and $Y_{i} - Y^_{i}$ . The gradient uses $(Y^_{i} - Y_{i})$ , not the other way around. If you flip the subtraction, the gradient sign flips, and weight updates move in the wrong direction.
Skipping the 1/N averaging in the gradient. The gradient is the mean of the errors, not the sum. If you forget to divide by $N$ , the effective learning rate becomes $N$ times larger, and the optimizer may overshoot.

Q: Perceptron output is always predicted value ( $Y^$ ), never actual value ( $Y$ ), correct?

A: Yes. Whatever comes out of the neural network is $Y^$ (predicted). You compare $Y^$ against the actual $Y$ from training data to get the error. Then you update the weights. The goal is to make future predictions closer to actual values. During training, the model never outputs actual $Y$ — only its current best guess $Y^$ .

Recap + Bridge: We ran batch gradient descent by hand on three houses. Initial loss: 7.5. After one iteration: ~1.53 (80% reduction). The weight update rule, the gradient formulas, and the forward pass all work together in a loop. This same loop scales to millions of instances and thousands of features — the math is identical, just the matrix dimensions grow. Next, we formalize this forward-backward flow as a computational graph — a visual tool for understanding how data and gradients move through the model.

Real-World & Domain Connection: This hand-worked example uses matrix operations that, at scale, run on GPUs. The matrix multiplication $XW$ for 3×2 matrices is very small, but the same operation (GEMM — General Matrix Multiply) is the most computationally intensive kernel in all of deep learning. NVIDIA's cuBLAS library, which powers PyTorch and TensorFlow on GPUs, contains hand-optimized assembly for matrix multiplication that achieves over 90% of theoretical peak floating-point performance. The three-step loop you just did by hand — forward pass, loss, gradient, update — is literally the training loop in every production ML system. When you see "model.fit(X_train, y_train)" in a Jupyter notebook, behind the scenes the machine is computing exactly the same gradient formulas on batches of data, thousands of times per second.

4.10 Computational Graphs

Hook: You have a model, a loss function, and gradient formulas. But how does the gradient actually flow backward from the loss to every single weight? The answer is a computational graph — a blueprint that shows exactly which operations depend on which values, and how the error signal ripples backward through every connection.

4.10.1 Definition and Walkthrough

Intuition + Analogy: Think of a factory assembly line. Raw materials (inputs) enter at the left. Each station performs one operation — mixing, cutting, painting. The finished product (prediction) exits at the right. At the end, a quality inspector (loss function) measures the defect rate. Now the inspector walks backward through the assembly line, telling each station: "your work contributed this much to the defect." Each station then adjusts its settings. The forward walk is the forward pass. The backward walk with defect attribution is backpropagation. The blueprint of the factory floor — which stations feed into which — is the computational graph.

Where the analogy breaks: in a real factory, adjustments happen slowly and individually. In a computational graph, the chain rule of calculus lets you compute all adjustments simultaneously from a single backward pass.

A computational graph is a directed graph where:

Nodes are operations (multiply, add, square, average).
Edges are data (tensors) flowing between operations.
Forward pass: data flows from inputs (left) to outputs (right).
Backward pass: gradients flow from outputs (right) to inputs (left) via the chain rule.

Purpose: Make the gradient computation explicit and modular. By decomposing a complex function (like MSE of a linear model) into elementary operations, you can compute the gradient of each operation independently and compose them via the chain rule. This is what automatic differentiation frameworks (PyTorch, TensorFlow) do internally.

Inputs & Outputs:

Input: Feature matrix $X \in R^{N \times (d + 1)}$ , weight vector $W \in R^{d + 1}$ , target vector $Y \in R^{N}$ .
Output: Scalar loss $J$ , and gradients $\frac{\partial J}{\partial W}$ for weight update.

Steps — for single-perceptron linear regression with batch GD:

Input nodes: $X$ (features) and $W$ (weights). These are leaf nodes — they have no incoming edges from other operations.
Multiply (MatMul): $Y^= X \cdot W$ . Forward: produces predictions $Y^$ . Backward: $\frac{\partial Y ^}{\partial W} = X^{⊤}$ (the transpose) — this tells each weight how its change affects the predictions.
Subtract: $e = Y^- Y$ . Forward: produces error vector $e$ . Backward: passes gradient through unchanged (derivative of subtraction is 1).
Square: $e^{2}$ . Forward: squares each error. Backward: $\frac{\partial}{\partial e} (e^{2}) = 2 e$ — a factor of 2 times the error.
Sum and scale: $J = \frac{1}{2 N} \sum e_{i}^{2}$ . Forward: produces scalar loss. Backward: $\frac{\partial J}{\partial e _{i}} = \frac{1}{N} e_{i}$ (the 2 from step 4 cancels with $\frac{1}{2}$ , leaving $\frac{1}{N} e_{i}$ ).
Chain rule composition: The gradient at the weight node is the product of gradients along the backward path: $\frac{\partial J}{\partial W} = \frac{\partial J}{\partial Y ^} \cdot \frac{\partial Y ^}{\partial W} = \frac{1}{N} e^{⊤} X$ (or in the lecture's notation: $\frac{1}{N} \sum (Y^_{i} - Y_{i}) \cdot X_{i}$ ).
Scale by learning rate: $η \cdot \frac{\partial J}{\partial W}$ .
Update: $W^{new} = W^{old} - η \cdot \frac{\partial J}{\partial W}$ .

Complexity & Cost: The backward pass has roughly the same computational cost as the forward pass — about 2× the operations. Memory stores intermediate values from the forward pass (activations) because they are needed during the backward pass (e.g., $e$ is needed to compute $\frac{\partial J}{\partial Y ^}$ ).

When to Use / Alternatives: Computational graphs are how every modern deep learning framework works. The alternative — deriving gradient formulas by hand for each new model — is impractical beyond simple models. The graph abstraction lets you build arbitrarily complex architectures and still get correct gradients automatically.

Variant handling (batch vs. mini-batch vs. stochastic):

Batch GD: The averaging step (divide by $N$ ) uses all instances.
Mini-batch GD: Average over only $B$ instances in the current mini-batch.
SGD: No averaging — use the error and gradient from a single instance directly.

The structure of the graph is identical across all three variants. Only the size of the data tensors and the presence/absence of the averaging node changes.

Trace — computational graph for our 3-house example:

Forward pass:

$X = 111123$ , $W = [00]$ → multiply → $Y^= 000$
$Y^$ , $Y = 245$ → subtract → $e = - 2 - 4 - 5$
Square: $e^{2} = [4, 16, 25]$ → sum → 45 → $\frac{1}{6}$ → $J = 7.5$

Backward pass:

$\frac{\partial J}{\partial e} = \frac{1}{3} [- 2, - 4, - 5] = [- \frac{2}{3}, - \frac{4}{3}, - \frac{5}{3}]$
$\frac{\partial e}{\partial Y ^} = 1$ (identity)
$\frac{\partial Y ^}{\partial W} = X^{⊤}$ → combined: $\frac{\partial J}{\partial W} = \frac{1}{3} X^{⊤} e = \frac{1}{3} [- 11 - 25] = [- 3.67 - 8.33]$ .

Assumptions & Scope: The computational graph representation assumes every operation is differentiable. Discrete operations (thresholding, argmax, sampling) break the gradient chain. Frameworks handle these via surrogate gradients or by stopping gradient flow at non-differentiable nodes. The graph also assumes static structure during a forward-backward pass — dynamic graphs (where the structure changes based on data) require more advanced frameworks like PyTorch's dynamic computation graph or JAX's jit.

Visual Intuition — drawing the graph yourself (exam skill):

Draw boxes connected by arrows. Leftmost box: " $X, W$ " (inputs). Arrow to: "Multiply: $Y^= XW$ ". Arrow to: "Subtract: $e = Y^- Y$ " (with $Y$ as a second input from the top). Arrow to: "Square: $e^{2}$ ". Arrow to: "Averaging + scaling: $J = \frac{1}{2 N} \sum$ ". Below, draw a parallel set of backward arrows pointing from right to left, labeled with the gradient expressions at each edge. Label the forward arrows "Forward pass (data)" and the backward arrows "Backward pass (gradients)."

If the exam asks for batch GD, annotate the averaging node with " $1/ N$ ". For mini-batch, annotate " $1/ B$ ". For SGD, remove the averaging node entirely.

Pitfalls:

Forgetting that the backward pass follows the chain rule in reverse order. If the forward path is A → B → C, the gradient flows C → B → A. The graph traversal order matters.
Omitting the averaging step for batch/mini-batch GD. The gradient must be divided by $N$ (or $B$ ). If you forget, the effective learning rate becomes $N$ times larger.
Confusing $\frac{\partial Y ^}{\partial W} = X^{⊤}$ with $X$ . The derivative of $XW$ with respect to $W$ is $X^{⊤}$ — the transpose. For a row-vector input, this distinction matters for the dimensions to multiply correctly.

Exam note: You might be asked to draw a computational graph and label both forward and backward flow. Note whether the problem uses batch, mini-batch, or stochastic — the averaging step differs. Be precise about gradient expressions at each edge.

Recap + Bridge: A computational graph traces data forward and gradients backward through every operation. It makes the chain rule explicit and prepares you for automatic differentiation — the engine behind every deep learning framework. With the model trained, the next question is: how good is it really? That is model evaluation.

Real-World & Domain Connection: Every modern deep learning framework (PyTorch, TensorFlow, JAX) builds a computational graph internally. In PyTorch, calling loss.backward() traverses the graph in reverse, applying the chain rule at each operation to populate .grad attributes on every parameter tensor. This is why you never have to derive gradients by hand for custom architectures — you define the forward pass, and the framework handles the backward pass automatically. The concept was pioneered by the Autograd system (developed at Harvard and later incorporated into PyTorch) and by TensorFlow's static graph compiler. Today, even non-ML software — like physics simulators and renderers — uses computational graphs for differentiable programming, allowing gradient-based optimization of systems that were never designed for it.

4.11 Model Evaluation

Hook: Your model got an MSE of 0.5 on the training data. Great! But when you show it a completely new house, the prediction is off by 50K. The model memorized the training examples instead of learning the pattern. How do you catch this before it embarrasses you? You hold out test data.

4.11.1 Training Error vs Test Error

Intuition + Analogy: You are studying for an exam. You do every problem in the textbook perfectly (zero training error). The exam arrives — and every question is new. You freeze. You memorized the textbook problems instead of understanding the concepts. That is overfitting. A classmate who made some mistakes on the textbook but solved novel exam problems understood the material. That is good generalization.

The textbook problems are the training set. The exam is the test set. Your goal is not to ace the textbook — it is to ace the exam. In machine learning, the goal is to minimize test error, not training error.

Formalize:

Training data: The portion of the dataset used to adjust weights during learning. The error computed on this data is the training error ( $J_{train}$ ).
Test data: A portion held aside, never seen during training. The error computed on this data is the test error ( $J_{test}$ ).

The goal of machine learning is to minimize $J_{test}$ . A model with low $J_{train}$ but high $J_{test}$ has overfit — it memorized the training examples rather than learning the underlying pattern. A model with high values for both has underfit — it failed to capture the pattern at all.

Standard procedure:

Split the dataset randomly: typically 80% training, 20% test.
Train the model on the training portion only. The test data stays locked in a drawer.
After training, pass test features through the trained model (using learned weights — frozen, no updates).
Compare test predictions against actual test targets.
Compute evaluation metrics on the test set.

The model is good only if both training error and test error are low. A large gap between the two ( $test error ≫ training error$ ) signals overfitting.

Worked Example: Auto MPG dataset — 392 vehicles, 8 features. Split: 314 training (80%), 78 test (20%). Model trains on 314 vehicles. Training MSE: 5.2. Test MSE: 5.8. The gap is small (0.6) — the model generalizes well. If test MSE were 25.0 while training MSE remained 5.2, the model would have overfit severely. Sense-check: a small gap between train and test error is a sign of good generalization.

4.11.2 Evaluation Metrics

Mean Square Error (MSE): Defined in §4.5.1. Applied to test data to measure deviation of predictions from actual values. Note: compute MSE on the test set using the same formula, but with frozen (learned) weights.

Root Mean Square Error (RMSE):

$RMSE = MSE$

The square root brings the units back to the original scale. If you are predicting dollars, MSE is in squared dollars (hard to interpret), while RMSE is in dollars. Always report RMSE alongside MSE for interpretability.

Mean Absolute Error (MAE): Defined in §4.5.2. Less sensitive to outliers. Units match the target variable directly.

4.11.3 R-squared ( $R^{2}$ ) Error

Formalize: $R^{2}$ (the coefficient of determination) answers: "How much better is my model than just guessing the mean every time?"

$R^{2} = 1 - \frac{\sum _{i = 1}^{N} ( Y ^ _{i} - Y _{i} ) ^{2}}{\sum _{i = 1}^{N} ( Y _{i} - Y ˉ ) ^{2}}$

Where:

Numerator: $SS_{res} = \sum (Y^_{i} - Y_{i})^{2}$ — Residual Sum of Squares. The error remaining after using your model.
Denominator: $SS_{tot} = \sum (Y_{i} - Y ˉ)^{2}$ — Total Sum of Squares. The error if you predicted the mean $Y ˉ$ for every instance.
$Y ˉ = \frac{1}{N} \sum Y_{i}$ — the mean of actual values.

The ratio $\frac{SS _{res}}{SS _{tot}}$ is the fraction of variance the model failed to explain. $1 - that$ is the fraction it did explain.

Worked Example: Actual prices: $Y = [2, 4, 5]$ . Mean: $Y ˉ = 3.67$ . Total sum of squares: $(2 - 3.67)^{2} + (4 - 3.67)^{2} + (5 - 3.67)^{2} = 2.79 + 0.11 + 1.78 = 4.68$ .

Model predictions: $Y^= [1.19, 2.02, 2.85]$ . Residual sum of squares: $(1.19 - 2)^{2} + (2.02 - 4)^{2} + (2.85 - 5)^{2} = 0.66 + 3.92 + 4.62 = 9.20$ .

$R^{2} = 1 - \frac{9.20}{4.68} = 1 - 1.97 = - 0.97$ .

Interpretation: The model is worse than just predicting the mean. This is iteration 1 with barely-trained weights — expected. After training, $R^{2}$ should approach 1. Sense-check: an $R^{2}$ near 1 means the model captures almost all the variance. An $R^{2}$ below 0 means the model is worse than a constant prediction — go back and debug.

Interpretation guide for $R^{2}$ :

$R^{2} = 1$ — Perfect fit. Every prediction equals the actual value.
$0.7 \leq R^{2} < 1$ — Good fit for many real-world regression problems.
$R^{2} = 0$ — The model is no better than predicting the mean every time.
$R^{2} < 0$ — The model is worse than the mean prediction. Something is broken — check your code, your data, or your model architecture.

Scope — $R^{2}$ does not tell the whole story: $R^{2}$ measures explained variance, not accuracy. A model can have $R^{2} = 0.99$ but still make predictions that are hundreds of units off if the variance in the data is enormous. Always pair $R^{2}$ with RMSE or MAE — they give complementary information. Also, $R^{2}$ always increases (or stays the same) when you add more features, even if those features are noise. Adjusted $R^{2}$ penalizes unnecessary features — it is the safer metric for feature selection.

Visual Intuition: Plot actual vs. predicted values as a scatter plot. Each point is one test instance. The x-axis is actual $Y$ , the y-axis is predicted $Y^$ . Draw the diagonal $y = x$ line — points on this line are perfect predictions. Points above the line are over-predictions. Points below are under-predictions. A tight cluster near the diagonal means good predictions (high $R^{2}$ ). A loose, scattered cloud means poor predictions (low $R^{2}$ ). The residual plot — showing $(Y^_{i} - Y_{i})$ on the y-axis against $Y^_{i}$ on the x-axis — should show random scatter around the zero line with no pattern. A pattern (e.g., a funnel shape, a curve) means the model's errors are systematic, not random — the model is missing structure in the data.

Pitfalls:

Evaluating on training data and calling it performance. This is like grading your own exam. The training error is optimistically biased. Always report test-set metrics.
Using $R^{2}$ for non-linear models without caution. $R^{2}$ is defined for linear regression with an intercept. For models without a bias term or with nonlinear outputs, the interpretation can break down (e.g., $R^{2}$ can exceed 1 or be undefined).
Ignoring the residual plot. A model can have decent RMSE and $R^{2}$ while having systematically biased errors for certain subsets of the data (e.g., always overpredicting small houses and underpredicting large ones). The residual plot catches this.
Leaking test data into training. If you standardize your features using the mean and standard deviation of the entire dataset (train + test) before splitting, you have leaked information from the test set into training. Fit the scaler on training data only, then transform both sets.

Recap + Bridge: Split your data. Train on the training set. Evaluate on the test set. Use MSE/RMSE for magnitude of error, MAE for outlier-robust error, and $R^{2}$ for variance explained. A small train-test gap means good generalization. Next, we see a full implementation of everything we have built — from data loading to evaluation — on a real dataset.

Real-World & Domain Connection: In regulated industries (finance, healthcare), model evaluation is not just best practice — it is law. The Federal Reserve's SR 11-7 guidance on model risk management requires banks to document training and test performance, analyze residuals, and monitor model drift over time. The FDA requires medical device software using ML to report performance on held-out test data with predefined metrics. In Kaggle competitions, the public leaderboard is computed on a subset of test data, but the final winner is determined by a private test set never shown to competitors — a direct application of the train/test split principle designed to prevent overfitting to the leaderboard. The metrics you just learned (MSE, MAE, $R^{2}$ ) are the vocabulary of model evaluation in every domain.

4.12 Implementation: Single Neuron Regression on Auto MPG Dataset

Hook: You have learned the math, the gradients, and the update rules. Now it is time to code it all from scratch — every line. No libraries. No shortcuts. Just Python, NumPy, and the formulas you already know. By the end, you will have a working regression model that predicts a car's fuel efficiency from its engine specs.

4.12.1 Dataset Description

The Auto MPG dataset contains data on 392 vehicles from the 1970s and early 1980s. Features:

Cylinders — number of cylinders in the engine.
Displacement — engine displacement in cubic inches.
Horsepower — engine power output.
Weight — vehicle weight in pounds.
Acceleration — time to accelerate from 0 to 60 mph.
Model year — the model year (e.g., 70 for 1970).
Origin — country of manufacture (1 = USA, 2 = Europe, 3 = Japan).

The target variable is MPG (miles per gallon) — a continuous fuel efficiency measure. This is a regression problem: predict a real number from continuous and categorical features.

4.12.2 Preprocessing Steps

Purpose: Transform raw data into a clean, standardized format suitable for gradient descent. Raw data contains missing values, irrelevant columns, and features on vastly different scales — all of which break or slow down training.

Inputs & Outputs:

Input: Raw dataset (rows of vehicles, columns of features + target).
Output: Standardized feature matrices $X_{train}, X_{test}$ and target vectors $Y_{train}, Y_{test}$ , with no missing values and all features on comparable scales.

Steps — with rationale:

Load the dataset. From local file, URL, or uploaded to the runtime environment (Google Colab in this case). This brings the raw data into memory.

Handle missing values. Drop rows with missing feature values — never drop entire feature columns, or you lose information. In the Auto MPG dataset, the horsepower column has a few missing entries (some older cars did not report horsepower). Dropping those rows (~6 instances) is safe for 392 total examples. In traditional ML, you could impute (fill in) missing values with the mean or median, but that is outside this module's scope.

Remove irrelevant features. The "car name" column is a string label — it has no predictive value for MPG. Drop it. Text features need special encoding (not covered here).

Separate features and target. The MPG column becomes $Y$ (target). All other columns become $X$ (feature matrix). Check shapes: $X \in R^{392 \times 7}$ , $Y \in R^{392}$ .

Train-test split. 80% for training ( $\approx 314$ ), 20% for testing ( $\approx 78$ ). Use train_test_split from scikit-learn or a simple random shuffle + index split.

Standardize (normalize) the features. This is critical. The weight feature has values in thousands (2000–5000 lbs). The cylinders feature has values 3–8. Without standardization, the weight gradient dominates — a small relative change in weight overwhelms any signal from cylinders. Standardization transforms each feature to have mean 0 and standard deviation 1:

$X_{std} = \frac{X - μ}{σ}$

where $μ$ is the mean and $σ$ is the standard deviation computed on the training data only. Then transform test data using the same $μ$ and $σ$ — do not fit on test data (that would leak information).

Critical pitfall — data leakage: If you standardize using mean and std of the entire dataset (train + test together) before splitting, you have leaked information about the test set's scale into the training process. Always: fit scaler on training data → transform training data → transform test data using the training scaler.

Complexity & Cost: Preprocessing is a one-time cost before training. Standardization is $O (N d)$ — one pass over the data. Train-test splitting is $O (N)$ . These steps are negligible compared to training time.

When to Use / Alternatives: Always standardize for gradient-based models (neural networks, linear/logistic regression with SGD). Tree-based models (Random Forest, XGBoost) do not need standardization — they split on raw feature values. For features with heavy outliers, consider outlier-resistant scaling (using median and IQR instead of mean and std).

4.12.3 Model Architecture

Formalize: The model is a single perceptron — the simplest neural network possible.

Components:

Input layer: $D = 7$ features (after dropping car name, before standardization).
Output: Exactly one neuron with identity activation — the raw weighted sum is the prediction. No hidden layers, no thresholding, no clipping.
Operation: $Y^= X \cdot W + W_{0}$ .

In matrix form (with bias column of ones prepended to $X$ ):

$Y^= XW, X \in R^{N \times (D + 1)}, W \in R^{D + 1}$

This is batch gradient descent — the mean is taken over all $N$ instances.

4.12.4 Class Implementation (Python Walkthrough)

Purpose: Encapsulate the entire model — weights, predictions, loss, gradients, training loop, and evaluation — in a single class. This mirrors how real ML frameworks organize code.

Class structure — SingleNeuronRegression:

__init__(self, learning_rate, num_iterations) — store hyperparameters. Initialize weights vector $W$ to zeros (or small random values). Create a loss_history list to track MSE per iteration for plotting.

predict(self, X) — forward pass. One line: return X @ self.W. This computes $Y^= XW$ for all instances at once. If the bias is a separate variable: return X @ self.W + self.bias.

compute_loss(self, Y_cap, Y) — compute MSE. Compute (Y_cap - Y)2 element-wise, then take the mean: return np.mean((Y_cap - Y)2). The professor uses the $\frac{1}{2 N}$ convention in the math but the code uses np.mean which gives $\frac{1}{N}$ — the factor of 2 difference is absorbed by the learning rate. In practice, either convention works; the loss curve's shape is identical, just scaled.

compute_gradient(self, X, Y_cap, Y) — gradient computation:

Error vector: e = Y_cap - Y (shape: $N \times 1$ ).
Bias gradient: (1/N) * np.sum(e) — mean of errors.
Weight gradient: (1/N) * X.T @ e — matrix multiply, equivalent to $\frac{1}{N} \sum e_{i} X_{i}$ .
Return both.

fit(self, X_train, Y_train) — training loop:

Prepend a column of ones to X_train for the bias term.
For i in range(num_iterations):
Y_cap = self.predict(X_train)
loss = self.compute_loss(Y_cap, Y_train) → append to history
grad_W = self.compute_gradient(X_train, Y_cap, Y_train)
self.W = self.W - self.lr * grad_W
If i % 100 == 0: print loss for monitoring.
Return loss_history.

test(self, X_test, Y_test) — evaluation:

Prepend ones column.
Y_cap = self.predict(X_test).
Compute MSE, MAE, $R^{2}$ on test data.
Return metrics.

self parameter: In Python, self refers to the class instance — it gives each method access to the instance's weights, learning rate, and loss history. Analogous to this in C++ or Java. When you call model.fit(X, Y), Python passes model as the first argument (self) automatically.

Trace — one training iteration: $X_{train}$ is $314 \times 8$ (7 features + bias column). $W$ is $8 \times 1$ (all zeros initially). Learning rate $η = 0.01$ .

Forward pass: $Y^= X_{train} W = 0$ (all zeros). Loss: ~300 (first iteration, high). Gradient: computed via X.T @ (Y_cap - Y) / 314. Update: $W \leftarrow W - 0.01 \times gradient$ . Loss after update: drops noticeably. After 1000 iterations, loss converges to ~5–6. The model has learned meaningful weights from the data.

4.12.5 Results

With 1000 iterations and learning rate $η = 0.01$ on the Auto MPG dataset, the model converges. The learning rate of 0.01 used here is a separate example from the hand-worked numerical example in §4.9 which used $η = 0.1$ on synthetic data. The hand-worked example was a toy problem for building intuition; the Colab implementation uses real data and a smaller learning rate (0.01) because the dataset is larger and noisier — too large a step would overshoot.

Sample learned weights (from the professor's Colab walkthrough):

$W_{0}$ (bias) — the base MPG value.
$W_{1}$ (cylinders) — more cylinders → lower MPG (negative weight, as expected).
$W_{2}$ (displacement) — larger engine → lower MPG.
$W_{3}$ (horsepower) — more power → lower MPG.
$W_{4}$ (weight) — heavier car → lower MPG (largest negative weight — weight dominates fuel consumption).
$W_{5}$ (acceleration) — faster acceleration → lower MPG. This holds with all other features constant. Faster cars consume more fuel.
$W_{6}$ (model year) — newer car → higher MPG (positive weight — fuel efficiency improved over decades).
$W_{7}$ (origin) — Japanese/European cars → higher MPG than American cars (historical trend).

Interpreting negative weights: A negative weight means "as this feature increases, MPG decreases." For example, a one-unit increase in standardized weight predicts a decrease in MPG equal to the weight's magnitude — assuming all other features stay the same. This is the ceteris paribus (all else equal) interpretation of linear regression weights.

4.12.6 Visualizations Produced

Loss curve: x-axis = iteration number (0 to 1000), y-axis = MSE. Starts high (around 300) and drops steeply in the first ~100 iterations, then flattens to converge near 5–6. The rapid early improvement confirms gradient descent is working. The flattening tells you the model has approached the minimum — further iterations yield diminishing returns.

Predicted vs. Actual plot: Scatter plot with actual MPG on x-axis and predicted MPG on y-axis. A red diagonal line ( $y = x$ ) marks perfect prediction. Blue dots are individual test vehicles. The vertical distance from each dot to the red line is the residual. Dots clustered tightly around the diagonal → good predictions. Dots scattered far from the diagonal → poor predictions.

Residual plot: y-axis = residual ( $Y^_{i} - Y_{i}$ ), x-axis = predicted MPG. A horizontal red line at residual = 0 marks the ideal — every prediction equals the actual value. Residual points above the line are over-predictions; below the line are under-predictions. Look for: random scatter (good — no pattern in errors), funnel shape (problem — variance increases with prediction magnitude), or curved pattern (problem — model is missing a nonlinear relationship).

Weight visualization: Bar chart with feature names on x-axis and learned weight values on y-axis. Positive bars (blue) = features that increase MPG. Negative bars (red) = features that decrease MPG. Bar height = magnitude of influence. The tallest bar tells you the most influential feature.

Q: Why is the red line in the residual plot always at zero? A: The red line at zero represents the ideal — a perfect model where every prediction equals the actual value. It is a reference line, not a data line. You want all your residual points as close to this zero line as possible. A residual of zero means that instance was predicted perfectly.

4.12.7 Exercise

Tweak the learning rate to different values (the provided code uses 0.01). Observe how it affects:

Convergence speed: higher rates converge faster but risk overshooting.
Final accuracy: if the rate is too high, the loss oscillates and never settles.
Stability: if the rate is too low, convergence is slow but smooth.

Try implementing a convergence threshold instead of hard-coding 1000 iterations: stop when the change in loss between iterations drops below a small value (e.g., $1 0^{- 6}$ ). Compare the final weights and number of iterations needed.

Exam note: In assignments, you will build custom regression functions from scratch — do not use prebuilt libraries like scikit-learn's LinearRegression. Pay close attention to how each component (prediction, loss, gradient, update) is implemented. Know the shape of every matrix at each step. The $1/ N$ averaging in the gradient confirms batch gradient descent.

Scope — batch vs. mini-batch in the Colab code: The Colab implementation uses batch gradient descent — the gradient is averaged over all $N$ training instances. Mini-batch and SGD variants would require restructuring the gradient computation to operate on subsets of the data. The batch approach is simplest and works well for datasets of this size (~300 training examples).

Pitfalls:

Data leakage through standardization. Compute $μ$ and $σ$ on training data only. Do not peek at the test set during preprocessing.
Not prepending the bias column of ones. If you forget and use only the raw features, the model has no bias term and is forced through the origin. Every prediction for an input of all zeros would be zero.
Using np.mean for MSE when the formula uses $\frac{1}{2 N}$ . Both work — the 2× difference is absorbed by the learning rate. But be consistent: if your gradient derivation assumed $\frac{1}{2 N}$ MSE, then your gradient formula already accounts for the 1/2 cancellation. Using np.mean (which is $\frac{1}{N}$ ) changes the effective learning rate by a factor of 2.
Forgetting to standardize test data with training statistics. The test data must be transformed using the same $μ$ and $σ$ from training. If you fit a new scaler on test data, the features are on a different scale and the learned weights produce garbage predictions.
Printing loss every iteration instead of every 100th. For 1000 iterations, printing every one floods the output. Print every 100th (or 50th) for readability.

Recap + Bridge: We implemented a single-neuron regression model from scratch — data loading, preprocessing, standardization, forward pass, loss, gradient, weight update, and evaluation on test data. The same structure (class with predict, loss, gradient, fit, test methods) extends to any model you build. Next, we formalize the distinction between the numbers the model learns (parameters) and the numbers you choose (hyperparameters).

Real-World & Domain Connection: The Auto MPG dataset comes from the StatLib library at Carnegie Mellon University and was used in the 1983 American Statistical Association Exposition. It is a classic benchmark for regression. In modern automotive engineering, similar regression models predict fuel efficiency during vehicle design — engineers tweak engine parameters in simulation and use regression to estimate MPG before building a physical prototype. The U.S. Environmental Protection Agency (EPA) uses more sophisticated models to certify fuel economy ratings, but the core principle — predict a continuous output from engineered features — remains the same. The preprocessing pipeline you just built (handle missing values, remove irrelevant features, standardize, split) is identical to what production ML systems run on every new dataset.

4.13 Hyperparameters and Parameters

4.13.1 Key Distinction

Parameters are the weights ( $W_{0}, W_{1}, \dots, W_{d}$ ) that the machine learns from data. They are fixed after training completes. Hyperparameters are the knobs you set before training begins — they control how the machine learns, not what it learns. They are never updated by the gradient descent loop.

Hook: Imagine you are teaching a child to shoot a basketball. The child adjusts their arm angle, release timing, and jump height as they practice — these are parameters they learn through trial and error. But you, the coach, decide how many practice shots they take per session, how fast you correct their form, and when to stop the drill. Those coaching decisions are hyperparameters.

Intuition + Analogy: Think of training a neural network like baking a cake. The flour, sugar, and eggs are the parameters — the machine combines them to produce the final model. The oven temperature, baking time, and mixing speed are the hyperparameters — you set them before the process starts, and they determine whether the cake rises properly. If the cake flops, you do not change the flour; you adjust the oven temperature (the hyperparameter).

Formalize: In linear regression with gradient descent, the update rule is:

$W_{j}^{(t + 1)} = W_{j}^{(t)} - η \cdot \frac{\partial J}{\partial W _{j}}$

Here:

$W_{0}, W_{1}, \dots, W_{d}$ are parameters. Updated every iteration by the gradient step.
$η$ (learning rate) is a hyperparameter. You choose it once, and it never changes during training (unless you use a learning rate schedule — itself another hyperparameter choice).
The number of iterations $T$ (or epochs) is a hyperparameter. You decide when to stop.
The convergence threshold $ε$ is a hyperparameter. Training halts when $∣ J^{(t + 1)} - J^{(t)} ∣ < ε$ .

The word "hyperparameter" literally means "above the parameters." It describes numbers that sit outside the model's learning mechanism and govern the training process itself. The enrichment text (Section 3.1.1) defines them as: "tunable parameters that are not updated in the training loop are called hyperparameters."

Worked Example: Suppose you train a perceptron with $η = 0.01$ , 1000 iterations, and $ε = 1 0^{- 6}$ .

Role	Value	Updated by gradient descent?	Set by you?
Parameter	$W_{0}, W_{1}, W_{2}$	Yes	No
Hyperparameter	$η = 0.01$	No	Yes
Hyperparameter	$T = 1000$	No	Yes
Hyperparameter	$ε = 1 0^{- 6}$	No	Yes

After training, the model stores only the final weights. The hyperparameters are discarded — they were just the recipe, not the result.

Q: Does hyperparameter tuning include both weight revision and learning rate adjustment? A: No. Hyperparameter tuning only adjusts the knobs ( $η$ , $T$ , $ε$ , etc.). The weights are parameters — they are revised automatically during training by gradient descent. When you perform a grid search over $η$ values, you are tuning a hyperparameter. When gradient descent updates $W_{j}$ each iteration, that is parameter learning.

Common Pitfall: Students often confuse "tuning" a model with training it. Training means running gradient descent to find the best weights. Tuning means trying different hyperparameter values across multiple training runs (cross-validation) to find the best configuration. They are distinct and sequential steps.

Visual Intuition: Imagine a dashboard with two groups of dials:

Bottom row (Parameters): These dials move on their own as the machine runs. You cannot touch them directly. They settle into final positions when training stops.
Top row (Hyperparameters): These are the dials you manually twist before pressing Start. They never move during the run.

Your job as a practitioner is to find the right top-row settings so the bottom-row dials converge to a good solution.

Assumptions & Scope: This distinction holds for all supervised learning models trained with gradient-based optimization. It applies whether you have 3 features or 3 million. The concepts of parameter and hyperparameter are universal across linear regression, logistic regression, neural networks, SVMs, and decision trees — though what counts as a hyperparameter varies by model class.

Real-World & Domain Connection: In industry, hyperparameter tuning is often automated through tools like Optuna, Hyperopt, or AWS SageMaker automatic model tuning. These tools run hundreds of training jobs with different hyperparameter combinations and pick the best one. The machine learning engineer defines the search space (which hyperparameters to tune and their ranges), and the tool handles the rest. This is called AutoML.

Exam note: The distinction between parameters and hyperparameters is a classic exam question. You should be able to: (1) define each term, (2) give three examples of each, (3) explain who sets them (machine vs. human), and (4) state when each is finalized (during training vs. before training). A one-sentence answer: Parameters are learned from data by the algorithm; hyperparameters are set by the practitioner before learning begins.

Recap + Bridge: Parameters = what the model learns. Hyperparameters = how the model learns. You do not train hyperparameters; you choose them. You do not choose parameters; the model learns them. Next, we turn to the data itself: can we improve results by carefully selecting which features to include? That is feature engineering.

4.14 Feature Engineering Discussion

4.14.1 Reducing Redundant Features

Feature engineering is the preprocessing step where you select, transform, or create input features before feeding them to a model. In linear models, each feature adds one weight — so redundant features waste computation without adding information.

Hook: Suppose you are filling out a form that asks for your weight in kilograms and your weight in pounds. Both fields capture exactly the same information — one is just a constant multiple of the other. Including both is wasteful. This is exactly what happens in a dataset when two features are perfectly correlated.

Intuition + Analogy: Imagine packing a suitcase for a trip. Every item costs you weight and space. If you pack two identical pairs of shoes, you double the burden without gaining any new outfit options. Redundant features are duplicate shoes — they make the model heavier (more weights) and slower to train, but add zero new predictive power.

Formalize: In linear regression with $d$ features, the model has $d + 1$ weights (including bias). Gradient descent must compute $d + 1$ partial derivatives per iteration. If two features $X_{i}$ and $X_{j}$ are perfectly correlated (correlation coefficient $ρ_{ij} = 1$ ), their weight updates carry redundant information. Removing one feature reduces the weight count without losing information.

Worked Example: You have a housing dataset with features: area_sqft, area_sqm, num_bedrooms, num_bathrooms, age_years.

Step	Action	Reason
1	Compute correlation matrix	Find redundant pairs
2	Spot $ρ$ (area_sqft, area_sqm) = 1.0	Perfect linear relationship
3	Drop `area_sqm`	It adds no new information
4	Retain `area_sqft`	Either one works; pick one
5	Check $ρ$ (num_bedrooms, num_bathrooms) = 0.86	High but not perfect — keep both

Result: 5 features reduced to 4. Training time drops because one fewer gradient needs computing per iteration.

Method 1 — Correlation plot: Build a correlation matrix comparing every feature against every other feature. Visualize as a heatmap. If two features have a correlation of exactly 1 (or extremely close), they are redundant. Keep one, drop the other.

Correlation range	Interpretation	Action
$0.0$ to $0.3$	Weak	Keep both
$0.3$ to $0.7$	Moderate	Keep both, monitor
$0.7$ to $0.95$	Strong	Consider dropping one
$0.95$ to $1.0$	Near-perfect	Drop one

Method 2 — Random Forest feature importance: Train a Random Forest on your data. Each feature receives an importance score based on how much it reduces impurity across all trees. Features with near-zero importance can be dropped. This captures nonlinear relationships and interactions that correlation may miss. This technique is covered in a parallel machine learning course (post-midterm).

Q: Can we ignore redundant features to reduce computation? A: Yes. Use correlation plots for linear redundancy and Random Forest importance for general redundancy. Feature engineering happens before neural network training — it is preprocessing. Several students asked this. The goal is a lean feature set: include everything the model needs and nothing it does not.

Pitfalls:

Moderate correlation is not redundancy. If $ρ = 0.8$ , the features are related but each may carry unique signal. Dropping one can hurt accuracy.
Domain knowledge matters. A doctor might tell you that blood_pressure_systolic and blood_pressure_diastolic are both essential despite being correlated — each captures a different physiological phenomenon.
Interaction effects are invisible to correlation. Two features may have $ρ \approx 0$ individually but together enable the model to capture important patterns. Dropping either one destroys that interaction.
The target matters. A feature with low correlation to other features may still correlate strongly with the target variable. Always check feature–target correlation before dropping.
With very few samples, even weak features can help regularize the model. Stripping down to the absolute minimum can backfire.

Visual Intuition: Imagine a scatterplot matrix. Redundant features appear as near-perfect diagonal lines. Non-redundant features appear as scattered clouds. Your job is to keep exactly one feature from each "nearly-diagonal" pair.

Assumptions & Scope: Correlation-based screening assumes linear relationships between features. If features relate nonlinearly (e.g., $X_{2} = X_{1}^{2}$ ), correlation may be low even though one feature determines the other. Random Forest importance handles this case better. Feature engineering is model-agnostic: the same reduced feature set can feed a linear model, an MLP, or an XGBoost model.

Real-World & Domain Connection: In credit scoring, raw datasets often contain 500+ features derived from transaction history. Banks routinely prune these to 30–50 features using correlation analysis and feature importance. This cuts model training time from hours to minutes and makes the model explainable to regulators. In deep learning, it is generally assumed that a well-engineered feature set is provided — feature engineering is treated as preprocessing done before the neural network is trained.

Recap + Bridge: Every feature costs one weight and one partial derivative per iteration. Drop redundant features before training — but use correlation thresholds and domain knowledge together, never correlation alone. With feature engineering covered, we now confront the fundamental limitation of this entire approach: a single perceptron can only draw straight lines. What do we do when the world is curved?

4.15 Limitations and What Comes Next

4.15.1 Summary and Transition

A single-perceptron linear neural network can only model linear relationships. To capture nonlinear patterns — curves, clusters, decision boundaries that bend — you need a multilayer perceptron (MLP): multiple neurons organized into hidden layers. Everything in the coming lectures builds on the foundations laid in this one.

Hook: Try to separate blue dots from red dots on a piece of paper using a single straight cut of scissors. If the dots form two neat clusters far apart, one straight cut works perfectly. But what if the red dots form a ring surrounding a blue cluster in the center? No single straight line can separate them. You would need to bend your cut. That is exactly the limitation of a single perceptron: it can only draw straight lines.

Intuition — Why We Need MLPs: A single perceptron computes:

$y^= f (W_{0} + W_{1} X_{1} + W_{2} X_{2} + \dots + W_{d} X_{d})$

This is a linear combination of inputs passed through an activation function. Even with a nonlinear activation like sigmoid, the decision boundary — the set of points where the output crosses a threshold — is always a straight line (a flat hyperplane).

An MLP breaks through this limitation by stacking layers:

$y^= f_{out} (W^{(2)} \cdot f_{hidden} (W^{(1)} \cdot X + b^{(1)}) + b^{(2)})$

Each hidden neuron computes its own linear combination. When you combine multiple neurons and apply nonlinear activations, the network can approximate any continuous function — including curves, spirals, and complex decision boundaries. This is the Universal Approximation Theorem.

Formalize the limitation: For a binary classifier with a single perceptron, the decision boundary is:

${X \in R^{d} ∣ W_{0} + W_{1} X_{1} + \dots + W_{d} X_{d} = 0}$

This is the equation of a $(d - 1)$ -dimensional hyperplane. No matter what weights you learn, the boundary is always flat. It cannot curve, loop, or form disconnected regions.

The XOR Problem — Canonical Example: Four points: (0,0) and (1,1) are class A; (0,1) and (1,0) are class B. A single perceptron cannot separate these classes — no straight line exists that puts (0,0) and (1,1) on one side with (0,1) and (1,0) on the other. Add one hidden layer with two neurons and a nonlinear activation, and the network solves it instantly. The XOR problem was the original motivation for multilayer networks — Minsky and Papert's 1969 book "Perceptrons" proved this limitation, which contributed to the first AI winter.

Real-World: Where Linear Models Fail and MLPs Succeed:

Problem	Why nonlinear	How MLP helps
Image classification	Pixel patterns have no linear structure	Hidden layers learn hierarchical features
Speech recognition	Sound waves are nonlinear over time	Layers capture temporal dependencies
Fraud detection	Fraud involves complex feature interactions	Hidden layers model interactions automatically
Medical diagnosis	Symptoms interact in nonlinear, conditional ways	MLPs learn conditional dependencies
Stock prediction	Markets show nonlinear dynamics	Deep networks model complex patterns

Visual Intuition: Imagine you are sculpting clay. A single perceptron gives you one flat paddle — you can only press flat surfaces. An MLP gives you a set of tools: fingers, knives, loop tools. By layering these tools (hidden layers), you can sculpt arbitrary shapes. Each hidden layer bends the input space a little more; after enough layers, any shape becomes possible.

What We Have Built So Far:

Section	Concept	Why it transfers to MLPs
4.1–4.4	Linear model, perceptron, activation	The building block neuron — MLPs just use more of them
4.5	Loss functions (MSE)	Same loss works for any regression output
4.6	Gradient descent	The training algorithm never changes — just more parameters
4.7	Convergence criteria	Same stopping rules apply
4.8	Cost visualization	Intuition extends (though surfaces get more complex)
4.9	Worked example	The hand-calculation skills transfer directly
4.10	Computational graphs	The chain rule scales to arbitrary depth — this is backprop
4.11	Model evaluation	Train/test split and metrics are model-agnostic
4.12	Implementation	The predict/loss/gradient/fit class pattern is universal
4.13	Parameters vs. hyperparameters	Same distinction, same tuning workflow
4.14	Feature engineering	Preprocessing is always required, regardless of model depth

If you understand gradient descent for one neuron (Section 4.6), you are 80% of the way to understanding backpropagation. Backpropagation is simply gradient descent applied to a chain of functions — the chain rule from calculus, computed layer by layer from output back to input.

Bridge to Next Lecture: The next module introduces the multilayer perceptron. You will learn:

How to stack neurons into hidden layers — width (neurons per layer) and depth (number of layers).
Why nonlinear activation functions (ReLU, tanh, sigmoid) in hidden layers are essential — without them, stacking layers collapses back into a single linear transformation, no matter how deep.
The backpropagation algorithm — how gradient descent computes weight updates across multiple layers using the chain rule through the computational graph.
How depth and width trade off — when to go deeper vs. wider.

Scope — staying linear: This entire module intentionally stays within the linear regime. Linear models cannot learn curves, but they are the foundation. Do not skip ahead — every concept here (gradients, loss, convergence, evaluation) is a prerequisite for deep networks. Students who struggle with backpropagation usually have a weak grasp of gradient descent on a single neuron. Master this module first.

Recap + Bridge: A single perceptron is a straight-line thinker. The world is curved. MLPs bend the line. Everything you learned about gradients, loss, and convergence still applies — you just need more neurons to capture the complexity of real data. See you in the next module: Multilayer Perceptrons.

Real-World & Domain Connection: The limitation of linear models was most famously shown by Minsky and Papert in their 1969 book "Perceptrons," which proved that a single-layer perceptron cannot solve the XOR problem. This result, combined with the lack of a training algorithm for multilayer networks at the time, triggered the first "AI winter" — a decade of reduced funding and interest in neural networks. The winter thawed in 1986 when Rumelhart, Hinton, and Williams published the backpropagation algorithm, showing how to train multilayer networks. Today, every deep learning model — from GPT-4 to AlphaFold — traces its lineage back to that breakthrough: stacking perceptrons and training them with gradient descent. The single neuron you studied in this module is the atom; the universe of deep learning is built from billions of them, connected in layers, trained by the same update rule you learned here.

4.16 Exam Guidance Summary

These are the key concepts and skills the exam is likely to test. Review each one against the relevant section.

Traditional ML vs. neural network framing of linear regression (§4.3). Same math, different training paradigms. Know when each approach is used.

Why thresholding is not used for regression (§4.2). Regression outputs need to be continuous real numbers. Thresholding produces binary values — it destroys the continuous output you need.

Identity activation function (§4.4.1): $f (z) = z$ , derivative $\frac{df}{d z} = 1$ . Passes the signal through unchanged. Used for unbounded regression. Gradient flows unimpeded during backpropagation.

ReLU activation function (§4.4.2): $f (x) = max (0, x)$ . Clips negatives to zero while passing positives unchanged. Used when outputs must be non-negative (e.g., prices, counts). Derivative: 0 for $x < 0$ , 1 for $x > 0$ . Be aware of the "dying ReLU" problem — neurons that always output zero receive no gradient and stop learning.

Weight update rule for gradient descent with MSE (§4.6.2): $W_{j}^{new} = W_{j}^{old} - η \cdot \frac{\partial J}{\partial W _{j}}$ . Be able to derive the gradient formulas for both bias ( $\frac{1}{N} \sum (Y^_{i} - Y_{i})$ ) and feature weights ( $\frac{1}{N} \sum (Y^_{i} - Y_{i}) \cdot X_{ij}$ ). Know the $\frac{1}{2 N}$ vs. $\frac{1}{N}$ MSE convention and how it affects the gradient.

Batch vs. mini-batch vs. stochastic gradient descent (§4.6.3–4.6.5). Know the differences in data usage, gradient quality, speed, and when to use each. Batch uses all $N$ instances. Mini-batch uses $B$ instances. Stochastic uses 1. The averaging step changes accordingly.

Computational graphs (§4.10). Be prepared to draw a graph with labeled nodes and edges for a given model. Annotate forward flow (data) and backward flow (gradients). Note whether the problem specifies batch, mini-batch, or stochastic — the averaging step differs.

Build custom regression classes from scratch (§4.12). In assignments, do not use prebuilt libraries. Implement predict, compute_loss, compute_gradient, fit, and test methods yourself. Pay attention to matrix shapes and the $1/ N$ averaging factor.

Convergence criteria (§4.7). Know all three: cost = 0 (rare), cost starts increasing (overshoot detected — revert to best weights), and tolerance threshold (stop when cost < $ε$ ).

Evaluation metrics (§4.11). MSE, RMSE (same units as target), MAE (outlier-resistant), and $R^{2}$ . Interpret $R^{2}$ : 1 = perfect fit, 0 = no better than predicting the mean, < 0 = worse than the mean. Always evaluate on test data, not training data.

Parameters vs. hyperparameters (§4.13). Parameters are learned weights ( $W_{j}$ ). Hyperparameters are set by you ( $η$ , $T$ , $ε$ , batch size). They are never updated by gradient descent.

Feature standardization (§4.12.2). Transform features to mean 0, std 1. Prevents large-scale features from dominating gradients. Fit scaler on training data only — then transform test data with the same statistics. Never fit on the combined dataset before splitting.

Single-layer limitation (§4.15). A single perceptron can only model linear relationships. To capture nonlinear patterns, you need hidden layers with nonlinear activations. This is the motivation for multilayer perceptrons — the topic of the next module.

4.17 Key Industry Applications

House price prediction: The canonical regression use case. Zillow's Zestimate uses similar regression principles at scale, combining hundreds of property features with recent sale prices. Automated Valuation Models (AVMs) in real estate rely on linear regression and its variants to produce instant property valuations.

Fuel efficiency prediction (Auto MPG): Regression on automotive data with multiple continuous features. Modern applications include EPA fuel economy certification, automotive design optimization (predicting MPG during the design phase before building prototypes), and fleet management systems that forecast fuel costs.

Stock price prediction: Regression on financial time series. While pure linear models struggle with market nonlinearity, they serve as baselines and are used in pairs trading, risk factor modeling (Fama-French models), and the Capital Asset Pricing Model (CAPM).

Temperature forecasting: Regression on meteorological features (humidity, pressure, wind speed, historical temperatures). Weather services use ensemble methods that include linear regression as a component model.

Crop yield prediction: Regression on environmental features (rainfall, fertilizers, temperature, soil quality). Used by agricultural technology companies and government agencies to forecast food production and optimize resource allocation.

Natural Language Processing: When vocabulary words become features (millions of dimensions for a full dictionary), batch gradient descent is computationally infeasible. Stochastic gradient descent processes one example at a time, making it possible to train models on text corpora with massive feature spaces. This was critical for early word embedding models like word2vec.

Optimization algorithms (Adam, AdaProp): These adaptive optimizers are industry-standard in all modern deep learning frameworks (TensorFlow, PyTorch, JAX). They build on gradient descent by adapting the learning rate per-parameter based on gradient history. While covered in a later module, they are the default choice for training deep networks — the direct descendants of the simple gradient descent you learned in this module.

Beyond the examples above: Linear regression (and its neural-network framing) serves as the output layer of virtually every deep regression model. The final layer of a neural network that predicts a continuous value is, mathematically, a linear regression. Every deep learning practitioner uses the concepts from this module — loss functions, gradient descent, train/test splits, standardization — every single day, regardless of how deep their networks go.

DNN Lecture 04 notes · Linear Neural Networks for Regression

Deep Neural Networks· postgraduate· 2026-07-13

Sections Breakdown

1Linear Regression — Supervised Learning Recap

Predicting a continuous value as a weighted sum of features plus a bias; the simplest neural network.

2Single Perceptron — Core Operation Review

Weighted sum followed by an activation; thresholding vs continuous transformation for regression.

3Linear Neural Networks vs Traditional Linear Regression

Same equation, two framings: closed-form normal equation versus iterative gradient descent.

4Activation Functions

Identity for unbounded regression and ReLU for non-negative outputs; choosing the output activation.

5Objective Functions — Loss and Cost

Mean Square Error and Mean Absolute Error, their trade-offs, and why convexity matters.

6Gradient Descent for Linear Regression

Deriving the weight update rule and batch, mini-batch, and stochastic variants.

7Convergence Criteria

When to stop training: zero loss, increasing loss, tolerance threshold, or fixed iterations.

8Cost Function Visualization

The convex error bowl in 1D, 2D contours, and higher dimensions.

9Worked Example: Batch Gradient Descent with Three Instances

A hand-computed batch gradient descent run on three houses.

10Computational Graphs

Forward and backward passes as a graph; the chain rule made explicit.

11Model Evaluation

Train/test split, MSE, RMSE, MAE, and R-squared.

12Implementation: Single Neuron Regression on Auto MPG Dataset

A from-scratch Python implementation with preprocessing and evaluation.

13Hyperparameters and Parameters

What the model learns versus what you set before training.

14Feature Engineering Discussion

Reducing redundant features before training.

15Limitations and What Comes Next

Why a single perceptron is linear and the motivation for multilayer perceptrons.

Postgraduate students in Deep Neural Networks and Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Linear Regression for Continuous Prediction

Must-know: Linear regression predicts one continuous value as a weighted sum of features plus a bias: $y^= w^{⊤} x + b$ . It is the simplest neural network — one neuron, no hidden layers.

$y^= w_{1} x_{1} + w_{2} x_{2} + \dots + w_{d} x_{d} + b$

⚠️ Top pitfall: Forgetting the bias term forces the line through the origin; if the true relationship has a nonzero intercept, every prediction is systematically wrong.

Self-check: Given size=1500, rooms=3 with weights 0.2 and 10 and bias 50, what is the predicted price?

Connects to: 4.2 Single Perceptron; 4.3 Linear NN vs Traditional; 4.11 Model Evaluation

Single Perceptron — Weighted Sum and Activation

Must-know: A perceptron computes a weighted sum $Z = \sum W_{i} X_{i}$ then passes it through an activation. For regression the activation is continuous (identity or ReLU), never a threshold.

$Z = W_{0} + i = 1 \sum d W_{i} X_{i}$

⚠️ Top pitfall: Using thresholding for regression destroys the continuous output — thresholding gives binary 0/1, but regression needs real numbers.

Self-check: Why must you NOT threshold the output of a regression perceptron?

Connects to: 4.1 Linear Regression; 4.4 Activation Functions; 4.15 Limitations

Linear Neural Network vs Traditional Linear Regression

Must-know: Same equation $Y = \sum W_{i} X_{i}$ . Traditional ML solves it with the closed-form normal equation; the neural framing uses gradient descent and is built to be extended with hidden layers.

$W^{*} = (X^{⊤} X)^{- 1} X^{⊤} Y (traditional) W \leftarrow W - η \nabla J (neural)$

⚠️ Top pitfall: Thinking a linear neural network can learn curves — it has zero hidden layers and a linear activation, so it is just linear regression in neural clothing.

Self-check: When would you prefer the neural framing over the closed-form solution?

Connects to: 4.1 Linear Regression; 4.6 Gradient Descent; 4.15 Limitations

Identity and ReLU Activations

Must-know: Identity $f (z) = z$ passes the weighted sum through unchanged for unbounded regression; ReLU $f (x) = max (0, x)$ clips negatives to zero for non-negative outputs. Pick the activation to match the output range.

$Identity: f (z) = z ReLU: f (x) = max (0, x)$

⚠️ Top pitfall: Using sigmoid or tanh at a regression output squashes values into a tiny range, so the model can never reach large prices; also ReLU has a zero-gradient region that can kill a neuron.

Self-check: Which activation do you use when the output must never be negative, and why?

Connects to: 4.2 Single Perceptron; 4.12 Implementation

Loss Functions — MSE and MAE

Must-know: MSE squares errors and averages them: $MSE = \frac{1}{2 N} \sum (Y^_{i} - Y_{i})^{2}$ . It is convex and differentiable, so gradient descent is guaranteed to reach the global minimum. MAE uses absolute errors and resists outliers.

$MSE = \frac{1}{2 N} i = 1 \sum N (Y^_{i} - Y_{i})^{2}$

⚠️ Top pitfall: Forgetting the $\frac{1}{2}$ is just a convention that cancels the 2 from differentiation; both $\frac{1}{2 N}$ and $\frac{1}{N}$ give the same optimal weights, only the effective learning rate differs.

Self-check: Why does MSE penalize a single large error more than MAE does?

Connects to: 4.6 Gradient Descent; 4.11 Model Evaluation

Gradient Descent Weight Update

Must-know: Gradient descent reduces the loss by stepping each weight opposite to its gradient: $W_{j} \leftarrow W_{j} - η \partial J / \partial W_{j}$ . The learning rate $η$ controls step size.

$W_{j}^{new} = W_{j}^{old} - η \cdot \frac{1}{N} i = 1 \sum N (Y^_{i} - Y_{i}) X_{ij}$

⚠️ Top pitfall: Setting $η$ too high makes the loss oscillate or diverge; too low and convergence takes forever. Also never flip $Y^_{i} - Y_{i}$ to $Y_{i} - Y^_{i}$ — the sign flips and updates go the wrong way.

Self-check: Write the gradient for the bias term $W_{0}$ and explain why it has no $X_{ij}$ factor.

Connects to: 4.5 Objective Functions; 4.9 Worked Example; 4.10 Computational Graphs

Convergence Criteria

Must-know: Stop training when loss hits zero (rare), when loss starts increasing (overshoot — revert to best weights), when loss falls below a tolerance $ε$ , or after a fixed iteration count.

$∣ J^{(t + 1)} - J^{(t)} ∣ < ε \Rightarrow stop$

⚠️ Top pitfall: Setting $ε$ far below the irreducible noise means training never stops; and a single noisy increase in mini-batch SGD is not always a real overshoot.

Self-check: Which convergence criterion does early stopping on a validation set automate?

Connects to: 4.6 Gradient Descent; 4.8 Cost Visualization

Cost Surface Visualization

Must-know: For linear regression with MSE the cost surface is a convex bowl — one global minimum, no traps. Gradient descent is guaranteed to reach it. In 1D it is a U-curve; in 2D a 3D bowl with concentric contours.

$J (w) is convex \Rightarrow one global minimum$

⚠️ Top pitfall: Assuming every loss surface is a bowl — deep networks with nonlinear activations produce non-convex surfaces with local minima and saddle points.

Self-check: On a contour plot, which direction does gradient descent move relative to the contour lines?

Connects to: 4.5 Objective Functions; 4.6 Gradient Descent

Batch Gradient Descent Worked Example

Must-know: On three houses with $η = 0.1$ , batch GD drops the loss from 7.5 to ~1.53 in one iteration. The same forward-pass, loss, gradient, update loop scales to millions of instances.

$Y^= XW, X = 111123$

⚠️ Top pitfall: Forgetting the bias column of ones in $X$ removes the bias term and forces the model through the origin; also mixing up the learning rate used in the problem statement.

Self-check: After iteration 1 with $η = 0.1$ , what are the new weights $W_{0}, W_{1}$ ?

Connects to: 4.6 Gradient Descent; 4.10 Computational Graphs

Computational Graphs and Backprop

Must-know: A computational graph shows operations as nodes and data as edges. Forward pass flows data left to right; backward pass flows gradients right to left via the chain rule. This is how frameworks compute gradients automatically.

$\frac{\partial J}{\partial W} = \frac{\partial J}{\partial Y ^} \cdot \frac{\partial Y ^}{\partial W} = \frac{1}{N} e^{⊤} X$

⚠️ Top pitfall: Forgetting that the backward pass follows the chain rule in reverse order, and omitting the $1/ N$ averaging node for batch/mini-batch GD.

Self-check: For batch GD, what label goes on the averaging node of the computational graph?

Connects to: 4.6 Gradient Descent; 4.9 Worked Example

Model Evaluation — Train/Test Split and Metrics

Must-know: Split data (e.g. 80/20), train on training only, evaluate on held-out test. Report MSE/RMSE for error magnitude, MAE for outlier resistance, and $R^{2}$ for variance explained.

$R^{2} = 1 - \frac{\sum ( Y ^ _{i} - Y _{i} ) ^{2}}{\sum ( Y _{i} - Y ˉ ) ^{2}}$

⚠️ Top pitfall: Evaluating on training data and calling it performance (overfitting hides); and leaking test statistics into standardization before the split.

Self-check: What does $R^{2} < 0$ tell you about your model?

Connects to: 4.5 Objective Functions; 4.12 Implementation

From-Scratch Single-Neuron Implementation

Must-know: Build a class with predict, compute_loss, compute_gradient, fit, and test methods. Prepend a bias column of ones, standardize features, and average the gradient over all N instances (batch GD).

$gradient = \frac{1}{N} X^{⊤} (Y^- Y)$

⚠️ Top pitfall: Data leakage: fit the standardizer on training data only, then transform test with the same statistics. Forgetting the bias column forces the model through the origin.

Self-check: Why must you standardize test data with training mean and std, not its own?

Connects to: 4.6 Gradient Descent; 4.11 Model Evaluation; 4.13 Hyperparameters

Parameters vs Hyperparameters

Must-know: Parameters $W_{j}$ are learned from data by gradient descent. Hyperparameters $η$ , iterations, tolerance, batch size are set by you before training and never updated by the loop.

$W_{j}^{(t + 1)} = W_{j}^{(t)} - η \frac{\partial J}{\partial W _{j}} (η is a hyperparameter)$

⚠️ Top pitfall: Confusing tuning with training — tuning searches hyperparameter values across runs; training finds the weights within one run.

Self-check: Is the learning rate a parameter or a hyperparameter? Who sets it?

Connects to: 4.6 Gradient Descent; 4.12 Implementation

Feature Engineering — Reducing Redundancy

Must-know: Every feature adds one weight and one gradient per iteration. Drop redundant features (correlation ≈ 1) before training, but use domain knowledge and correlation thresholds together — moderate correlation is not redundancy.

$ρ_{ij} = 1 \Rightarrow drop one of X_{i}, X_{j}$

⚠️ Top pitfall: Dropping a feature just because it correlates with another — it may still correlate strongly with the target, or carry interaction effects invisible to correlation.

Self-check: Two features have $ρ = 0.8$ . Should you drop one? Why or why not?

Connects to: 4.12 Implementation; 4.15 Limitations

Single-Perceptron Limitation and MLPs

Must-know: A single perceptron can only draw a straight (hyper)plane decision boundary — it cannot model curves or XOR. Stacking hidden layers with nonlinear activations (an MLP) approximates any continuous function.

${X : W_{0} + i = 1 \sum d W_{i} X_{i} = 0} is a flat hyperplane$

⚠️ Top pitfall: Assuming stacking linear layers alone adds power — without nonlinear hidden activations, many linear layers collapse into one linear transformation.

Self-check: Why can a single perceptron never solve XOR?

Connects to: 4.2 Single Perceptron; 4.4 Activation Functions

Previous LectureLecture 03 · Perceptron Learning and Introduction to Regression

Next LectureLecture 05 · Gradient Descent Variants, Classification, and Evaluation

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Linear Neural Networks for Regression

4.1 Linear Regression — Supervised Learning Recap

4.1.1 Definition and Explanation

4.1.2 Student Questions and Answers

4.2 Single Perceptron — Core Operation Review

4.2.1 Definition and Explanation

4.3 Linear Neural Networks vs Traditional Linear Regression

4.3.1 Definition and Explanation

4.4 Activation Functions

4.4.1 Identity Function (Linear Activation)

4.4.2 ReLU — Rectified Linear Unit

4.4.3 Symbol Registry — Activation Functions

4.4.4 Choosing Activation Functions for Output Layers

4.5 Objective Functions — Loss and Cost

4.5.1 Mean Square Error (MSE)

4.5.2 Mean Absolute Error (MAE)

4.5.3 MSE vs MAE — Comprehensive Comparison

4.5.4 Convexity and Why It Matters

4.6 Gradient Descent for Linear Regression

4.6.1 Why Gradients? Why Not Direct Adjustment?

4.6.2 The Weight Update Rule

4.6.3 Batch Gradient Descent

4.6.4 Mini-Batch Gradient Descent

4.6.5 Stochastic Gradient Descent (SGD)

4.7 Convergence Criteria

4.7.1 Definition and Explanation

4.8 Cost Function Visualization

4.8.1 1D Error Surface (One Weight)

4.8.2 3D Bowl and Contours (Two Weights)

4.8.3 Beyond Two Dimensions

4.9 Worked Example: Batch Gradient Descent with Three Instances

4.9.1 Problem Setup

4.9.2 Matrix Representation

4.9.3 Iteration 1 — Full Walkthrough

4.9.4 Iterations 2 and 3

4.9.5 What "Learned Weights" Mean — Interpretation

4.10 Computational Graphs

4.10.1 Definition and Walkthrough

4.11 Model Evaluation

4.11.1 Training Error vs Test Error

4.11.2 Evaluation Metrics

4.11.3 R-squared (R2) Error

4.12 Implementation: Single Neuron Regression on Auto MPG Dataset

4.12.1 Dataset Description

4.12.2 Preprocessing Steps

4.12.3 Model Architecture

4.12.4 Class Implementation (Python Walkthrough)

4.12.5 Results

4.12.6 Visualizations Produced

4.12.7 Exercise

4.13 Hyperparameters and Parameters

4.13.1 Key Distinction

4.14 Feature Engineering Discussion

4.14.1 Reducing Redundant Features

4.15 Limitations and What Comes Next

4.15.1 Summary and Transition

4.16 Exam Guidance Summary

4.17 Key Industry Applications

DNN Lecture 04 notes · Linear Neural Networks for Regression

Sections Breakdown

Exam Revision Notes

Linear Regression for Continuous Prediction

Single Perceptron — Weighted Sum and Activation

Linear Neural Network vs Traditional Linear Regression

Identity and ReLU Activations

Loss Functions — MSE and MAE

Gradient Descent Weight Update

Convergence Criteria

Cost Surface Visualization

Batch Gradient Descent Worked Example

Computational Graphs and Backprop

Model Evaluation — Train/Test Split and Metrics

From-Scratch Single-Neuron Implementation

Parameters vs Hyperparameters

Feature Engineering — Reducing Redundancy

Single-Perceptron Limitation and MLPs

Was this lecture useful?

BitsNotes AI Assistant

4.11.3 R-squared ( $R^{2}$ ) Error