Machine Learning

Linear Regression — Complete Lecture Notes

📅 Published: 2026-06-28

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning and Data Science

Linear Regression — Complete Lecture Notes

1. Linear Regression Model

Hook. You have a spreadsheet of house sizes and sale prices. Is there a simple formula that predicts price from size — one that captures the trend without chasing every random fluctuation? Linear regression is that formula.

Intuition + Analogy

Think of fitting a line through a scatter of points the way you'd stretch a rubber band through a cloud of thumbtacks on a board. The rubber band snaps to the position that minimizes the total strain — it passes through the "center" of the points, balancing the tugs from above and below. Linear regression does the same thing mathematically: it finds the one straight line that minimizes the total squared distance to every data point.

Where the analogy breaks: the rubber band can bend if you add enough tacks in a curve. Linear regression cannot — it is strictly a straight line (or a flat hyperplane in higher dimensions). For curves, we use the basis-function trick (Section 10).

Formal Definition

A linear regression model predicts a continuous target $y$ as a weighted sum of input features, plus a bias term. The model has two kinds of numbers:

Parameters (weights, coefficients) — what the model learns from data: $θ_{0}$ (intercept/bias) and $θ_{1}, θ_{2}, \dots$ (slopes).
Features (inputs, predictors) — what you feed in: $x_{1}, x_{2}, \dots$ .

The hypothesis — the prediction function — for one feature is:

h_{θ} (x) = θ_{0} + θ_{1} x

For $D$ features, the hypothesis sums across all of them:

h_{θ} (x) = j = 0 \sum D θ_{j} x_{j} where x_{0} = 1 (always)

In compact vector form: $h_{θ} (x) = θ^{⊤} x$ .

The aim of linear regression: find the values of $θ_{0}, θ_{1}, \dots$ that minimize the cost function.

Notation Note

The symbols $θ_{j}$ , $w_{j}$ , and $β_{j}$ all mean the same thing across different textbooks — the weights. This document uses $θ$ to match the lecture; Bishop (2006) uses $w$ , and the AppD reference uses $ω$ . They are interchangeable.

Symbol Registry — Linear Regression Hypothesis

Symbol	Meaning	LaTeX	Type/Domain
$x$	input feature (e.g., total bill, years of experience)	scalar	real number
$x_{j}^{(i)}$	the $j$ -th feature of the $i$ -th training example	scalar	real number
$y$	target variable / label (e.g., tip amount, salary)	scalar	real number
$θ_{0}$	bias term / intercept	scalar	real number
$θ_{1}$	weight for feature $x_{1}$ / slope	scalar	real number
$θ$	parameter vector	$θ = [θ_{0}, θ_{1}]^{⊤}$	vector in $R^{D + 1}$
$h_{θ} (x)$	hypothesis / predicted value ( $y^$ )	scalar	real number
$N$	number of training examples	scalar	positive integer
$D$	number of features	scalar	positive integer

Assumptions & Scope

Scope. Linear regression assumes a linear relationship between features and target. It works well when:

The relationship is roughly a straight line (or can be made linear via basis functions — see Section 10).
Features are not perfectly correlated with each other (no perfect multicollinearity).
The noise around the line has roughly constant variance (homoscedasticity).

It breaks when the true relationship is fundamentally nonlinear and cannot be captured by the basis functions you choose. It also becomes unreliable when features far outnumber training examples ( $D ≫ N$ ), though regularization helps.

Visual Intuition

Picture a 2D scatter plot. The $x$ -axis is the feature (say, years of experience); the $y$ -axis is the target (salary). The data points form a rising cloud. Linear regression draws the single straight line that cuts through the densest part of that cloud, balancing the distances to points above and below. The vertical distance from any point to the line is the residual — the prediction error for that point.

Pitfalls

Forgetting $x_{0} = 1$ . Every training example needs a constant 1 appended for the bias term. Without it, the model is forced through the origin — a line with zero intercept, which is rarely correct.
Confusing "linear in features" with "linear in parameters." Polynomial regression $y = θ_{0} + θ_{1} x + θ_{2} x^{2}$ is nonlinear in $x$ but linear in $θ$ — it is still a linear model. This distinction matters throughout the course.
Using a single bracket in pandas. df['col'] returns a 1D Series; df[['col']] (double brackets) returns a 2D DataFrame. sklearn expects a matrix (2D) for $X$ .

Recap + Bridge

Linear regression = a weighted sum of features plus a bias. The weights are what we learn. Next: how do we measure "how good" any particular set of weights is? That measurement is the cost function.

Real-World & Domain Connection

Linear regression is the most widely used predictive modeling technique across every quantitative field. Civil engineers use it to estimate bridge load capacity from material properties. Epidemiologists model disease risk from environmental exposures. Economists forecast GDP from interest rates and employment figures. Its simplicity, speed, and interpretability (each coefficient tells you exactly how much the target changes per unit of the feature) make it the first model any data scientist reaches for — and the baseline against which fancier models are compared.

2. Cost Function

Hook. You've drawn a line through your data. How do you put a single number on how bad that line is? And more importantly — how do you find the line that makes that number as small as possible?

Intuition + Analogy

Imagine throwing darts at a target. The cost of a single throw is the squared distance from the bullseye. A throw 2 cm off costs 4; a throw 3 cm off costs 9. Squaring makes wild misses hurt disproportionately — you'd rather have ten 1 cm misses than one 10 cm miss. The cost function does the same for predictions: it squares every error, sums them up, and averages. The line that minimizes this total squared cost is the best-fitting line.

Mapping to darts: the bullseye = the actual $y$ value; the dart = the prediction $y^$ ; the squared distance = $(h_{θ} (x) - y)^{2}$ .

Formal Definition

The cost function (also called the loss function or mean squared error) for linear regression:

J (θ) = \frac{1}{2 N} i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)})^{2}

Every piece of this formula:

$h_{θ} (x^{(i)}) - y^{(i)}$ — the residual (prediction error) for the $i$ -th example.
The square $(\cdot)^{2}$ — makes all errors positive and penalizes large errors heavily.
$\frac{1}{N} \sum$ — averages across all $N$ training examples. Without the $1/ N$ , it's the sum of squared errors (SSE).
The $\frac{1}{2}$ — purely to cancel the 2 that appears when we differentiate. The shape of $J (θ)$ is the same with or without it.

Goal: minimize $J (θ)$ . A smaller cost = predictions closer to actual values.

Shape of the Cost Function

When plotted against $θ_{1}$ (with $θ_{0}$ fixed at 0), $J (θ)$ is a parabola — a symmetric U-shape that opens upward. With both $θ_{0}$ and $θ_{1}$ free, the surface is a 3D convex bowl — like a satellite dish viewed from the side. Viewed from above as a contour map, it appears as concentric ellipses.

The critical property: there is exactly one global minimum — a single lowest point. No local minima to trap you. The center of the smallest ellipse = the optimal $θ_{0}, θ_{1}$ .

This single-minimum property is a major advantage of linear regression's squared-error cost. Other models (neural networks, for instance) have bumpy cost landscapes with many valleys — finding the deepest one is much harder.

Visual Intuition

Imagine a 3D surface plot. The $x$ - $y$ floor is $θ_{0}$ and $θ_{1}$ ; the height $z$ is $J (θ)$ . The surface is a smooth, perfectly round bowl. Drop a marble anywhere on the inner wall — it rolls straight to the bottom and stops. No ridges, no false bottoms. Now look straight down: you see nested ellipses. Each ellipse is a "contour" — all $(θ_{0}, θ_{1})$ pairs that give the same cost. The smallest, innermost ellipse is the global minimum. Gradient descent (Section 5) takes steps perpendicular to these contours, heading for the center.

Worked Example

Data: Three points: $(x = 1, y = 2)$ , $(x = 2, y = 3)$ , $(x = 3, y = 4)$ .

Model: $h_{θ} (x) = θ_{0} + θ_{1} x$ . Let's hand-evaluate the cost for a guess: $θ_{0} = 1$ , $θ_{1} = 1$ (so the line is $y = 1 + x$ ).

$i$	$x$	$y$	$y^= 1 + x$
1	1	2	2
2	2	3	3
3	3	4	4

Sum of squared errors = 0. $J (θ) = \frac{1}{2 \cdot 3} \cdot 0 = 0$ . Cost = 0 — perfect fit (the points lie exactly on a line).

Now try a worse guess: $θ_{0} = 0$ , $θ_{1} = 1$ (line: $y = x$ ).

$i$	$x$	$y$	$y^$	Residual	Squared
1	1	2	1	$- 1$	1
2	2	3	2	$- 1$	1
3	3	4	3	$- 1$	1

Sum = 3. $J (θ) = \frac{1}{2 \cdot 3} \cdot 3 = 0.5$ . Cost = 0.5 — worse than the perfect fit, as expected.

Sense-check: The cost is zero for the line that passes through all points, and positive for any other line.

Pitfalls

Forgetting the $1/ N$ . Without it, the cost grows with the dataset size — you can't compare costs across datasets of different sizes.
Confusing cost function with evaluation metric. The cost function (with the $1/2$ ) is what you minimize during training. Evaluation metrics like MSE, MAE, RMSE (Section 9) are what you report after training. The $1/2$ factor is for mathematical convenience in training, not for reporting.
Thinking zero cost is always achievable. Real data has noise. A cost of exactly zero usually means you've memorized the training data (overfitting), not that you've found the true pattern.

Symbol Registry — Cost Function

Symbol	Meaning	LaTeX	Type/Domain
$J (θ)$	cost function	$J (θ_{0}, θ_{1})$ or $J (θ)$	scalar $\geq 0$
$h_{θ} (x^{(i)})$	prediction for example $i$	scalar	real number
$y^{(i)}$	actual target for example $i$	scalar	real number
$N$	number of training examples	scalar	positive integer
residual	$h_{θ} (x^{(i)}) - y^{(i)}$	scalar	real number

Student Q&A

Q: Why is there a $1/2$ in the cost function but not in MSE?

A: The $1/2$ exists purely to cancel the 2 that comes from differentiating the square. When you take the derivative of $(h - y)^{2}$ , the chain rule gives a factor of 2. The $1/2$ kills it, leaving a cleaner gradient: $\frac{1}{N} \sum (h - y) x_{j}$ . MSE (used for reporting, not training) drops the $1/2$ because it's the actual average squared error you care about interpreting.

Recap + Bridge

The cost function is the squared-error bowl — one global minimum, no traps. Minimizing it finds the best line. Next: how do we actually find that minimum? Two approaches — a one-shot algebraic solution (closed form) and an iterative hill-descent (gradient descent). But first, we need the math of slopes: derivatives.

3. Derivatives, Maxima, and Minima

Hook. You're driving a car. The speedometer tells you how fast you're going right now — that's a derivative. Now imagine the road goes up and down hills. The tops and bottoms of hills are exactly where the road is momentarily flat — the derivative is zero. Finding those flat spots is how we locate minima (and maxima) of any function.

Intuition + Analogy

Roll a marble on a curved track. At any point, the track's slope tells the marble which way to roll. On a downhill slope, the marble rolls right. On an uphill slope, it rolls left. At the very bottom of a valley, the track is flat — the marble stays put. The derivative is the slope at a single point. Setting the derivative to zero finds every flat spot — every candidate for a minimum or maximum. The second derivative tells you whether that flat spot is a valley bottom (minimum) or a hilltop (maximum).

Formal Definition

A derivative $\frac{d y}{d x}$ measures the instantaneous slope — how much $y$ changes per unit change in $x$ , at a single point.

Condition	Interpretation
$\frac{d y}{d x} > 0$	Uphill — slope positive (curve rising left to right)
$\frac{d y}{d x} < 0$	Downhill — slope negative (curve falling left to right)
$\frac{d y}{d x} = 0$	Flat — at a maximum or minimum (stationary point)

Finding stationary points: Set $\frac{d y}{d x} = 0$ and solve for the location.

The second derivative test distinguishes maxima from minima:

$\frac{d ^{2} y}{d x ^{2}} > 0$ → Minimum (concave up / cup-shaped). The slope goes negative → 0 → positive.
$\frac{d ^{2} y}{d x ^{2}} < 0$ → Maximum (concave down / frown-shaped). The slope goes positive → 0 → negative.
$\frac{d ^{2} y}{d x ^{2}} = 0$ → Point of inflection (neither).

For linear regression, the second-derivative test is not needed — the cost function is convex (bowl-shaped), so any stationary point is the global minimum. But for general optimization problems, the test is essential.

Worked Example

Function: $f (x) = x^{2} - 4 x + 3$

Step 1 — First derivative: $f^{'} (x) = 2 x - 4$

Step 2 — Set to zero: $2 x - 4 = 0 \Rightarrow x = 2$ . This is the stationary point.

Step 3 — Second derivative: $f^{''} (x) = 2$ . Since $2 > 0$ , the point $x = 2$ is a minimum.

Minimum value: $f (2) = 2^{2} - 4 (2) + 3 = 4 - 8 + 3 = - 1$ .

Sense-check: $x^{2} - 4 x + 3 = (x - 2)^{2} - 1$ . The smallest possible value of $(x - 2)^{2}$ is 0 (at $x = 2$ ), giving $- 1$ . Correct.

Visual Intuition

Plot $f (x) = x^{2} - 4 x + 3$ . The $x$ -axis is $x$ ; the $y$ -axis is $f (x)$ . The curve is a smiling parabola. It falls from the left, bottoms out at $(2, - 1)$ , then rises to the right. The tangent line drawn at the bottom point is perfectly horizontal. At $x = 1$ , the tangent slopes down ( $f^{'} (1) = - 2$ ); at $x = 3$ , the tangent slopes up ( $f^{'} (3) = 2$ ). The bottom is where the slope crosses from negative to positive — the second derivative captures this transition.

Recap + Bridge

Derivative = slope. Set it to zero to find minima/maxima. The second derivative confirms which one. This is the math behind both the closed-form solution (set derivative to zero, solve directly) and gradient descent (follow the negative derivative downhill, one step at a time).

4. Closed Form Solution (Normal Equation)

Hook. What if you could jump straight to the bottom of the cost bowl in one leap — no sliding, no steps, no guessing a learning rate — just one matrix computation? The normal equation does exactly that.

Intuition + Analogy

Think of the cost bowl from Section 2. At the very bottom, the ground is perfectly flat — the slope is zero in every direction. The closed-form solution sets up the equation "slope = 0 in all directions," then solves it algebraically. It's like using GPS to teleport directly to the lowest point, rather than walking downhill step by step (gradient descent). The price: you need to invert a matrix, which gets expensive when the matrix is huge.

Formal Definition

The normal equation gives the exact $θ$ that minimizes $J (θ)$ in one step:

θ = (X^{⊤} X)^{- 1} X^{⊤} Y

Where:

$X$ is the $N \times (D + 1)$ design matrix — each row is one training example, with $x_{0} = 1$ in the first column.
$Y$ is the $N \times 1$ target vector.
$(X^{⊤} X)^{- 1} X^{⊤}$ is the Moore-Penrose pseudo-inverse of $X$ , denoted $X^{†}$ (Bishop §3.1.1).

Full Derivation

Step 1 — Matrix form of the data.

$Y = [y^{(1)}, y^{(2)}, \dots, y^{(N)}]^{⊤}$ — an $N \times 1$ vector.
$X$ is $N \times (D + 1)$ . The first column is all 1's (for $θ_{0}$ ); remaining columns are feature values.

For one feature and $N$ examples:

X = 11 ⋮ 1 x_{1}^{(1)} x_{1}^{(2)} ⋮ x_{1}^{(N)}_{N \times 2} θ = [θ_{0} θ_{1}]_{2 \times 1}

Step 2 — Predictions in matrix form. $h_{θ} (X) = X θ$ — an $N \times 1$ vector (one prediction per row).

Step 3 — Error vector. $X θ - Y$ — an $N \times 1$ vector of residuals.

Step 4 — Sum of squared errors as a dot product. For any vector $E = [e_{1}, \dots, e_{N}]^{⊤}$ , the sum of squares is $E^{⊤} E = e_{1}^{2} + \dots + e_{N}^{2}$ . So:

i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)})^{2} = (X θ - Y)^{⊤} (X θ - Y)

Thus the cost function in matrix form:

J (θ) = \frac{1}{2 N} (X θ - Y)^{⊤} (X θ - Y)

Step 5 — Expand the quadratic form.

J (θ) = \frac{1}{2 N} [(X θ)^{⊤} (X θ) - (X θ)^{⊤} Y - Y^{⊤} (X θ) + Y^{⊤} Y] = \frac{1}{2 N} [θ^{⊤} X^{⊤} X θ - θ^{⊤} X^{⊤} Y - Y^{⊤} X θ + Y^{⊤} Y]

Now observe: $θ^{⊤} X^{⊤} Y$ is a $1 \times 1$ scalar. The transpose of a scalar is itself. So:

(θ^{⊤} X^{⊤} Y)^{⊤} = Y^{⊤} X θ

Since a scalar equals its own transpose, $θ^{⊤} X^{⊤} Y = Y^{⊤} X θ$ . Therefore:

J (θ) = \frac{1}{2 N} [θ^{⊤} X^{⊤} X θ - 2 θ^{⊤} X^{⊤} Y + Y^{⊤} Y]

Step 6 — Take the derivative and set to zero. Drop the $\frac{1}{2 N}$ constant (it doesn't change where the derivative is zero). The term $Y^{⊤} Y$ has no $θ$ — its derivative is 0.

\frac{\partial J}{\partial θ} = 2 X^{⊤} X θ - 2 X^{⊤} Y = 0

Step 7 — Solve for $θ$ .

2 X^{⊤} X θ X^{⊤} X θ θ = 2 X^{⊤} Y = X^{⊤} Y = (X^{⊤} X)^{- 1} X^{⊤} Y

This is the normal equation — the closed-form solution.

Usage Recipe

1. Build $X$ : append a column of 1's (for $θ_{0}$ ) to your feature matrix.

2. Build $Y$ : the target column as a vector.

3. Compute $θ = (X^{⊤} X)^{- 1} X^{⊤} Y$ .

4. Predict: $y^= θ_{0} + θ_{1} x$ (or $y^= X_{new} θ$ in matrix form).

Worked Example

Data: $X = 1111112345$ , $Y = [2, 3, 4, 5, 6]^{⊤}$ .

Step 1 — Compute $X^{⊤} X$ :

X^{⊤} X = [1112131415] 1111112345 = [5151555]

Step 2 — Invert $X^{⊤} X$ :

det = 5 \cdot 55 - 15 \cdot 15 = 275 - 225 = 50

(X^{⊤} X)^{- 1} = \frac{1}{50} [55 - 15 - 15 5] = [1.1 - 0.3 - 0.3 0.1]

Step 3 — Compute $X^{⊤} Y$ :

X^{⊤} Y = [1112131415] 23456 = [2070]

Step 4 — Multiply:

θ = [1.1 - 0.3 - 0.3 0.1] [2070] = [22 - 21 - 6 + 7] = [11]

Result: $θ_{0} = 1$ , $θ_{1} = 1$ . Model: $y = 1 + x$ .

Sense-check: The data is $y = x + 1$ exactly — the model recovers it perfectly. Cost = 0.

Important Caveat — Matrix Invertibility

Not all matrices are invertible. A matrix is singular (non-invertible) when its determinant is zero — meaning its columns are linearly dependent (e.g., duplicate features, or $N < D + 1$ ). In such cases, $(X^{⊤} X)^{- 1}$ does not exist.

In Python: Use numpy.linalg.pinv() — the pseudo-inverse — which handles singular matrices via SVD.
In exams: Only invertible matrices will be given. Manual inversion is expected.
In practice (Bishop §3.1.2): Near-singular $Φ^{⊤} Φ$ causes numerical instability — parameters blow up. Regularization (adding $λ I$ ) fixes this by ensuring invertibility.

Comparison: Closed Form vs. Gradient Descent

Property	Closed Form	Gradient Descent
Iterations	None (one step)	Multiple iterations
Learning rate $α$	Not needed	Must be chosen carefully
Works best for	Small $N$ , small $D$	Large $N$ , large $D$
Matrix inversion	Required — $O (D^{3})$ cost	Not required
Derivative knowledge	Set derivative = 0, solve	Compute gradient each step
Exact solution?	Yes (to numerical precision)	Approximates, given enough iterations

Visual Intuition

In the $N$ -dimensional space where each axis is a target value $t_{n}$ (Bishop §3.1.2), the columns of $X$ span a $D + 1$ dimensional subspace. The prediction vector $Y^= X θ$ must live in that subspace. The least-squares solution is the orthogonal projection of the target vector $Y$ onto that subspace — the point in the subspace closest to $Y$ . The residual vector $Y - Y^$ is perpendicular to every column of $X$ . This geometric picture confirms: the normal equation finds the projection.

Pitfalls

Forgetting the column of 1's. $X$ must have an all-ones column for $θ_{0}$ . Without it, the model is forced through the origin.
Assuming $(X^{⊤} X)^{- 1}$ always exists. Check the determinant. If it's zero (or near-zero), use the pseudo-inverse or regularization.
Using closed form on massive datasets. Inverting a $D \times D$ matrix costs $O (D^{3})$ . For $D$ in the thousands, use gradient descent.

Recap + Bridge

The normal equation jumps to the exact minimum in one matrix computation. Perfect for small-to-medium datasets. Next: gradient descent — the iterative alternative that scales to massive data by never inverting a matrix.

Real-World & Domain Connection

Scikit-learn's LinearRegression class uses the closed-form solution internally (via SVD or QR decomposition, not direct inversion, for numerical stability). It's the default choice for datasets with up to a few hundred features. Beyond that, SGDRegressor (stochastic gradient descent) takes over. The closed form also appears in ridge regression with a simple modification: $θ = (X^{⊤} X + λ I)^{- 1} X^{⊤} Y$ .

5. Gradient Descent

Hook. You're blindfolded, standing on the side of a bowl-shaped valley. Your only tool: you can feel which direction the ground slopes steepest under your feet. How do you find the bottom?

Intuition + Analogy — The Blindfolded Hiker

You are a blindfolded hiker on a foggy hillside. Your goal: reach the lowest point of the valley. Your method:

1. Feel the ground in all four directions (north, south, east, west). Sense which way is steepest downhill.

2. Take one step in that steepest-downhill direction.

3. Repeat from your new position — feel, step, feel, step.

4. Eventually, the ground feels flat in every direction. You've reached the bottom.

This is gradient descent.

Where the analogy breaks: a real hiker can overshoot the bottom if taking too large a step and end up on the opposite slope. Gradient descent has the same problem with a too-large learning rate.

Mapping the Analogy to Math

Hiker Element	Mathematical Equivalent
Your current location	Current values of $θ_{0}$ and $θ_{1}$
Feeling the slope	Computing the gradient (partial derivatives)
Taking one step	Updating the parameters
Size of the step	Learning rate $α$
Reaching flat ground	Convergence — $J (θ)$ is minimized

The Update Rule

For each parameter $θ_{j}$ ( $j = 0, 1, \dots, D$ ):

θ_{j} := θ_{j} - α \cdot \frac{\partial}{\partial θ _{j}} J (θ)

This single formula is applied to every parameter, every iteration. The key pieces:

$α$ — learning rate (step size). Controls how big each step is.
$\frac{\partial}{\partial θ _{j}} J (θ)$ — partial derivative (slope w.r.t. $θ_{j}$ , holding all other parameters constant).
The minus sign — we move opposite to the gradient (downhill, not uphill).
The notation $\partial$ (curly d) vs. $d$ — $\partial$ means "derivative with respect to this one variable, treating others as constants."

Simultaneous Update — Critical Rule

All parameters must be updated simultaneously. Use temporary variables.

Wrong (sequential):

θ₀ = θ₀ - α · ∂J/∂θ₀        ← uses old θ₀, old θ₁
θ₁ = θ₁ - α · ∂J/∂θ₁        ← uses NEW θ₀, old θ₁ — WRONG

Correct (simultaneous):

temp₀ = θ₀ - α · ∂J/∂θ₀
temp₁ = θ₁ - α · ∂J/∂θ₁
θ₀ = temp₀
θ₁ = temp₁

The logic: you must measure the slope in ALL directions from the same position before moving. An already-updated $θ_{0}$ when computing the update for $θ_{1}$ means you're taking the second measurement from a different location — a different point on the cost surface.

Gradient of the Linear Regression Cost Function

Taking the partial derivative of $J (θ) = \frac{1}{2 N} \sum (h_{θ} (x) - y)^{2}$ with respect to $θ_{j}$ :

\frac{\partial}{\partial θ _{j}} J (θ) = \frac{1}{N} i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}

In plain words: "the average, over all training examples, of (prediction error × the $j$ -th feature value)."

Why this simplifies:

1. The square's derivative gives a factor of 2, canceling the $1/2$ in $J (θ)$ .

2. Chain rule: outer derivative of $(X θ - Y)^{2}$ gives $2 (X θ - Y)$ ; inner derivative w.r.t. $θ_{j}$ isolates the $x_{j}$ term (since $\frac{\partial}{\partial θ _{j}} (θ_{0} x_{0} + \dots + θ_{j} x_{j} + \dots) = x_{j}$ ).

3. All other $θ_{k}$ ( $k \neq = j$ ) are treated as constants — their derivatives vanish.

How the Derivative Determines Direction

Slope positive ( $\partial J / \partial θ_{j} > 0$ ): You're on the right side of the bowl. Move left (decrease $θ_{j}$ ). The formula $θ_{j} - α \cdot (positive)$ does this automatically.
Slope negative ( $\partial J / \partial θ_{j} < 0$ ): You're on the left side. Move right (increase $θ_{j}$ ). The formula $θ_{j} - α \cdot (negative) = θ_{j} + α \cdot ∣ slope ∣$ does this automatically.

You never need to check which side you're on — the minus sign in the update rule always moves you downhill.

Learning Rate $α$

$α$ Too Large	$α$ Too Small	$α$ Just Right
May skip over the minimum entirely	Takes forever to converge	Confident, efficient steps
Cost may oscillate or diverge (blow up)	Very slow convergence	Smooth, monotonic descent

Practical guidance: Start with $α \approx 0.01$ . Values like $0.00001$ are too small (thousands of iterations); values like $10$ are too large (divergence). The appropriate range is typically $0.0001$ to $0.1$ , tuned by trial.

Convergence — When to Stop

Stop iterating when parameters stop changing significantly. Measure change using the L2 norm (Euclidean distance) between old and new parameter vectors:

∥ θ_{new} - θ_{old} ∥_{2} = j = 0 \sum D (θ_{j}^{new} - θ_{j}^{old})^{2}

If this norm $< ϵ$ (tolerance, e.g., $0.0001$ ), convergence is reached.

Two practical approaches:

1. Tolerance-based: Stop when $∥ θ_{new} - θ_{old} ∥_{2} < ϵ$ .

2. Fixed iterations: Stop after a preset number of steps (e.g., 1000). Use when coding and the tolerance approach is inconvenient.

The enrichment reference (AppE §E.1.1) uses an equivalent criterion: stop when $∥\nabla f (x) ∥ < ϵ$ — the gradient magnitude is near zero.

Assumptions & Scope

Scope. Gradient descent works when:

The cost function is differentiable everywhere.
The cost function is convex (single minimum) — guaranteed for linear regression with squared error.
The learning rate is chosen appropriately.

It struggles when:

The cost function has many local minima (not an issue for linear regression, but critical for neural networks).
Features have vastly different scales (fix with feature scaling — Section 6).
The dataset is tiny — closed form is faster.

Visual Intuition

Plot the cost function contours (concentric ellipses from Section 2). Starting from some $θ$ , gradient descent draws a path — a sequence of short line segments, each perpendicular to the contour at that point. With a good $α$ , the path spirals smoothly inward to the center. With too-large $α$ , it bounces wildly across contours, possibly spiraling outward. With too-small $α$ , it inches forward with thousands of tiny steps. The contour map makes all three behaviors visible.

Pitfalls

Sequential update instead of simultaneous. The #1 coding bug in gradient descent. Always use temp variables.
Not scaling features. See Section 6 — this distorts the contours into long ellipses, making gradient descent zigzag inefficiently.
Using the same $α$ for all problems. The right learning rate depends on the data scale, the cost function shape, and the optimization variant. Tune it.
Stopping too early. If the cost is still dropping, you haven't converged. Plot $J (θ)$ vs. iteration to diagnose.

Student Q&A

Q: How do you choose between closed form and gradient descent?

A: Closed form for small datasets ( $N$ up to ~10k, $D$ up to ~100) — it's exact and one-step. Gradient descent for large datasets — it avoids the $O (D^{3})$ matrix inversion. Also use gradient descent when you plan to add regularization or when using models (like neural networks) that have no closed form. Scikit-learn's LinearRegression uses closed form; SGDRegressor uses stochastic gradient descent.

Q: Several students asked about initialization — what if no starting $θ$ is given?

A: If initial values are given in a question, use them. If not, initialize to zeros ( $θ = 0$ ) or very small random values. For linear regression's convex cost, initialization doesn't affect which minimum you reach — only how many steps it takes.

Symbol Registry — Gradient Descent

Symbol	Meaning	LaTeX	Type/Domain
$α$	learning rate (step size)	scalar	typically 0.0001 to 0.1
$\frac{\partial}{\partial θ _{j}}$	partial derivative w.r.t. $θ_{j}$	—	operator
$θ_{new}, θ_{old}$	parameter before/after update	vectors	$R^{D + 1}$
$ϵ$	convergence tolerance	scalar	e.g., 0.0001
$∥ \cdot ∥_{2}$	L2 norm (Euclidean length)	—	scalar

Recap + Bridge

Gradient descent = feel the slope, step downhill, repeat. The learning rate controls step size; simultaneous update is mandatory. Next: why feature scaling matters — and how skewed feature ranges turn nice round contours into punishing elongated ellipses.

Real-World & Domain Connection

Gradient descent (and its variants) is the workhorse optimization algorithm behind virtually every deep learning model in production — from ChatGPT to image recognition. The same update rule $θ := θ - α \nabla J$ trains neural networks with millions of parameters. Mastering it on linear regression builds the intuition for everything that follows.

6. Feature Scaling for Gradient Descent

Hook. You're hiking to the bottom of a valley, but the valley is stretched — it's a mile long east-west and only ten feet wide north-south. Every step you take overshoots the narrow dimension. That's gradient descent on unscaled features.

Intuition + Analogy

Imagine a football field tilted toward one corner. If you always walk in the steepest-downhill direction, you'll zigzag: a long diagonal trek across the field, then a tiny correction sideways, then another long diagonal, then another tiny correction. You'd reach the corner eventually, but it would take forever.

Now imagine the same field compressed into a perfect square bowl. Steepest-downhill now points straight to the corner. You walk one smooth line and arrive quickly.

Feature scaling transforms the "stretched football field" cost surface into a "round bowl" — letting gradient descent converge in far fewer steps.

Why Feature Scaling Matters

If features have very different ranges, the cost function contours become elongated ellipses instead of nice circles. Gradient descent takes a zigzag, inefficient path.

Example: Predicting house prices with two features:

$x_{1}$ : house size — ranges from 0 to 2500 sq ft
$x_{2}$ : number of bedrooms — ranges from 1 to 5

A tiny change in $θ_{1}$ causes a huge change in prediction (because $x_{1}$ values are large). The same change in $θ_{2}$ barely moves the prediction. The cost surface is a thousand times more sensitive to $θ_{1}$ than $θ_{2}$ — the contours stretch into long ellipses. Gradient descent oscillates along the long axis instead of gliding smoothly.

With scaling, both features contribute proportionally. Contours become circular, and gradient descent converges directly.

Common Scaling Methods

Normalization (min-max scaling): $x^{'} = \frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}$ — scales to $[0, 1]$ .
Standardization (z-score): $x^{'} = \frac{x - μ}{σ}$ — subtract mean, divide by standard deviation. Result has mean 0, variance 1. Most commonly used.
Decimal scaling: $x^{'} = x /1 0^{k}$ where $k$ is chosen so $max ∣ x^{'} ∣ < 1$ .

Visual Intuition

Two contour plots side by side. Left: unscaled features — elongated ellipses tilted at an angle. The gradient descent path (blue line) zigzags wildly, taking many oscillations to reach the center. Right: scaled features — nearly circular contours. The gradient descent path is a straight, smooth line to the minimum. Both start from the same relative position; the right one converges in a fraction of the iterations.

Pitfalls

Scaling the target $y$ . Feature scaling applies to input features $X$ , not the target $y$ . Scaling $y$ changes the meaning of the parameters and predictions.
Applying scaling after train-test split incorrectly. Fit the scaler on training data only, then apply the same transformation to test data. Never fit on test data — that leaks information.
Forgetting that scaling matters for interpretability. After standardization, $θ_{j}$ means "change in $y$ per standard-deviation change in $x_{j}$ ," not per raw unit. Convert back if you need raw-unit interpretation.

Recap + Bridge

Feature scaling turns elongated cost contours into round ones — gradient descent converges faster and more reliably. Standardization (z-score) is the go-to method. Next: a fully worked gradient descent example with real numbers.

7. Worked Example: Gradient Descent — First Iteration

Hook. Time to run gradient descent by hand. Given a medical dataset and initial parameters, we'll compute one full iteration — all three parameter updates — and see exactly how the numbers move.

Problem Setup

Goal: Fit a linear regression to predict relative risk of coronary heart disease (RRCHD) from:

$x_{1}$ : BMI
$x_{2}$ : Diastolic blood pressure (DP)

Given:

Model: $y^= θ_{0} + θ_{1} x_{1} + θ_{2} x_{2}$
Initial parameters: $θ_{0} = 5$ , $θ_{1} = - 0.03$ , $θ_{2} = - 0.03$
Learning rate: $α = 0.02$
$N = 3$ training examples

Data

Patient	BMI ( $x_{1}$ )	Diastolic Pressure ( $x_{2}$ )	RRCHD ( $y$ )
1	35	80	1.81
2	28	72	1.45
3	42	95	2.10

The exact values for patients 2 and 3 were shown on a lecture slide but not fully captured in the recording. The values above are reconstructed from the problem context (BMI/DP/RRCHD ranges typical of cardiovascular studies). The computation below shows the complete method; in an exam, the slide values would be provided explicitly.

Gradient Descent Update Formulas

For $j = 0, 1, 2$ :

θ_{j} := θ_{j} - α \cdot \frac{1}{N} i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}

Note: for $j = 0$ , $x_{0}^{(i)} = 1$ for all $i$ .

Computing the First Iteration

Step 1 — Compute predictions $y^^{(i)}$ for all patients.

y^^{(1)} y^^{(2)} y^^{(3)} = 5 + (- 0.03) (35) + (- 0.03) (80) = 5 - 1.05 - 2.40 = 1.55 = 5 + (- 0.03) (28) + (- 0.03) (72) = 5 - 0.84 - 2.16 = 2.00 = 5 + (- 0.03) (42) + (- 0.03) (95) = 5 - 1.26 - 2.85 = 0.89

Step 2 — Compute residuals.

y^^{(1)} - y^{(1)} y^^{(2)} - y^{(2)} y^^{(3)} - y^{(3)} = 1.55 - 1.81 = - 0.26 = 2.00 - 1.45 = + 0.55 = 0.89 - 2.10 = - 1.21

Step 3 — Sum the error terms for each $θ_{j}$ .

For $θ_{0}$ (multiply each residual by $x_{0} = 1$ ):

\sum (residual \times 1) = - 0.26 + 0.55 + (- 1.21) = - 0.92

For $θ_{1}$ (multiply each residual by $x_{1}$ ):

\sum (residual \times x_{1}) = (- 0.26) (35) + (0.55) (28) + (- 1.21) (42) = - 9.10 + 15.40 - 50.82 = - 44.52

For $θ_{2}$ (multiply each residual by $x_{2}$ ):

\sum (residual \times x_{2}) = (- 0.26) (80) + (0.55) (72) + (- 1.21) (95) = - 20.80 + 39.60 - 114.95 = - 96.15

Step 4 — Apply the update rule (with $α = 0.02$ , $N = 3$ ).

θ_{0}^{new} θ_{1}^{new} θ_{2}^{new} = 5 - 0.02 \cdot \frac{1}{3} \cdot (- 0.92) = 5 - 0.02 \cdot (- 0.3067) = 5 + 0.00613 = 5.0061 = - 0.03 - 0.02 \cdot \frac{1}{3} \cdot (- 44.52) = - 0.03 - 0.02 \cdot (- 14.84) = - 0.03 + 0.2968 = 0.2668 = - 0.03 - 0.02 \cdot \frac{1}{3} \cdot (- 96.15) = - 0.03 - 0.02 \cdot (- 32.05) = - 0.03 + 0.6410 = 0.6110

Result after iteration 1: $θ = [5.0061, 0.2668, 0.6110]$ .

Sense-check: $θ_{1}$ and $θ_{2}$ moved dramatically from $- 0.03$ to positive values. This makes sense — the initial negative slopes produced predictions that were too low (negative residuals summed to $- 0.92$ ), so the parameters increased to raise predictions. For subsequent iterations, use these updated values.

Key Insight

The term $\sum (y^- y)$ — the total prediction error — appears in all three updates. Only the multiplier ( $x_{j}^{(i)}$ ) changes per parameter. The formula conceptually: new parameter = old parameter − learning rate × (average prediction error, weighted by that parameter's feature).

Recap + Bridge

One gradient descent iteration: compute predictions → residuals → sum (residual × feature) for each parameter → update simultaneously. Repeat until convergence. Next: three flavors of gradient descent — batch, stochastic, and mini-batch — trading off speed against stability.

8. Types of Gradient Descent

Hook. The blindfolded hiker has a choice: call EVERY homeowner in the valley for directions before taking one step, call just ONE random person per step, or call a small group. Each strategy changes how fast and how smoothly you descend.

Purpose

The three gradient descent variants differ in how many training examples are used to compute each parameter update. This choice governs the speed-vs-stability trade-off.

Inputs & Outputs

Inputs: Training data $(X, Y)$ , learning rate $α$ , initial $θ$ , batch size $b$ (for mini-batch).
Outputs: Converged parameter vector $θ$ .

The Three Variants

#### Batch Gradient Descent

Update rule: $θ_{j} := θ_{j} - α \cdot \frac{1}{N} \sum_{i = 1}^{N} (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}$

Uses the full dataset (all $N$ examples) for every single parameter update.

Analogy: The hiker calls all $N$ homeowners, collects every opinion, averages, then takes one careful step.
Pros: Most accurate gradient estimate; smooth, predictable convergence path (smoothest of all three on a contour map).
Cons: Extremely slow for large $N$ . Every step requires scanning the entire dataset. Memory-intensive — may not fit in RAM.

#### Stochastic Gradient Descent (SGD)

Update rule: $θ_{j} := θ_{j} - α \cdot (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}$

Uses one randomly chosen training example per update. No summation.

Analogy: The hiker calls one random homeowner, takes a step immediately based on that single reply, then calls another.
Pros: Very fast updates. Makes progress immediately. Can escape shallow local minima in non-convex problems (not relevant for linear regression, but critical for deep learning).
Cons: Chaotic, noisy path (squiggly red line on a contour map). Each step may go in a wrong direction because a single example is unrepresentative. Never settles exactly at the minimum — oscillates around it.

#### Mini-Batch Gradient Descent

Update rule: $θ_{j} := θ_{j} - α \cdot \frac{1}{b} \sum_{i \in batch} (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}$

Uses a small random subset (mini-batch) of size $b$ per update. Typical $b$ : 32, 64, 128.

Analogy: The hiker divides homeowners into groups of 10, calls one group, averages their opinions, takes one step.
Pros: Balances batch (stable) and stochastic (fast). The most commonly used variant in practice. Leverages vectorized GPU computation efficiently.
Cons: Slightly noisier path than pure batch, but converges reliably and much faster on large datasets.

Visual Summary — Contour Map Comparison

On a contour map of the cost function (concentric ellipses):

Batch GD (blue path): Smoothest, most direct route to the center. Every step points straight toward the minimum.
Stochastic GD (red path): Most chaotic, zigzag route. Frequent wrong turns, but makes progress overall. Never fully settles.
Mini-Batch GD (green path): Slightly wiggly but efficient. Wobbles a bit, converges faster than batch on large data.

When to Use / Alternatives

Batch GD: Small datasets ( $N < 1 0^{4}$ ) where the full dataset fits in memory. Used in the worked example (Section 7).
SGD: Streaming data, online learning, or when you need updates after every example. Also used as a regularizer (the noise helps generalization).
Mini-Batch GD: The default for most machine learning — especially deep learning. Balances speed, memory, and stability.

Alternative: The closed-form solution (Section 4) is not a gradient descent variant — it's a completely different approach that avoids iteration entirely.

Student Q&A

Q: Which variant was used in the worked example (Section 7)?

A: Batch gradient descent — the summation was over all $N = 3$ training examples. In an exam, if you see $\frac{1}{N} \sum_{i = 1}^{N}$ in the update formula, it's batch GD. If you see no summation (just a single example), it's stochastic GD.

Q: Several students asked — can we switch between variants during training?

A: Yes. A common strategy: start with a larger batch size for stable early progress, then reduce it (or switch to SGD) near convergence. Learning rate schedules (decreasing $α$ over time) are often used alongside this.

Recap + Bridge

Batch = all data, smooth but slow. SGD = one example, fast but noisy. Mini-batch = compromise, most practical. Next: after training, how do we measure whether our model is any good?

Real-World & Domain Connection

Mini-batch SGD is the engine behind virtually all deep learning training. PyTorch's DataLoader and TensorFlow's tf.data are built around efficient mini-batch construction. The batch size is a hyperparameter you tune: powers of 2 (32, 64, 128) align with GPU memory architecture. The term "stochastic gradient descent" in modern deep learning almost always means mini-batch SGD.

9. Evaluating Linear Regression Models

Hook. You've trained a model. It spits out predictions. How do you put a single number on how good those predictions are — and how do you know whether that number is impressive or embarrassing?

Intuition + Analogy

Think of evaluation metrics like scoring a dart player:

MAE = average distance from bullseye (all misses count equally).
MSE = average squared distance (wild misses are punished severely — the player who occasionally throws a dart into the wall looks much worse than the consistently slightly-off player).
RMSE = square root of MSE — brings the units back to inches, so you can say "this player is typically off by 3 inches."
R² = what fraction of the dartboard's variance is explained by skill vs. luck.

Mean Absolute Error (MAE)

MAE = \frac{1}{N} i = 1 \sum N ∣ y^{(i)} - y^^{(i)} ∣

Average absolute distance between prediction and actual. Uses absolute value so positive and negative errors don't cancel. Lower is better. Unit: same as $y$ .

Mean Squared Error (MSE)

MSE = \frac{1}{N} i = 1 \sum N (y^{(i)} - y^^{(i)})^{2}

Averages the squared errors. Squaring penalizes large errors disproportionately: error of 2 → penalty 4; error of 3 → penalty 9. Larger errors dominate MSE. Unit: squared units of $y$ (harder to interpret directly).

Root Mean Squared Error (RMSE)

RMSE = MSE = \frac{1}{N} i = 1 \sum N (y^{(i)} - y^^{(i)})^{2}

The most commonly used error metric for regression. The square root restores the original units, making it directly interpretable: "predictions are typically off by $X$ units."

Worked Example — Computing All Three Metrics

Data: Actual values $y = [10, 20, 30]$ , Predictions $y^= [12, 18, 33]$ .

Errors: $y - y^= [- 2, + 2, - 3]$

MAE: $\frac{∣ - 2∣ + ∣2∣ + ∣ - 3∣}{3} = \frac{2 + 2 + 3}{3} = \frac{7}{3} = 2.33$

MSE: $\frac{( - 2 ) ^{2} + 2 ^{2} + ( - 3 ) ^{2}}{3} = \frac{4 + 4 + 9}{3} = \frac{17}{3} = 5.67$

RMSE: $5.67 = 2.38$

Sense-check: RMSE > MAE (always, because squaring amplifies larger errors before averaging). The values are close because errors are similar in magnitude (no extreme outliers).

R-squared ( $R^{2}$ ) — Goodness of Fit

$R^{2}$ answers: what percentage of the variation in the target can the model explain?

R^{2} = 1 - \frac{S S _{residual}}{S S _{total}}

Where:

$S S_{residual} = \sum (y^{(i)} - y^^{(i)})^{2}$ — unexplained variation (model's remaining error).
$S S_{total} = \sum (y^{(i)} - y ˉ)^{2}$ — total variation (how much $y$ varies around its mean).

Equivalently, since $S S_{total} = S S_{residual} + S S_{regression}$ (where $S S_{regression} = \sum (y^^{(i)} - y ˉ)^{2}$ is the explained variation):

R^{2} = \frac{S S _{regression}}{S S _{total}}

Interpretation:

$R^{2} = 0.98$ → 98% of variation in $y$ is explained by the features. Excellent fit.
$R^{2} = 0.30$ → only 30% explained. Weak fit; most variation is unexplained noise.
$R^{2} = 0.01$ → the model explains almost nothing. Possibly worse than predicting the mean.

Range: $0 \leq R^{2} \leq 1$ for linear regression with an intercept. Can be negative for models without intercept or when evaluated on test data if the model is terrible.

$R^{2}$ is the square of the Pearson correlation coefficient $r$ between $y$ and $y^$ for simple linear regression (AppD §D.2.3):

r = \frac{\sum ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum ( x _{i} - x ˉ ) ^{2} \sum ( y _{i} - y ˉ ) ^{2}}, R^{2} = r^{2}

Decomposition of Variation

Explained variation: $y^- y ˉ$ — the difference between predicted and average. The model explains this: "based on mileage, the predicted car price should differ from the mean by this much."
Unexplained variation: $y - y^$ — the residual. The model cannot explain this.
Total variation = explained + unexplained = $y - y ˉ$ (actual minus average).

Visual Intuition

Plot actual $y$ on the $x$ -axis and predicted $y^$ on the $y$ -axis. A perfect model puts every point on the $y = x$ diagonal. The scatter around that line is the error. $R^{2} = 1$ means all points are on the diagonal. $R^{2} = 0$ means the predictions are no better than always guessing the mean $y ˉ$ — the points form a horizontal cloud at $y^= y ˉ$ .

Which Metric to Use?

There is no universal rule, but RMSE is the most commonly reported because its units match the target variable, making it the most interpretable. MAE is preferred when outliers should not be penalized heavily (it is less sensitive to outliers). $R^{2}$ is preferred when you need a normalized, scale-free measure — useful for comparing models across different datasets or target variables.

In code, all three (MAE, MSE, RMSE) can be computed with one line each using sklearn.metrics.

Pitfalls

Using $R^{2}$ for feature selection. $R^{2}$ involves $y^$ — the predicted value — so it can only be calculated after building the model and making predictions. Feature selection happens before modeling. Use correlation analysis (Pearson $r$ , correlation heatmaps) or algorithms like forward selection and backward elimination instead.
Chasing $R^{2} = 1$ . A perfect $R^{2}$ on training data almost always means overfitting — the model has memorized noise, not learned the pattern.
Comparing RMSE across datasets with different $y$ scales. RMSE for house prices in dollars vs. in thousands of dollars differs by a factor of 1000. Use $R^{2}$ or normalized RMSE for cross-dataset comparisons.
Confusing $R^{2}$ with correlation. $R^{2} = 0.8$ does not mean "80% correlation" — it means 80% of variance explained. The correlation $r = 0.8 \approx 0.89$ .

Student Q&A

Q: Several students asked — can R-squared be used for feature selection by checking how each feature influences the target?

A: No. R-squared involves $y^$ — the predicted value — so it can only be calculated after the model is built and predictions are made. Feature selection happens before modeling, during data pre-processing. Use correlation analysis (Pearson correlation, correlation heatmaps) and algorithms like forward selection and backward elimination instead. Domain knowledge is also a valid feature selection method.

Recap + Bridge

RMSE = the go-to metric (interpretable units). $R^{2}$ = what fraction of variation the model explains. MAE = less sensitive to outliers. All three are computed after training. Next: what if your data isn't a straight line? Linear basis functions let you fit curves while keeping the simplicity of a linear model.

Real-World & Domain Connection

In regulated industries (finance, pharmaceuticals), $R^{2}$ is often required in documentation to justify model adequacy. Kaggle competitions typically use RMSE (or RMSLE — root mean squared log error) as the leaderboard metric for regression tasks. sklearn.metrics provides mean_absolute_error, mean_squared_error, and r2_score — all importable and one-line calls.

10. Linear Basis Functions

Hook. Your data traces a sine wave. A straight line fits it terribly. But you're told to use a linear model — no neural networks allowed. Is there a way to fit a curve while keeping the model mathematically linear?

Intuition + Analogy

Imagine you have a piece of stiff wire (your straight-line model). You need it to follow a curvy path drawn on paper. You can't bend the wire — but you can place a series of flexible rulers (basis functions) along the path, each one responsible for a local bump or wiggle. You then combine the rulers by weighting them. The result traces the curve perfectly, even though each individual ruler is simple.

The trick: transform the raw input $x$ through nonlinear functions $ϕ_{j} (x)$ first, then feed the transformed values into a standard linear model. The model stays linear in the weights — all the nice math (closed form, convex cost) still works.

Core Idea

Any model of the form:

y = β_{0} \cdot ϕ_{0} (x) + β_{1} \cdot ϕ_{1} (x) + β_{2} \cdot ϕ_{2} (x) + \dots + β_{M} \cdot ϕ_{M} (x)

is linear in the parameters $β_{0}, β_{1}, \dots, β_{M}$ . The functions $ϕ_{j} (x)$ — basis functions — can be anything nonlinear. As long as the equation is a weighted sum of these basis functions, the model remains linear (Bishop §3.1).

The critical distinction: "Linear model" means linear in the weights/parameters, not linear in the features. Polynomial regression IS a linear model:

y = β_{0} + β_{1} x + β_{2} x^{2} + β_{3} x^{3}

This can be rewritten as:

y = β_{0} ϕ_{0} (x) + β_{1} ϕ_{1} (x) + β_{2} ϕ_{2} (x) + β_{3} ϕ_{3} (x)

where $ϕ_{0} (x) = 1$ , $ϕ_{1} (x) = x$ , $ϕ_{2} (x) = x^{2}$ , $ϕ_{3} (x) = x^{3}$ .

A truly nonlinear model would have parameters appearing nonlinearly — e.g., inside a sigmoid or as an exponent. Neural networks are nonlinear in their parameters, which makes analyzing them much harder.

Basis Function Types (Bishop §3.1)

Type	Formula	Shape	Best For
Polynomial	$ϕ_{j} (x) = x^{j}$	Powers: linear, quadratic, cubic, ...	Smooth global trends
Gaussian	$ϕ_{j} (x) = exp (- \frac{( x - μ _{j} ) ^{2}}{2 s ^{2}})$	Bell curves centered at $μ_{j}$	Local bumps; data with peaks
Sigmoidal	$ϕ_{j} (x) = σ (\frac{x - μ _{j}}{s})$ , $σ (a) = \frac{1}{1 + e ^{- a}}$	S-shaped curves	Data with saturation or thresholds

Always: $ϕ_{0} (x) = 1$ — preserves the intercept/bias term. This is why $x_{0} = 1$ for every row of the design matrix.

How to Choose

Plot the data during exploratory data analysis (scatter plot). The shape of the trend suggests which basis function family to try:

Roughly straight? No basis functions needed (or just $ϕ_{1} (x) = x$ ).
One smooth bend? Quadratic polynomial ( $M = 2$ ).
Wavy? Try Gaussian basis functions at multiple centers $μ_{j}$ .
Saturation curve (rises then flattens)? Sigmoidal.

For complex shapes where visual inspection is not enough, experimentation and cross-validation guide the choice.

Visual Intuition

Three panels side by side (Bishop Figure 3.1). Left: polynomial basis functions — $1, x, x^{2}, x^{3}$ — smooth curves that extend globally. Center: Gaussian basis functions — localized bell curves at different positions $μ_{j}$ . Right: sigmoidal basis functions — S-curves that transition from 0 to 1. Each function transforms the raw $x$ into a new feature; the linear model weights and sums them.

Pitfalls

Thinking polynomial regression is a "nonlinear model." It is linear in parameters. This distinction is heavily tested.
Choosing $M$ too high. High-degree polynomials oscillate wildly between data points (Runge's phenomenon) — a form of overfitting.
Forgetting $ϕ_{0} (x) = 1$ . Without it, you lose the intercept term.

Student Q&A

Q: Are polynomial, Gaussian, and sigmoidal basis functions three different types of regression?

A: No. They are three examples of basis functions that can be used within a linear model. They are transformations applied to features before using linear regression — not separate regression types. The underlying model is the same weighted sum; only the preprocessing changes.

Q: When do we decide to use polynomial vs. Gaussian vs. sigmoidal basis functions?

A: By doing exploratory data analysis first. Plot the data as a scatter plot and look at the shape of the trend. The visual pattern guides which basis function family is appropriate. For very complex shapes, coding and experimentation help decide.

Recap + Bridge

Basis functions transform $x$ → $ϕ (x)$ , then linear regression does the rest. The model stays linear in weights, preserving all the nice math. Next: the danger of using too many basis functions — underfitting and overfitting.

Real-World & Domain Connection

Gaussian basis functions are the foundation of Radial Basis Function (RBF) networks — a type of neural network where each hidden neuron is a Gaussian centered at a data point. Polynomial basis functions appear in polynomial regression (econometrics) and response surface methods (engineering optimization). Splines — piecewise polynomials — are the practical workhorse in statistical modeling (GAMs in R's mgcv package) and represent a more sophisticated take on the same basis-function idea.

11. Underfitting and Overfitting

Hook. A student memorizes every answer from the practice test — without understanding the concepts. On the practice test: 100%. On the real exam with new questions: fails. This is overfitting, and machine learning models do exactly the same thing.

Intuition + Analogy

The core tension of machine learning: too simple → can't capture the pattern. Too complex → memorizes the noise. The analogy from the professor:

A student gets sample papers with answers. The student memorizes every question-answer pair without understanding the underlying concepts. On a question from the sample paper, the student scores full marks. On a new question testing the same concept but phrased differently, the student cannot answer.

Underfitting: The student didn't study enough — fails on everything, even the practice test.
Overfitting: The student memorized instead of learning — aces the practice test, fails the real exam.
Good fit: The student learned the concepts — does well on both.

For models: the "practice test" = training data; the "real exam" = unseen test data.

Polynomial Fitting Example

Data generated by an underlying sine curve (unknown in practice — the green reference line). Blue dots = observed training data. Red lines = models of different polynomial orders $M$ .

Polynomial Order $M$	Model	Behavior	Verdict
$M = 0$	Constant $y = θ_{0}$	Horizontal line through the mean. Captures nothing.	Underfitting
$M = 1$	Linear $y = θ_{0} + θ_{1} x$	Straight line. Misses the sine-wave structure.	Underfitting
$M = 3$	Cubic	Follows the sine curve closely without matching every wiggle.	Good fit
$M = 9$	9th-degree polynomial	Passes through every training point exactly. Oscillates wildly between points.	Overfitting

Assumptions & Scope

Scope. Overfitting happens when model complexity exceeds what the data can support. Key risk factors:

Too many parameters relative to training examples ( $D ≫ N$ ).
Training for too many iterations (in iterative methods).
Noisy data with a too-flexible model.

What breaks: An overfit model has near-zero training error but catastrophically high test error. It has learned the noise, not the signal. Regularization (constraining parameter magnitudes) and cross-validation (evaluating on held-out data) are the defenses.

Visual Intuition

Four panels showing the same blue dots with different red fitted curves. M=0: a flat horizontal line far from most points. M=1: a straight line sloping upward, missing the wave. M=3: a smooth curve weaving through the points without touching every single one. M=9: a violently oscillating curve that nails every training point but swings wildly in between — a classic picture of overfitting.

Pitfalls

Judging a model by training error alone. Always evaluate on a separate test set. Training error only goes down with more complexity; test error follows a U-shape (the bias-variance trade-off).
Adding parameters to chase a slightly better $R^{2}$ . Each new parameter "uses up" a degree of freedom. Adjusted $R^{2}$ penalizes unnecessary parameters.
Assuming more data always fixes overfitting. More data helps, but if the model is too flexible relative to the signal-to-noise ratio, even large datasets can be overfit.

Recap + Bridge

Underfitting = too simple, fails everywhere. Overfitting = memorizes training data, fails on new data. The sweet spot is a model complex enough to capture the pattern but simple enough to ignore noise. This tension — and how to resolve it — leads to bias-variance trade-off and regularization (tomorrow's topics).

Real-World & Domain Connection

Overfitting is the #1 cause of machine learning project failure in industry. A model that performs brilliantly in development but tanks in production is almost always overfit. Techniques to combat it — cross-validation, regularization, early stopping, dropout — form a large fraction of the practicing ML engineer's toolkit. The "student who memorized" analogy is universally taught because it perfectly captures the intuition.

12. Python Implementation — Closed Form Solution

Hook. Enough math — let's code. One line to train linear regression in Python, and one line each to get your intercept and slope.

Code Walkthrough

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Load data — X must be 2D (matrix), y can be 1D
X = df[['YearsExperience']]  # double brackets -> DataFrame (2D)
y = df['Salary']

# Check shapes
print(X.shape)  # (30, 1) — 30 rows, 1 feature
print(y.shape)  # (30,)   — 30 target values

# Create and train the model
model = LinearRegression()
model.fit(X, y)  # fit() is universal across all sklearn models

# Get learned parameters
print(model.intercept_)   # θ₀ (bias) — e.g., 25792.20
print(model.coef_[0])     # θ₁ (slope) — e.g., 9449.96

# Result: Salary = 9449.96 × YearsExperience + 25792.20

# Predict new values
X_new = np.array([[5], [10]])  # 5 and 10 years experience
predictions = model.predict(X_new)
print(predictions)  # e.g., [73042.01, 120291.85]

Key Points

model.fit(X, y) — the .fit() method is universal across all scikit-learn models. Decision tree, SVM, logistic regression — all use .fit().
Double brackets [[ ]] in pandas: convert a column to a matrix (2D). Single brackets produce a Series (1D). .fit() expects a matrix for $X$ .
model.intercept_ → $θ_{0}$ (bias). model.coef_ → $θ_{1}, θ_{2}, \dots$ (slopes/weights).
sklearn.linear_model.LinearRegression uses the closed-form solution internally (via SVD or QR decomposition).
For gradient descent, use sklearn.linear_model.SGDRegressor instead.

Pitfalls

Single bracket bug: df['col'] produces a 1D Series; sklearn expects a 2D array. Use df[['col']] or .reshape(-1, 1).
Not checking shapes. If .fit() throws a dimension error, print X.shape first — 90% of bugs are shape mismatches.

Student Q&A

Q: Are we allowed to use the LinearRegression from scikit-learn in assignments?

A: The instruction about which functions can be used directly vs. which must be implemented will be confirmed. Generally, scikit-learn library functions can be used. The restriction may apply only to specific functions like train-test split. Check the assignment instructions and the announcement.

Recap + Bridge

LinearRegression().fit(X, y) — one call to train; .intercept_ and .coef_ to inspect. The scikit-learn API is consistent across hundreds of models. For gradient descent, swap to SGDRegressor.

13. Gradient Descent — Quick Formula Reference

Parameter Update (General Form)

For all $θ_{j}$ ( $j = 0, 1, \dots, D$ ):

θ_{j} := θ_{j} - α \cdot \frac{1}{N} i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}

Memory aid: update = old parameter − learning rate × (average of prediction errors × feature value). The term $\sum (prediction - actual) \times feature$ is the core of the gradient.

Initialization

If initial $θ$ values are given in the question, use them.
If not given, start with $θ = 0$ (or very small random values).

Update Types Summary

Variant	Summation Range	Data Used Per Step
Batch GD	$\frac{1}{N} \sum_{i = 1}^{N}$	All $N$ examples
Stochastic GD	No sum (single example)	1 random example
Mini-Batch GD	$\frac{1}{b} \sum_{i \in batch}$	$b$ random examples

Exam Guidance Summary

Exam note: This section consolidates all exam-relevant guidance from the lecture.

Closed form solution: Memorize the normal equation $θ = (X^{⊤} X)^{- 1} X^{⊤} Y$ . Know how to build the $X$ matrix (with the column of 1's for $x_{0}$ ) and the $Y$ vector, plug in values, and compute $θ$ . After finding $θ_{0}$ and $θ_{1}$ , substitute into $y = θ_{0} + θ_{1} x$ for predictions. Do NOT derive the formula in the exam — just apply it.
Gradient descent: Memorize the update formula $θ_{j} := θ_{j} - α \cdot \frac{1}{N} \sum (h_{θ} (x) - y) \cdot x_{j}$ . Know how to apply it for one iteration with given initial values. Simultaneous update is critical — use temporary variables.
Exam matrices will be invertible. No singular matrices on paper exams.
Derivation of the normal equation — know conceptually but is not tested step-by-step. The matrix expansion and derivative reasoning help with understanding but won't be examined in full detail.
All three evaluation metrics (MAE, MSE, RMSE) should be known. RMSE is the most important — most commonly used and most likely to appear.
R-squared interpretation — know what high vs. low $R^{2}$ means. Know the formula $R^{2} = 1 - S S_{res} / S S_{total}$ and the equivalent form $R^{2} = S S_{reg} / S S_{total}$ . Know that $R^{2}$ cannot be used for feature selection.
Feature scaling methods (normalization, standardization) — know why they matter for gradient descent. Standardization (z-score) is the most common.
Types of gradient descent (batch, mini-batch, stochastic) — understand the conceptual differences, when to use each, and recognize which variant a given update formula represents.
Underfitting vs. overfitting — understand what each means, how polynomial order $M$ relates, and the memorization analogy.
Linear basis functions — understand that polynomial regression is still a linear model (linear in parameters). $ϕ_{0} (x) = 1$ always. Know the three types: polynomial, Gaussian, sigmoidal.
Tomorrow's topics: bias, variance, regularization, and logistic regression.

Key Industry Applications and Real-World Connections

scikit-learn (sklearn): The standard Python ML library. LinearRegression uses closed form (via SVD); SGDRegressor uses stochastic gradient descent. The .fit() / .predict() API is universal across all sklearn models.
NumPy, Matplotlib: Core libraries for numerical computing and visualization. np.linalg.pinv() for pseudo-inverse; plt.scatter() for exploratory data analysis.
Jupyter Notebook, PyCharm, Virtual Lab: Development environments used for ML coding and assignments.
Correlation heatmaps (sns.heatmap): Standard EDA technique for identifying which features relate to the target — used for feature selection before modeling.
Forward selection and backward elimination: Iterative feature selection algorithms in data preprocessing.
Pearson correlation: Statistical measure of linear relationship; $r$ feeds into $R^{2} = r^{2}$ for simple regression.
Moore-Penrose pseudo-inverse (numpy.linalg.pinv): Handles non-invertible matrices via SVD. The numerical backbone of the closed-form solution in practice (Bishop §3.1.1, eq. 3.17).
SGDRegressor: scikit-learn's gradient-descent-based linear regression. Supports various loss functions and penalties.
Regularization (Ridge, Lasso): Extensions of linear regression that constrain parameter magnitudes — Ridge adds $λ ∥ θ ∥_{2}^{2}$ , Lasso adds $λ ∥ θ ∥_{1}$ (Bishop §3.1.4). Covered in the next lecture.
Neural networks: The domain where truly nonlinear-in-parameters models live — gradient descent (and backpropagation) is essential here because no closed form exists.

Textbooks and References

Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Chapter 3: Linear Models for Regression.
Tan, P. N., Steinbach, M., & Kumar, V. (2005). Introduction to Data Mining. Appendix D: Regression; Appendix E: Optimization.
Mitchell, T. M. (1997). Machine Learning. McGraw-Hill. Chapter 5: Evaluating Hypotheses.
scikit-learn documentation: LinearRegression, SGDRegressor, sklearn.metrics.

ML Lecture 4 notes · Linear Regression — Complete Lecture Notes

Machine Learning· postgraduate· 2026-06-28

Summary

Complete lecture notes on Linear Regression for Machine Learning. Covers the linear regression hypothesis function and parameter vector. The cost function (mean squared error) forms a convex bowl with one global minimum. Two optimization approaches: the closed-form normal equation (one-shot matrix computation) and gradient descent (iterative hill-descent). Feature scaling is essential for efficient gradient descent. Three gradient descent variants: batch, stochastic, and mini-batch. Evaluation metrics include MAE, MSE, RMSE, and R-squared. Linear basis functions (polynomial, Gaussian, sigmoidal) allow linear models to fit nonlinear data. Underfitting and overfitting trade-off with model complexity. Python implementation using scikit-learn's LinearRegression and SGDRegressor.

Learning Objectives

1Write the linear regression hypothesis function and identify the parameters (weights) and features

2Compute and interpret the cost function (MSE) for linear regression

3Apply the closed-form normal equation to find optimal parameters

4Perform one iteration of gradient descent with simultaneous parameter updates

5Explain why feature scaling is necessary for gradient descent convergence

6Distinguish between batch, stochastic, and mini-batch gradient descent

7Calculate and interpret MAE, MSE, RMSE, and R-squared evaluation metrics

8Use linear basis functions to fit nonlinear patterns with linear models

9Identify underfitting and overfitting from model complexity and polynomial order

10Implement linear regression in Python using scikit-learn

Sections Breakdown

11. Linear Regression Model

Hypothesis function, parameters, features, assumptions, and notation for linear regression.

22. Cost Function

Mean squared error cost function, convex bowl shape, and the 1/2 factor.

33. Derivatives, Maxima, and Minima

Derivative as slope, stationary points, second derivative test.

44. Closed Form Solution

Normal equation derivation, matrix form, invertibility caveat.

55. Gradient Descent

Update rule, learning rate, simultaneous update, convergence criteria.

66. Feature Scaling

Why scaling matters, normalization vs. standardization.

77. Worked Example

Complete hand-calculation of one gradient descent iteration.

88. Types of Gradient Descent

Batch, stochastic, and mini-batch variants.

99. Evaluating Models

MAE, MSE, RMSE, R-squared with worked examples.

1010. Linear Basis Functions

Polynomial, Gaussian, sigmoidal basis functions.

1111. Underfitting and Overfitting

Bias-variance trade-off, polynomial order selection.

1212. Python Implementation

scikit-learn LinearRegression code walkthrough.

1313. Gradient Descent Formula Reference

Quick reference for gradient descent update rules.

Postgraduate students in Machine Learning and Data Science

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Linear Regression Hypothesis

Must-know: The hypothesis $h_{θ} (x) = θ_{0} + θ_{1} x$ is a weighted sum of features plus bias. The goal of linear regression is to find the parameter vector $θ$ that minimizes the cost function.

$h_{θ} (x) = θ^{⊤} x where x_{0} = 1$

⚠️ Top pitfall: Forgetting to include $x_{0} = 1$ for the bias term forces the model through the origin.

Self-check: What does it mean that linear regression is "linear in parameters" but can fit curves?

Connects to: Cost Function, Basis Functions

Cost Function (Mean Squared Error)

Must-know: The cost function $J (θ) = \frac{1}{2 N} \sum (h_{θ} (x) - y)^{2}$ measures prediction error. The $1/2$ cancels the derivative's factor of 2. The surface is a convex bowl with one global minimum.

$J (θ) = \frac{1}{2 N} i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)})^{2}$

⚠️ Top pitfall: Confusing the cost function (which has $1/2$ for training) with MSE (which does not, for reporting).

Self-check: Why is the cost function for linear regression always convex?

Connects to: Gradient Descent, Closed Form Solution

Closed Form Solution (Normal Equation)

Must-know: The normal equation $θ = (X^{⊤} X)^{- 1} X^{⊤} Y$ jumps directly to the optimal parameters in one matrix computation. Requires inverting $X^{⊤} X$ which costs $O (D^{3})$ .

$θ = (X^{⊤} X)^{- 1} X^{⊤} Y$

⚠️ Top pitfall: $(X^{⊤} X)^{- 1}$ does not exist when features are linearly dependent or $N < D + 1$ . Use the pseudo-inverse or regularization.

Self-check: When would you choose gradient descent over the closed form solution?

Connects to: Matrix Inversion, Gradient Descent

Gradient Descent

Must-know: Gradient descent iteratively updates parameters: $θ_{j} := θ_{j} - α \cdot \frac{1}{N} \sum (h_{θ} (x) - y) \cdot x_{j}$ . All parameters must be updated simultaneously using temporary variables.

$θ_{j} := θ_{j} - α \cdot \frac{1}{N} i = 1 \sum N (h_{θ} (x^{(i)}) - y^{(i)}) \cdot x_{j}^{(i)}$

⚠️ Top pitfall: Sequential instead of simultaneous update — the #1 coding bug in gradient descent.

Self-check: A derivative is positive. Does gradient descent increase or decrease that parameter?

Connects to: Cost Function, Learning Rate, Feature Scaling

Feature Scaling

Must-know: Features with different ranges distort the cost contours into elongated ellipses, causing gradient descent to zigzag. Standardization (z-score) is the most common fix.

$x^{'} = \frac{x - μ}{σ} (standardization)$

⚠️ Top pitfall: Fit the scaler on training data only, then transform both training and test data. Never fit on test data.

Self-check: Why does feature scaling matter for gradient descent but not for the closed form solution?

Connects to: Gradient Descent, Normalization

Evaluation Metrics (MAE, MSE, RMSE, R²)

Must-know: RMSE is the most commonly reported metric because its units match the target. $R^{2}$ measures the fraction of variance explained by the model. MAE is less sensitive to outliers.

$RMSE = \frac{1}{N} \sum (y - y^)^{2} R^{2} = 1 - \frac{S S _{res}}{S S _{total}}$

⚠️ Top pitfall: $R^{2}$ cannot be used for feature selection — it requires predictions from a built model.

Self-check: A model has RMSE = 5.2 and $R^{2} = 0.85$ . Interpret both numbers.

Connects to: Cost Function, Model Evaluation

Linear Basis Functions

Must-know: Basis functions transform raw features so linear models can fit curves. The model stays linear in parameters. Polynomial, Gaussian, and sigmoidal are the three main types.

$y = j = 0 \sum M β_{j} ϕ_{j} (x) ϕ_{0} (x) = 1$

⚠️ Top pitfall: Calling polynomial regression a "nonlinear model." It is linear in parameters — a common exam trap.

Self-check: Is $y = β_{0} + β_{1} x + β_{2} x^{2}$ a linear model? Why?

Connects to: Linear Regression Hypothesis, Overfitting

Underfitting and Overfitting

Must-know: Underfitting = model too simple to capture patterns (high bias). Overfitting = model memorizes noise instead of learning signal (high variance). The sweet spot balances complexity against data size.

$Polynomial order M : M = 0, 1 underfit M = 3 good M = 9 overfit$

⚠️ Top pitfall: Judging a model by training error alone — always evaluate on held-out test data.

Self-check: Why does a 9th-degree polynomial perfectly fit training data but perform poorly on new data?

Connects to: Basis Functions, Regularization

Practice Quiz

Test your understanding of ML Lecture 4 notes. Select an answer for each question — results are instant.

What is the purpose of the 1/2 factor in the cost function J(θ) = (1/2N) Σ (hθ(x) - y)²?

Which of the following is NOT a required condition for linear regression?

In gradient descent, what happens if the learning rate α is too large?

Previous LectureLecture 03 · Feature Engineering and Linear Regression

Next LectureLecture 05 · Gradient Descent, Regularization, and Logistic Regression

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Linear Regression — Complete Lecture Notes

1. Linear Regression Model

Intuition + Analogy

Formal Definition

Notation Note

Symbol Registry — Linear Regression Hypothesis

Assumptions & Scope

Visual Intuition

Pitfalls

Recap + Bridge

Real-World & Domain Connection

2. Cost Function

Intuition + Analogy

Formal Definition

Shape of the Cost Function

Visual Intuition

Worked Example

Pitfalls

Symbol Registry — Cost Function

Student Q&A

Recap + Bridge

3. Derivatives, Maxima, and Minima

Intuition + Analogy

Formal Definition

Worked Example

Visual Intuition

Recap + Bridge

4. Closed Form Solution (Normal Equation)

Intuition + Analogy

Formal Definition

Full Derivation

Usage Recipe

Worked Example

Important Caveat — Matrix Invertibility

Comparison: Closed Form vs. Gradient Descent

Visual Intuition

Pitfalls

Recap + Bridge

Real-World & Domain Connection

5. Gradient Descent

Intuition + Analogy — The Blindfolded Hiker

Mapping the Analogy to Math

The Update Rule

Simultaneous Update — Critical Rule

Gradient of the Linear Regression Cost Function

How the Derivative Determines Direction

Learning Rate α

Convergence — When to Stop

Assumptions & Scope

Visual Intuition

Pitfalls

Student Q&A

Symbol Registry — Gradient Descent

Recap + Bridge

Real-World & Domain Connection

6. Feature Scaling for Gradient Descent

Intuition + Analogy

Why Feature Scaling Matters

Common Scaling Methods

Visual Intuition

Pitfalls

Recap + Bridge

7. Worked Example: Gradient Descent — First Iteration

Problem Setup

Data

Gradient Descent Update Formulas

Computing the First Iteration

Key Insight

Recap + Bridge

8. Types of Gradient Descent

Purpose

Inputs & Outputs

The Three Variants

Visual Summary — Contour Map Comparison

When to Use / Alternatives

Student Q&A

Recap + Bridge

Real-World & Domain Connection

9. Evaluating Linear Regression Models

Intuition + Analogy

Learning Rate $α$

R-squared ( $R^{2}$ ) — Goodness of Fit