Mathematical Foundations for Machine Learning

Challenges of Gradient Descent and Constrained Optimization

Published: 2026-07-11

Level: postgraduate

Audience: Postgraduate students in Machine Learning and related quantitative fields

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Gradient Descent and the Update Rule — covered in Lecture 11 (section 11.1)
Local vs Global Minima and Convexity — covered in Lecture 11 (section 11.2)
The Hessian Matrix and Critical Points — covered in Lecture 10 (sections 10.5, 10.6)
Two-Variable Taylor Series and the Gradient — covered in Lecture 10 (section 10.4)

Challenges of Gradient Descent and Constrained Optimization

Machine learning models learn by minimizing a loss function. But the path to the minimum is not a straight line. This lecture unpacks three obstacles that derail gradient descent. They are overfitting, local minima, and uneven curvature. Then it gives you the tools to fight back: constrained optimization, regularization, feature scaling, and the Hessian matrix.

12.1 Gradient Descent Recap

Hook: You are blindfolded on a foggy hillside. You feel the ground under your feet — it slopes down in one direction. You take a step that way. Then you feel again. Step. Feel. Step. That is gradient descent: a blind walk downhill, trusting only the local slope, hoping to reach the valley floor.

12.1.1 Review of Gradient Descent Variants

Gradient descent is an iterative algorithm that walks downhill on a loss surface. A loss surface is a landscape where height represents error and the lowest point is the best model. You start at some point on this surface and take steps toward the lowest point. Each step is guided by the gradient $\nabla L (w)$ , a vector pointing in the direction of steepest increase. Since you want to go downhill, you move opposite to the gradient.

Intuition: Think of gradient descent like a hiker descending a mountain in thick fog. They cannot see the valley. They can only feel the slope beneath their boots. The gradient is the steepness underfoot. Negative gradient means stepping downhill. Repeat until the ground feels flat.

Where the analogy breaks: a real hiker can sometimes see the valley from a ridge and take a shortcut. Gradient descent is completely blind — it only knows the slope at the current position, not the terrain 100 meters ahead.

There are three strategies for computing the gradient, distinguished by how much data you use per step:

Batch gradient descent — uses the entire dataset to compute each gradient step. Accurate but slow. Every step sees all $n$ training points, so the gradient is precise. Cost per step: $O (n d)$ where $n$ is dataset size and $d$ is feature count.
Mini-batch gradient descent — uses a small random subset (e.g., 32 or 64 points). Balances speed and noise. This is the workhorse of deep learning.
Stochastic gradient descent (SGD) — uses exactly one training example per step. Very noisy but very fast. The noise can actually help by jittering the optimizer out of shallow local minima.

All three variants share the same update rule:

$w_{new} = w_{old} - α \nabla L (w_{old})$

In plain language: new position = old position − step size × direction of steepest uphill.

The symbol $w$ (short for weight, from early connectionist work) represents the trainable parameter. In one dimension, $w$ is a scalar; in a model with millions of parameters, $w$ is a vector. The gradient $\nabla L (w)$ always has the same shape as $w$ — it provides one slope value per parameter.

Comparing the three variants:

Variant	Data per step	Gradient quality	Speed per step	Typical use
Batch GD	All $n$	Exact, smooth	Slowest	Convex problems, small datasets
Mini-batch GD	$b$ samples	Noisy estimate	Moderate	Deep learning (standard)
SGD	1 sample	Very noisy	Fastest	Online learning, streaming data

12.1.2 The Learning Rate Parameter

Alpha ( $α$ ) is the learning rate, sometimes called the step size. It tells the optimizer how big a step to take along the negative gradient direction.

Scope: The learning rate is a hyperparameter — you set it before training starts (unlike $w$ , which the algorithm learns). It is the single most important knob in gradient-based optimization.

Too large → You overshoot the minimum. Instead of settling, you bounce back and forth across the valley. In the worst case, you diverge entirely.
Too small → You crawl toward the minimum. Training takes forever. You might stall on a shallow slope and stop prematurely.
Just right → You descend smoothly and converge efficiently.

Choosing $α$ is a core tuning decision. The process of adaptively selecting the learning rate at each step is called line search, a topic covered in a later lecture. In practice, practitioners often use adaptive optimizers (Adam, RMSprop) that adjust $α$ automatically per parameter.

Pitfalls:

Treating $α$ as a set-and-forget value. The best learning rate often changes during training. Learning rate schedules (step decay, cosine annealing) raise or lower $α$ over time.
Using the same $α$ for all parameters. In deep networks, different layers may need different learning rates. Adaptive methods (Adam, AdaGrad) handle this.
Thinking $α > 1$ is always "too large." While $α$ is typically in $(0, 1)$ , values like $α = 0.001$ or $0.0001$ are common. The right range depends entirely on the scale of your gradients.
Confusing oscillation with convergence. A high $α$ can make the loss bounce without settling — it looks flat but isn't at the minimum.

Tiny worked example: Suppose we minimize $L (w) = (w - 5)^{2}$ starting from $w = 0$ with $α = 0.1$ .

Gradient: $\nabla L (w) = 2 (w - 5)$
At $w = 0$ : $\nabla L (0) = - 10$
Update: $w_{new} = 0 - 0.1 \times (- 10) = 1$
At $w = 1$ : $\nabla L (1) = - 8$
Update: $w_{new} = 1 - 0.1 \times (- 8) = 1.8$

After many iterations, $w$ approaches 5, the true minimum where $L (5) = 0$ . With $α = 1.1$ , you would overshoot $w = 5$ and diverge.

Visual intuition: Imagine a plot with parameter $w$ on the x-axis and loss $L (w)$ on the y-axis. The curve is a parabola opening upward — a convex bowl. The minimum sits at $w = 5$ . From $w = 0$ , the negative gradient points toward the minimum. Each step moves you rightward, but each successive step is smaller because the slope gets gentler near the bottom.

12.1.3 Symbol Registry

Symbol	Meaning	LaTeX	Type / Domain
$w$	weight / parameter vector	$w \in R^{d}$	$d$ -dimensional vector
$α$	learning rate	$α$	scalar, typically $α \in (0, 1)$
$\nabla L (w)$	gradient of loss at $w$	$\nabla L (w)$	same shape as $w$
$w_{new}$	updated parameter	$w_{new}$	same shape as $w$
$w_{old}$	current parameter	$w_{old}$	same shape as $w$

Recap: Gradient descent walks downhill on the loss surface. It uses the negative gradient as a compass and the learning rate as a stride length. The recipe is simple: $w_{new} = w_{old} - α \nabla L$ . But getting the step size right is where things get tricky.

Bridge: Even with the perfect learning rate, gradient descent faces three fundamental obstacles that can stop it cold. We tackle them next.

Real-world connection: Every time you train a neural network in PyTorch or TensorFlow, gradient descent runs underneath. PyTorch's `torch.optim.SGD` implements this exact update rule, with extras like momentum andweight decay. Google's AlphaGo and AlphaFold used gradient-based optimization to find their parameters across enormous spaces.

12.2 Three Fundamental Challenges of Gradient Descent

Hook: Gradient descent sounds foolproof — walk downhill, find the bottom. So why does training a neural network sometimes fail? Why does the loss refuse to drop, or drop to zero on training data while real-world accuracy stays terrible? Three culprits ambush the optimizer on its way down.

12.2.1 Overview

Gradient descent is conceptually simple. But in practice — especially with complex models like neural networks — it runs into three major problems: overfitting, local minima, and uneven curvature. Each of these derails the optimizer in a different way.

12.2.2 Challenge 1: Overfitting

Overfitting happens when a model learns the training data too well. It memorizes the noise instead of learning the underlying pattern. Training error drops toward zero. But when you show the model new data, it fails badly.

Analogy: A student who memorizes the answers to a practice exam but cannot solve a new problem on the actual test. The practice scores are perfect. The real exam is a disaster. This is overfitting: high training performance, terrible generalization.

The model isn't reasoning — it's having a photographic memory for the specific examples it saw. When the test question uses slightly different numbers, it has no idea what to do.

In ML terms, overfitting means you have low bias but high variance. The model hugs every training point. It builds an extremely complex function to do so. When any training point changes, the whole learned function must twist to handle it. That twisting is high variance.

Pitfall: Overfitting directly harms gradient descent. A highly overfit model produces a loss surface full of local minima. Instead of one clean bowl, you get a landscape of potholes. The optimizer can get trapped in any one of them.

12.2.3 Challenge 2: Local Minima

A local minimum is a point on the loss surface lower than its immediate neighbors, but not the lowest point overall. The global minimum is the absolute lowest point on the whole surface.

Consider a loss function shaped like a wavy line with many dips. Each dip is a local minimum. If gradient descent starts near one of these dips, it slides in and stops. Why? At the bottom of a dip, the gradient is zero. With a zero gradient, the update term $α \nabla L (w)$ becomes zero. No more movement. You are stuck — at a height that could be far above the true bottom.

Analogy: You are walking downhill in the dark. You step into a small ditch and the ground under your feet feels flat. You think you have reached the valley floor. But you are actually just in a pothole on the side of the mountain. The real valley is further down, but you have no way to know — the ground at your feet says "you're done."

This is why convexity matters so much: a convex bowl has zero potholes. One global minimum. No tricks.

Neural networks are especially prone to rugged loss surfaces with many local minima. This is because a neural network learns a very complex, highly nonlinear function. It stacks layers of nonlinear activations (ReLU, sigmoid, tanh) upon each other. The surface carved out in parameter space is not a simple convex bowl.

A convex function has exactly one minimum. Start gradient descent anywhere on a convex surface and — with a reasonable learning rate — you will eventually reach the global minimum. Linear regression with MSE produces a convex loss surface. Most neural network architectures do not.

Visual intuition: Picture a loss surface as a landscape with $w_{1}$ and $w_{2}$ on the two horizontal axes and loss on the vertical axis. A convex surface looks like a giant salad bowl — smooth, symmetric, one deepest point. A non-convex surface looks like the surface of the moon — craters, ridges, and basins everywhere. Where you start determines which crater you end up in.

Pitfalls:

Assuming all loss surfaces are convex. Only linear models with squared loss are guaranteed convex. Once you add nonlinear activations or certain loss functions (like cross-entropy with softmax in a deep net), convexity vanishes.
Relying only on local gradient information. In a non-convex world, the gradient at your current spot tells you nothing about whether there's a deeper basin one ridge away. This is why random restarts and momentum help.
Thinking "local minimum" means "bad solution." In very high dimensions (millions of parameters), most local minima of neural networks are surprisingly good — often close to the global optimum in terms of test error. The real problem is saddle points, not local minima. A saddle point is flat in some directions and curved in others — and in high dimensions, the vast majority of stationary points are saddles, not minima.

12.2.4 Challenge 3: Uneven Curvature

Uneven curvature means the loss surface tilts much more steeply in some directions than in others. The result is an elongated, narrow valley instead of a symmetric bowl.

Imagine a long, narrow ravine. The walls on the sides are very steep. Walk along the valley floor and it is nearly flat. If gradient descent starts on one of those flat regions, the gradient is nearly zero. The update is tiny. You barely move. You are stuck on a plateau.

Analogy: Think of a skateboarder in an empty swimming pool. Near the deep end, the walls are steep — one push and you race down. But on the shallow-end floor, the ground is flat. You push and barely roll. Gradient descent on a flat plateau is the shallow-end floor problem. The slope is so gentle, you go nowhere.

Uneven curvature is like a swimming pool stretched 50 meters long but only 2 meters wide from wall to wall. Downhill is easy if you aim at a wall. But along the length, every push barely moves you.

Uneven curvature is common when your input features have different scales. Say one feature ranges from 0 to 50 (age). Another ranges from 10,000 to 500,000 (salary in rupees). The loss surface tilts much more in the salary direction because salary has more raw variance. That creates the narrow-valley problem.

Pitfalls:

Plateau paralysis. A near-zero gradient on a flat region tricks you into thinking you have converged. You haven't — you're just on a shallow slope.
Zigzagging. In a narrow valley, the steepest direction points across the valley, not along it. Gradient descent takes a step toward the opposite wall, then back, then across again — crawling forward painfully slowly. This is why momentum helps: it accumulates velocity along the valley direction.
Feature scale blindness. If you never normalize your features, gradient descent will always struggle with uneven curvature. Section 12.6 will fix this.

The three challenges form a connected web: model complexity creates overfitting, overfitting creates local minima, and unequal feature scales create uneven curvature. The rest of this lecture builds the tools to fight each one. Regularization fights overfitting. Lagrange multipliers control weights. Feature scaling fixes curvature.

Recap: Three obstacles ambush gradient descent. Overfitting turns the loss surface into a minefield of potholes. Local minima trap you in craters short of the true bottom. Uneven curvature creates plateaus and narrow valleys where progress stalls. The toolkit: constrained optimization, regularization, and feature scaling.

Bridge: Next we zoom into the statistics behind overfitting — the bias-variance tradeoff. This explains why complex models produce jagged loss surfaces and why regularization works.

Real-world connection: In 2015, Google researchers found that depth helps with all three challenges. Very deep networks trained with SGD settle in broad, flat minima that generalize better than narrow, sharp ones. This idea, from "Deep Learning and the Information Bottleneck Principle" (Tishby & Zaslavsky, 2015), changed how we view optimization. Google's TPUs are built to run gradient descent on huge, non-convex surfaces.

12.3 Bias, Variance, and the Overfitting Mechanism

Hook: There is a hard mathematical ceiling on how well any model can perform. You cannot drive total error to zero. The remaining error splits into two parts — bias and variance — that fight each other in a zero-sum game. As one goes down, the other must go up.

12.3.1 Reducible and Irreducible Error

When you fit a model to data, the total generalization error decomposes into two parts:

$Error = Reducible Error + Irreducible Error$

Reducible error is the part you can shrink by picking a better model. Switch from linear regression to a neural network, and this drops. Irreducible error is the part no model can eliminate. It comes from inherent noise in the data. It also comes from a basic fact: any finite dataset can only approximate the true underlying distribution.

Analogy: Shooting arrows at a target. Reducible error is like adjusting your aim — you can practice and get closer to the bullseye. Irreducible error is like wind gusts on the archery range. You can aim perfectly and the wind still blows your arrow off course. No amount of skill changes the wind.

12.3.2 Bias-Variance Tradeoff

The irreducible error further decomposes into two components that oppose each other:

Bias — The difference between your model's average prediction and the true value. High bias means your model is too simple to capture the pattern. Think: fitting a straight line to data that follows a curve. The line cannot bend, so it is systematically wrong everywhere.
Variance — How much your model's predictions change when you train on a different sample of the data. High variance means your model is too sensitive. Small changes in training data produce wildly different predictions.

The key insight: bias and variance trade off against each other. For a given problem, their sum is fixed:

$Bias + Variance = Constant$

A model cannot have both low bias and low variance simultaneously. If bias is low, variance must be high. If variance is low, bias must be high. This is the bias-variance tradeoff.

Analogy: Think of two types of cooks. A high-bias cook follows a simple recipe exactly every time — the dish is consistent but never exciting. If the recipe is wrong for the ingredients, the dish is always wrong in the same predictable way. A high-variance cook improvises heavily — the result is sometimes brilliant, sometimes inedible, varying wildly based on subtle changes in ingredients. You cannot be both perfectly consistent and perfectly adaptable. You trade one for the other.

In this analogy: the recipe is your model, the ingredients are the training data, and the dish is the prediction.

Here is what happens with common model types:

Model	Bias	Variance	Behavior
Linear regression	High	Low	Stable across different data samples; systematically wrong for nonlinear patterns
Polynomial regression (degree 13)	Low	High	Wiggles through every training point; predictions shift dramatically when data changes
Neural networks	Low	High	Enough parameters to memorize training data; sensitive to initialization and data order
Decision trees (shallow)	High	Low	Simple rules; doesn't capture complex interactions
Decision trees (deep)	Low	High	Complex rule cascades; different splits with each sample

12.3.3 How Model Complexity Creates Local Minima

The polynomial complexity demonstration:

Start with a straight line (degree 1) on one-dimensional data. The loss surface is a perfect convex bowl — one global minimum.

Increase degree to 2, 3, 4. Still convex. Still manageable.

Push to degree 7. Local minima begin to appear. The loss surface gets bumpy.

At degree 9, one model coefficient jumps to -61 — far from the small ~0.003 range seen in the degree-1 model.

At degree 13, the sum of all weights reaches 2206. Training error hits zero. Test error stays at 0.625.

This is the signature of overfitting: zero training error, high test error, and exploding weights.

This polynomial demonstration mirrors exactly what happens with neural networks. A neural network learns a highly complex, high-dimensional nonlinear function by composing many nonlinear activation layers. You cannot visualize this function in hundreds of dimensions, but the effect is identical: the loss surface develops many local minima.

Scope: The bias-variance decomposition is most informative in the classical statistical learning framework (fixed dataset, MSE loss). In modern deep learning with overparameterized models, researchers have observed a "double descent" phenomenon. After the interpolation threshold (where training error hits zero), test error can actually decrease again with more complexity. This does not invalidate the bias-variance framework. But it adds an important caveat about the regime where models have more parameters than training points.

12.3.4 The Weight Explosion Problem

When a model overfits, its learned weights explode to very large positive or negative values. This is not random — it is the model contorting itself to pass through every single training point.

The weight explosion is a key diagnostic signal. Overfitting is not just about wrong predictions. It is also about pathological parameter magnitudes. Large weights mean the model is hypersensitive: a tiny change in input produces a huge swing in prediction.

This explosion is precisely why we need constrained optimization. If we can cap how large the weights can grow, we can prevent overfitting. That is the entire motivation behind regularization, which Section 12.5 formalizes using Lagrange multipliers.

Visual intuition: Draw a 2D plot with model complexity (polynomial degree) on the x-axis and error on the y-axis. Plot two curves:

Training error — starts high (underfitting), drops steadily with complexity, approaches zero at high degree.
Test error — starts high, dips to a minimum at moderate complexity (the sweet spot), then rises again as overfitting takes over.

The gap between training and test error at high complexity is the overfitting zone. The sweet spot — where test error is lowest — is where bias and variance are most balanced for the given problem.

Pitfalls:

Confusing training error with model quality. Training error always decreases with more complexity. Only test error tells you about generalization.
Ignoring weight magnitudes. A model that achieves 95% test accuracy with small weights is almost always better than one that hits 95.1% with a weight of 5000. The latter will fail on even slightly different test data.
Thinking bias-variance is always a tradeoff at fixed model size. You can reduce both bias and variance simultaneously by adding more high-quality training data. The tradeoff is between model families (simpler vs. more complex), not between individual models.
Forgetting that the irreducible error is a theoretical quantity. In the real world, you almost never know the true irreducible error — the inherent noise level of your problem. You can only estimate it.

Several students asked about the practical implications of bias-variance. The core insight: if your model has high bias (underfitting), collect more features or use a more flexible model. If your model has high variance (overfitting), collect more training examples or add regularization. This diagnostic framework — bias vs. variance — tells you which lever to pull.

Recap: Total error = reducible error + irreducible error, and irreducible = bias + variance. Bias is simplification error; variance is sensitivity error. They are two sides of a coin — lowering one raises the other. Overfitting means low bias, high variance, and exploding weights.

Bridge: The weight explosion tells you exactly what you need: a constraint on weight size. This brings us to Lagrange multipliers — the mathematical machinery that makes constrained optimization possible.

Real-world connection: In 2020, OpenAI trained GPT-3 with 175 billion parameters. The bias-variance tradeoff matters hugely at that scale. Without careful regularization — weight decay, dropout, data augmentation — such a model would overfit its internet text. That GPT-3 generalizes to new prompts shows gradient descent can work in 175-billion-dimensional space.

12.4 Constrained Optimization — Lagrange's Method

Hook: You want the highest possible profit, but you cannot spend more than your budget. You want the fastest car, but it must meet fuel-efficiency rules. You want the best model, but its weights cannot explode. Every "best" comes with a "but." Lagrange's method lets you solve both at once.

12.4.1 The Core Idea: Optimization with a Budget

Constrained optimization means you want to minimize or maximize an objective function $F (x, y)$ , but you are not free to explore the entire space. You have a constraint $G (x, y) = C$ (equality) or $G (x, y) \leq C$ (inequality) that limits where you can go.

Analogy: A hiker climbing a mountain chained to a fence. The hiker wants the highest altitude possible. But the chain limits how far from the fence they can go. The constrained optimum is the highest point within the chain's reach. Even though the true peak is higher and visible in the distance.

In ML: the objective function $F$ is your loss (MSE, cross-entropy). The constraint $G$ is a budget on weight size — $\sum β_{j}^{2} \leq C$ . You seek the lowest loss, but weights must stay within budget.

12.4.2 Geometric Interpretation

Consider a function $F (x, y)$ to maximize. It forms a 3D surface over the xy-plane. Its level curves — slices of equal height — are drawn as contour lines on the plane. The constraint $G (x, y) = C$ is a curve on that same plane.

The gradient $\nabla F$ is a vector that always points perpendicular to level curves, toward the direction of steepest increase — toward the peak. The gradient $\nabla G$ points perpendicular to the constraint curve.

At the constrained optimum, the constraint curve just barely "kisses" a level curve of $F$ . They touch at exactly one point without crossing. At that kiss-point, the two gradients are parallel — they point either in exactly the same direction or exact opposite directions.

Why? If the gradients were not parallel, the constraint curve would cut across the level curve. That means there is a point further along the constraint sitting on a higher level curve. So you haven't reached the optimum yet.

Visual intuition: Draw the xy-plane. Plot several concentric level curves of $F$ as ovals around the peak. Draw the constraint $G (x, y) = C$ as a curve. As the level curves expand outward from the peak, the optimum appears. It is where the constraint curve just grazes the outermost level curve it can reach. Like a ring expanding until it touches a wire. The contact point is the constrained optimum.

12.4.3 The Gradient Parallelism Condition

Because $\nabla F$ and $\nabla G$ are parallel at the optimum, one is a scalar multiple of the other:

$\nabla F = λ \nabla G$

Here $λ$ (lambda) is the Lagrange multiplier. It can be positive or negative, depending on whether the gradients point the same way or opposite ways. It measures the shadow price — how much the optimal value of $F$ would change if you relaxed the constraint by one unit.

This single vector equation, combined with the constraint equation $G (x, y) = C$ , gives you exactly enough equations to solve for $x$ , $y$ , and $λ$ .

12.4.4 Symbol Registry — Lagrange's Method

Symbol	Meaning	LaTeX	Type / Domain
$F (x, y)$	objective function to optimize	$F : R^{2} \to R$	real-valued
$G (x, y)$	constraint function	$G (x, y)$	real-valued
$\nabla F$	gradient of objective	$\nabla F = [\frac{\partial F}{\partial x}, \frac{\partial F}{\partial y}]$	2D vector
$\nabla G$	gradient of constraint	$\nabla G$	2D vector
$λ$	Lagrange multiplier	$λ \in R$	scalar
$C$	constraint bound	$C$	scalar constant

12.4.5 Lagrange's Algebraic Form

There is an equivalent algebraic formulation of the same idea. Construct the Lagrangian:

$L (x, y, λ) = F (x, y) - λ (G (x, y) - C)$

Then take partial derivatives with respect to $x$ , $y$ , and $λ$ , set each to zero, and solve the resulting system. The sign of $λ$ depends on whether you are maximizing ( $-$ ) or minimizing ( $+$ ):

$L (x, y, λ) = F (x, y) + λ (G (x, y) - C) (minimization form, common in ML)$

To see why this works: $\frac{\partial L}{\partial λ} = G (x, y) - C = 0$ recovers the constraint. $\frac{\partial L}{\partial x} = 0$ and $\frac{\partial L}{\partial y} = 0$ together give $\nabla F = \pm λ \nabla G$ — the gradient parallelism condition.

The algebraic and geometric approaches are mathematically identical. Choose whichever feels more natural. Both earn full exam credit.

12.4.6 Worked Example 1: Maximize $X Y$ Subject to $X + Y = 6$

Problem: Maximize $F (x, y) = x y$ such that $x + y = 6$ .

Step 1 — Write the constraint in standard form.

$G (x, y) = x + y - 6 = 0$

Step 2 — Compute both gradients.

$\nabla F = [\frac{\partial}{\partial x} (x y), \frac{\partial}{\partial y} (x y)] = [y, x]$

$\nabla G = [\frac{\partial}{\partial x} (x + y - 6), \frac{\partial}{\partial y} (x + y - 6)] = [1, 1]$

Step 3 — Apply the gradient parallelism condition $\nabla F = λ \nabla G$ .

$[y, x] = λ [1, 1]$

Component by component:

$y x = λ (1) = λ (2)$

This tells us $x = y = λ$ . The optimum must lie where $x$ equals $y$ .

Step 4 — Substitute into the constraint.

$x + y = 6 ⟹ λ + λ = 6 ⟹ 2 λ = 6 ⟹ λ = 3$

Step 5 — Read off the optimum.

$x = 3, y = 3$

Step 6 — Compute the maximum value.

$F (3, 3) = 3 \times 3 = 9$

Sense check: Among all pairs summing to 6 — like $(1, 5)$ , $(2, 4)$ , $(2.5, 3.5)$ — the product $1 \times 5 = 5$ , $2 \times 4 = 8$ , $2.5 \times 3.5 = 8.75$ . The product grows as the two numbers get closer, peaking when they are equal at $(3, 3)$ . The answer makes geometric sense: a square of fixed perimeter has the largest area among all rectangles.

12.4.7 Worked Example 2: Minimize $X^{2} + Y^{2}$ Subject to $X Y = 1$

Problem: Minimize $F (x, y) = x^{2} + y^{2}$ such that $x y = 1$ .

Step 1 — Write the constraint in standard form.

$G (x, y) = x y - 1 = 0$

Step 2 — Compute both gradients.

$\nabla F = [2 x, 2 y], \nabla G = [y, x]$

Step 3 — Set $\nabla F = λ \nabla G$ .

$[2 x, 2 y] = λ [y, x]$

Component by component:

$2 x 2 y = λ y (1) = λ x (2)$

Step 4 — Multiply (1) and (2).

$(2 x) (2 y) = (λ y) (λ x) ⟹ 4 x y = λ^{2} x y$

Since $x y = 1$ (the constraint):

$4 = λ^{2} ⟹ λ = \pm 2$

Step 5 — Case 1: $λ = 2$ .

From (1): $2 x = 2 y ⟹ x = y$ .

From the constraint: $x y = 1 ⟹ x^{2} = 1 ⟹ x = \pm 1$ .

So $(x, y) = (1, 1)$ or $(- 1, - 1)$ .

Objective value: $F (1, 1) = 1^{2} + 1^{2} = 2$ . Same for $F (- 1, - 1)$ .

Step 6 — Case 2: $λ = - 2$ .

From (1): $2 x = - 2 y ⟹ x = - y$ .

From the constraint: $(- y) (y) = 1 ⟹ - y^{2} = 1$ . No real solution. This case is impossible for real numbers — discard it.

Step 7 — State the result. The minimum value is 2, achieved at $(1, 1)$ and $(- 1, - 1)$ .

Sense check: On the curve $x y = 1$ (a hyperbola), the closest points to the origin minimize $x^{2} + y^{2}$ (squared distance). The points $(1, 1)$ and $(- 1, - 1)$ are symmetric across the origin and visually are the closest points on that hyperbola to $(0, 0)$ . Makes sense.

Scope / Assumptions:

The objective function $F$ and constraint $G$ must be differentiable — you need clean gradients for the parallelism condition.
Lagrange's method finds candidate stationary points. It does not tell you whether they are maxima, minima, or saddle points. You must check with a second-derivative test or by reasoning about the function's behavior.
For inequality constraints ( $G (x, y) \leq C$ ), the method gets more complex — you must also check the interior of the feasible region (the unconstrained optimum). The exam deals with equality constraints.
The method extends to any number of variables and to multiple constraints (one $λ_{i}$ per constraint), though the exam covers only two variables with one constraint.

12.4.8 Student Q&A on Lagrange's Method

Q: In the Lagrangian algebraic form, are we using the function itself or its gradient?

A: The algebraic form uses the function itself. You build $L (x, y, λ) = F (x, y) - λ (G (x, y) - C)$ . Then you take partial derivatives of $L$ — which is computing gradients. Setting them to zero and solving is algebraically equivalent to the geometric $\nabla F = λ \nabla G$ approach. Same answer, different route.

Q: Can there be multiple constraints?

A: Yes. For $k$ constraints $G_{1}, G_{2}, \dots, G_{k}$ , you add one Lagrange multiplier per constraint: $L = F - λ_{1} (G_{1} - C_{1}) - λ_{2} (G_{2} - C_{2}) - \dots$ . For your current course and examinations, stay focused on a single constraint — that is what you will be tested on.

Q: How do we visualize an inequality constraint like $x + y \leq 6$ ?

A: The line $x + y = 6$ is the boundary. The inequality $x + y \leq 6$ includes that line plus the entire region on the origin side of it. When the constraint is active, the solution lies on the boundary. When the constraint is slack, the solution lies strictly inside. For exam equality-constraint problems, the solution will always be on the boundary.

Several students asked about the difference between the Lagrangian method and finding local extrema. The key distinction: standard calculus (set $\nabla F = 0$ ) finds unconstrained stationary points. Lagrange multipliers find stationary points confined to a constraint surface. The constraint restricts the search space; $λ$ enforces that restriction mathematically.

Recap: Lagrange's method finds the optimum of $F$ confined to the curve $G = C$ . The key insight: at the optimum, $\nabla F$ and $\nabla G$ are parallel, so $\nabla F = λ \nabla G$ . Together with the constraint itself, this gives a solvable system. Two equivalent approaches exist — the geometric gradient-parallelism method and the algebraic Lagrangian — pick either.

Bridge: Now we take Lagrange's method and apply it directly to the weight explosion problem. This gives us regularization — the single most important tool for preventing overfitting in machine learning.

Real-world connection: Joseph-Louis Lagrange invented these multipliers in 1788 for classical mechanics, like modeling a pendulum. Today they underpin nearly all regularized machine learning. Calling `Ridge(alpha=0.1)` in scikit-learn or setting `weight_decay=1e-4` in PyTorch uses Lagrange's method. Google's OR-Tools uses Lagrange relaxation for logistics and routing at massive scale.

12.5 Regularization as Constrained Optimization

Hook: You saw weights explode to 2206 when a 13th-degree polynomial overfits. The solution is obvious in principle: put a cap on weight size. But how do you turn that cap into a practical optimization procedure that gradient descent can handle? The answer is Lagrange multipliers — and the result is called regularization.

12.5.1 Symbol Registry — Regularization

Symbol	Meaning	LaTeX	Type / Domain
MSE	Mean Squared Error (loss)	$MSE$	scalar $\geq 0$
$β_{j}$	model coefficient for feature $j$	$β_{j}$	scalar (real)
$λ$	regularization strength	$λ$	scalar $\geq 0$
$C$	constraint budget (hard cap)	$C$	scalar $\geq 0$

12.5.2 L2 Regularization (Ridge)

Ridge regression (L2 regularization) adds a penalty proportional to the sum of squared weights:

$Minimize: MSE + λ j = 1 \sum d β_{j}^{2}$

In constrained form, this is equivalent to:

$Minimize MSE such that j = 1 \sum d β_{j}^{2} \leq C$

The two forms are mathematically equivalent. For any $λ$ value, there is a corresponding budget $C$ . For any budget $C$ , there is a corresponding Lagrange multiplier $λ$ . The penalized form (MSE $+ λ \sum β_{j}^{2}$ ) is what you actually code and optimize. The constrained form ( $\sum β_{j}^{2} \leq C$ ) is the theoretical justification — it shows you exactly why regularization prevents overfitting: it caps weight magnitude.

Analogy: A speed limiter on a car. The driver (gradient descent) wants to go as fast as possible toward minimal loss. The limiter ( $λ$ ) caps the maximum speed to prevent dangerous overshooting (weight explosion). A high $λ$ = a very restrictive limiter (weights kept near zero). A low $λ$ = a permissive limiter (weights can grow more freely).

Where the analogy breaks: the speed limiter is a hard cutoff. Ridge penalizes large weights continuously. The bigger the weight, the bigger the penalty. This creates a smooth incentive to stay small rather than an absolute wall.

Ridge never pushes weights all the way to zero. It shrinks them proportionally toward zero but never eliminates them entirely. This is why ridge keeps all features in the model — it just reins in the extreme ones.

12.5.3 L1 Regularization (Lasso)

Lasso (L1 regularization) uses absolute values instead of squares:

$Minimize: MSE + λ j = 1 \sum d ∣ β_{j} ∣$

The equivalent constrained form:

$Minimize MSE such that j = 1 \sum d ∣ β_{j} ∣ \leq C$

Lasso has a critical property that ridge lacks: it can push weights all the way to zero. This performs automatic feature selection — coefficients that are not useful simply disappear. The model becomes sparse.

Why the difference? The geometry of the constraint regions explains it. The L2 constraint $\sum β_{j}^{2} \leq C$ is a sphere (circle in 2D). The L1 constraint $\sum ∣ β_{j} ∣ \leq C$ is a diamond (in 2D). The diamond has sharp corners on the axes. When the MSE contours hit a corner of the diamond, one of the coefficients becomes exactly zero. The sphere has no sharp corners, so ridge almost never produces exact zeros.

Visual intuition: Draw the 2D plane with $β_{1}$ on the x-axis and $β_{2}$ on the y-axis. Draw the L2 constraint as a circle centered at the origin with radius $C$ . Draw the L1 constraint as a diamond with vertices at $(C, 0)$ , $(- C, 0)$ , $(0, C)$ , $(0, - C)$ . Overlay elliptical MSE contours (concentric ellipses centered at the unconstrained optimum). For ridge, the optimum is where an ellipse first touches the circle. That is typically in the interior of a quadrant, giving two nonzero coefficients. For lasso, the optimum is often where an ellipse hits a corner of the diamond. That is on an axis, giving one zero coefficient.

Property	Ridge (L2)	Lasso (L1)
Penalty term	$λ \sum β_{j}^{2}$	$λ \sum ∣ β_{j} ∣$
Constraint shape	Sphere (smooth)	Diamond (sharp corners)
Shrinks to zero?	No — shrinks but never eliminates	Yes — produces exact zeros
Feature selection	No — keeps all features	Yes — automatic feature selection
Differentiability	Everywhere differentiable	Not differentiable at zero
When to use	Many small/medium features, all relevant	Many features, only some relevant
Solution path	Smooth shrinkage	Piecewise linear; sudden drops to zero

12.5.4 The Sign of Lambda

The sign of $λ$ in the Lagrangian depends on how the gradients relate. When both point the same direction (both trying to maximize), use $\nabla F = λ \nabla G$ . When they point opposite directions (one maximizing, one minimizing), use $\nabla F = - λ \nabla G$ .

In the regularization setting: you are minimizing MSE (going downhill). The constraint $\sum β_{j}^{2} \leq C$ is trying to shrink weights — pulling them toward the origin. The MSE gradient for a complex model is trying to explode them outward. These gradients point in opposite directions. That gives:

$\nabla (MSE) = - λ \nabla (\sum β_{j}^{2}) ⟹ \nabla (MSE) + λ \nabla (\sum β_{j}^{2}) = 0$

This is why the penalized form reads $MSE + λ \sum β_{j}^{2}$ — the plus sign comes from the opposing gradient directions in the Lagrangian algebra. The constraint's gradient gets a positive sign in the final objective.

Pitfalls:

Setting $λ$ too high (ridge). All weights get pushed too close to zero, the model becomes a near-constant predictor, and you get high bias with low variance — underfitting.
Setting $λ$ too low. The penalty is negligible, weights can still explode, and the model overfits just like an unregularized model.
Using ridge when you need feature selection. If only 3 out of 500 features matter, ridge will still try to use all 500 (with tiny coefficients). Lasso will zero out the 497 irrelevant ones.
Forgetting that $λ$ is optimized separately. The best $λ$ is found through cross-validation, not gradient descent on the main loss. You try different $λ$ values, evaluate on a validation set, and pick the best.

12.5.5 Student Q&A on Regularization

Q: We have used regularization in ML with a $λ$ parameter (like $λ = 0.1$ ). We never set a $C$ value as a budget. Where does $C$ fit in?

A: The two forms are dual representations of the same idea. "Minimize MSE + $λ \sum β_{j}^{2}$ " (penalized form) and "Minimize MSE such that $\sum β_{j}^{2} \leq C$ " (constrained form) are mathematically equivalent via Lagrange duality. For any $λ$ , there is a corresponding $C$ ; for any $C$ , there is a corresponding $λ$ . The penalized form is what you implement. The constrained form is the theory that explains why it works. It puts a hard ceiling on weight magnitude. That prevents the explosion you saw in the degree-13 polynomial.

Recap: Regularization is constrained optimization applied to model weights. Ridge (L2) uses a squared penalty — it shrinks all weights but never kills them. Lasso (L1) uses an absolute value penalty — it can push irrelevant weights all the way to zero. That gives automatic feature selection. The tuning parameter $λ$ is found through cross-validation.

Exam note: The bias-variance tradeoff, overfitting mechanisms, and regularization as constrained optimization are conceptual exam topics. Expect short-answer or multiple-choice questions on Ridge vs. Lasso, and on the equivalence between the penalized form and the constrained form. Lagrange's method is the guaranteed problem-solving question (5-10 marks).

Bridge: Regularization fights the symptom (weight explosion). But the root cause is often unequal feature scales creating uneven curvature. To treat the root cause, we need feature scaling.

Real-world connection: Every major deep learning framework includes L2 regularization, called "weight decay" in PyTorch. When Meta trained LLaMA-2 (70 billion parameters), weight decay was critical — without it the model would have exploded on terabytes of text. The Hugging Face hub relies on regularization so fine-tuned weights generalize.

12.6 Feature Processing and Scaling

Hook: Picture predicting income from two features: age (22 to 65) and salary in rupees (200,000 to 5,000,000). The optimizer sees salary moving over millions and age moving in dozens. Guess which feature it pays attention to? The one with bigger numbers — even if age is actually the better predictor.

12.6.1 Symbol Registry — Scaling

Symbol	Meaning	LaTeX	Type / Domain
$x$	original feature value	$x$	scalar
$μ$	population mean	$μ$	scalar
$x ˉ$	sample mean	$x ˉ$	scalar
$σ$	population standard deviation	$σ$	scalar $\geq 0$
$s$	sample standard deviation	$s$	scalar $\geq 0$
$z$	z-score (standardized value)	$z$	scalar, $μ_{z} = 0, σ_{z} = 1$
$x_{m i n}$	minimum value in feature	$x_{m i n}$	scalar
$x_{m a x}$	maximum value in feature	$x_{m a x}$	scalar
$IQR$	interquartile range	$IQR = Q_{3} - Q_{1}$	scalar $\geq 0$

12.6.2 Why Scaling Matters for Optimization

Analogy: Think of the loss surface as a stretchable rubber sheet. When features have equal scales, the sheet is pulled into a perfect symmetric bowl — a marble rolls straight to the center. When one feature has much larger variance, the sheet is stretched into a long, narrow ravine. The marble ricochets between the steep walls while barely creeping along the shallow floor.

The optimizer faces the same problem: it ricochets (oscillates) across the narrow dimension and barely moves in the flat direction.

When features have different scales, the loss surface tilts more in the direction of the high-variance feature. It becomes elongated — a narrow ravine rather than a symmetric bowl. Gradient descent struggles on this terrain. It oscillates (Section 12.8). It gets stuck on plateaus.

Scaling brings all features to a comparable range. The loss surface becomes more bowl-shaped. Gradient descent converges faster and more reliably. This is not just a nice-to-have — it is a prerequisite for stable optimization.

Scope: Scaling addresses uneven curvature from feature scale differences. It does not fix non-convexity caused by model architecture (neural network nonlinearities). A scaled loss surface for a neural network is still non-convex — it is just less elongated. That is still a big improvement.

12.6.3 Scaling Methods Overview

Method	Formula	Key property	Best for
Mean centering	$x - x ˉ$	Centers at zero; scale unchanged	Preprocessing step
Z-score	$\frac{x - μ}{σ}$	Unit variance; centered at zero	Default for optimization
Min-max	$\frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}$	Bounds to $[0, 1]$	Images, neural nets
Robust (IQR)	$\frac{x - median}{IQR}$	Resilient to outliers	Data with extreme values
Max-absolute	$\frac{x}{< t d > x _{m a x} < / t d >< t d >}$	Bounds to $[- 1, 1]$	Already-centered data

12.6.4 Z-Score Standardization

The z-score transformation standardizes each feature to have mean 0 and standard deviation 1:

$z = \frac{x - μ}{σ}$

In practice, you replace the population parameters $(μ, σ)$ with their sample estimates $(x ˉ, s)$ , since you rarely know the true population values.

After z-score scaling, every feature has exactly the same spread — unit variance. No single variable dominates the loss surface. This is called normalization (confusingly, the term is overloaded — here it means scaling to zero mean, unit variance).

Concrete example: Raw ages: $[30, 35, 45, 65]$ .

Sample mean: $x ˉ = 43.75$
Sample std: $s = \frac{( 30 - 43.75 ) ^{2} + ( 35 - 43.75 ) ^{2} + ( 45 - 43.75 ) ^{2} + ( 65 - 43.75 ) ^{2}}{3} = \frac{189.06 + 76.56 + 1.56 + 451.56}{3} \approx 15.48$
Z-scores: $[(30 - 43.75) /15.48, (35 - 43.75) /15.48, (45 - 43.75) /15.48, (65 - 43.75) /15.48] = [- 0.89, - 0.57, 0.08, 1.37]$

The transformed values have mean 0 and standard deviation 1. The model now sees these balanced values, not the raw ages.

For optimization tasks, z-score is the recommended starting point. It gives unit variance — every feature has equal pull on the loss surface.

12.6.5 Min-Max Scaling

Min-max squashes values into $[0, 1]$ :

$x_{scaled} = \frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}$

Widely used in image processing and neural networks. Pixel values are naturally non-negative (0-255), so bounding to $[0, 1]$ is natural.

Weakness: Min-max does not enforce unit variance. The scaled features can still have very different spreads. Consider a feature with raw range [10, 10000]. After min-max scaling, most values cluster near 0.001. A few extreme values approach 1.0. The distribution stays lopsided. Gradient descent still struggles.

12.6.6 Other Scaling Methods

Mean centering ( $x - x ˉ$ ): Centers at zero but preserves spread. Alone, it does not solve the uneven-curvature problem — it is a preprocessing step, not a complete solution.
Robust scaling ( $\frac{x - median}{IQR}$ ): Uses median andinterquartile range (IQR = $Q_{3} - Q_{1}$ ) instead of mean andstandard deviation. The best defense whenyour data has extreme outliers —the median and IQR barely budge with outliers.
Max-absolute scaling ( $\frac{x}{∣ x _{m a x} ∣}$ ): Bounds to $[- 1, 1]$ . Useful when data is already centered near zero and you only need range normalization.

12.6.7 Scaling and the Loss Surface Geometry

Here is what happens to the loss surface:

Scenario	Loss surface shape	Gradient descent behavior
Unscaled data	Tapered, elongated ravine	Oscillates in steep direction; creeps in flat direction; may stall on plateaus
Z-score scaled	Near-convex, symmetric bowl	Converges cleanly; every start point has clear path to minimum
Min-max scaled (low correlation)	Still tapered	Min-max does not equalize variance; the taper remains; gradient descent still oscillates

Z-score consistently produces a near-convex surface. Min-max, even with low correlation, can still leave a tapered, non-convex surface because it does not forcibly equalize variance.

Visual intuition: Imagine three contour plots (top-down view of the loss surface), each with $β_{1}$ on the x-axis and $β_{2}$ on the y-axis.

Unscaled: Contours are extremely elongated ellipses stretched along one axis — looks like a cigar. Gradient descent zigzags perpendicular to the long axis.
Z-score: Contours are nearly perfect circles. Gradient descent follows a smooth inward spiral to the center.
Min-max: Contours are still somewhat elongated (though less than unscaled). Zigzagging persists, just less severe.

12.6.8 Student Q&A on Scaling

Q: Does every neural network problem need z-score normalization as the first step?

A: Yes — always normalize input data before training. For structured (tabular) data with varying scales, z-score is the strong default. For images (pixel values 0-255), min-max scaling to $[0, 1]$ is standard and works well. For text embeddings, normalization depends on the embedding method. The key rule: you must scale. Which method you pick depends on your data type.

Q: Is scaling needed for text data?

A: Yes. Normalization applies to all data formats — tables, images, text representations. However, text is different: raw characters or word counts may need TF-IDF normalization or embedding-based representations that are already approximately normalized. The principle stays the same — all features should have comparable scale.

Q: Can z-score ever be worse than min-max?

A: In edge cases, yes. If your data is extremely non-normal or has severe outliers, robust scaling (IQR-based) may be better. The practical workflow: try z-score first; if it underperforms, test min-max and robust scaling. In practice, accuracies from both methods are often close, with z-score giving a slight edge for optimization-heavy workflows.

Several students asked why scaling is necessary if both train and test data use the same scale. There are two answers. First, computational: gradient descent converges faster and more reliably on a well-scaled surface. Second, statistical: ML models are biased toward features with higher variance. Salary (200K-5M) will dominate age (22-65) simply because its numbers are bigger — even if age is the genuinely more informative predictor. The model wastes effort learning from the wrong variable. Scaling removes this numeric bias.

Recap: Feature scaling transforms all features to comparable ranges. It turns an elongated, oscillatory loss surface into a symmetric bowl. Z-score ( $z = (x - μ) / σ$ ) is the default: every feature gets mean 0 and variance 1. Min-max bounds to [0,1] and suits images. Robust scaling (median/IQR) handles outliers. The action matters more than the choice — you must scale.

Exam note: Scaling methods and their effect on loss surface geometry are conceptual exam material. Expect short-answer or multiple-choice questions comparing z-score, min-max, and the consequences of not scaling.

Bridge: Scaling fixes uneven curvature from the input side. But to measure curvature directly — to diagnose it with numbers, not just visual inspection — you need a mathematical tool. That tool is the Hessian matrix.

Real-world connection: At web-scale companies like Google and Meta, feature normalization is step one of every pipeline. Google's TFX includes `tft.scale_to_z_score` as a built-in transform. Training recommenders on billions of interactions with wildly different feature ranges makes z-score mandatory. Without it, gradient descent chases the biggest numbers and misses what predicts behavior.

12.7 The Hessian Matrix and Loss Surface Curvature

Hook: The gradient tells you which direction is steepest — a first-order measurement. But it cannot tell you whether the floor under your feet is bowl-shaped, saddle-shaped, or flat. For that, you need second-order information. You need the Hessian.

12.7.1 Symbol Registry — Hessian Matrix

Symbol	Meaning	LaTeX	Type / Domain
$H$	Hessian matrix	$H \in R^{d \times d}$	symmetric matrix
$\frac{\partial ^{2} L}{\partial x ^{2}}$	second partial derivative w.r.t. $x$	$\frac{\partial ^{2} L}{\partial x ^{2}}$	scalar
$\frac{\partial ^{2} L}{\partial y ^{2}}$	second partial derivative w.r.t. $y$	$\frac{\partial ^{2} L}{\partial y ^{2}}$	scalar
$\frac{\partial ^{2} L}{\partial x \partial y}$	mixed partial derivative	$\frac{\partial ^{2} L}{\partial x \partial y}$	scalar

12.7.2 Definition and Role

The Hessian matrix is the matrix of all second partial derivatives of the loss function. While the gradient $\nabla L$ gives the first-order slope, the Hessian $H$ gives the second-order curvature — how the slope itself changes as you move.

For a loss function $L$ of two variables:

$H = \frac{\partial ^{2} L}{\partial x ^{2}} \frac{\partial ^{2} L}{\partial y \partial x} \frac{\partial ^{2} L}{\partial x \partial y} \frac{\partial ^{2} L}{\partial y ^{2}}$

The diagonal entries ( $\frac{\partial ^{2} L}{\partial x ^{2}}$ , $\frac{\partial ^{2} L}{\partial y ^{2}}$ ) measure the curvature along each axis independently. The off-diagonal entries ( $\frac{\partial ^{2} L}{\partial x \partial y} = \frac{\partial ^{2} L}{\partial y \partial x}$ , always equal for smooth functions by Clairaut's theorem) measure how curvature changes when both variables move together. This is the correlation effect.

Analogy: The gradient is like a speedometer — it tells you how fast you are going and in what direction. The Hessian is the curvature of the road. It tells you whether the road ahead is straight (zero curvature), bending left (positive), or bending right (negative). A speedometer alone cannot tell you whether you are on a flat highway or a winding mountain pass. The Hessian can.

Or more precisely: the gradient is the slope under your foot. The Hessian is how that slope changes as you take a tiny step — the rate of change of the rate of change.

12.7.3 Eigenvalues and Correlation Effects

The eigenvalues of the Hessian numerically capture the strength of curvature in each direction. For a 2-variable problem, there are two eigenvalues $λ_{1}$ and $λ_{2}$ :

Both large and roughly equal → symmetric bowl (good for optimization).
One huge, one tiny → narrow valley (bad — gradient descent oscillates).
One positive, one negative → saddle point (gradient descent can get stuck in high dimensions).

When features are uncorrelated: the off-diagonal entries of $H$ are near zero. The eigenvalues are roughly equal. The loss surface is a symmetric bowl.

When features are highly correlated: the off-diagonal entries grow large. The curvature in the joint direction (both variables moving together) far exceeds the curvature in the orthogonal direction. One eigenvalue becomes huge, the other small. This signals a narrow-valley loss surface — exactly the terrain where gradient descent oscillates.

Numerical spot-check: Consider two perfectly correlated features. The loss $L (w_{1}, w_{2}) = (w_{1} + w_{2})^{2}$ . The Hessian:

$\frac{\partial ^{2} L}{\partial w _{1}^{2}} = 2, \frac{\partial ^{2} L}{\partial w _{2}^{2}} = 2, \frac{\partial ^{2} L}{\partial w _{1} \partial w _{2}} = 2$

$H = [2222]$

Eigenvalues: $λ_{1} = 4$ , $λ_{2} = 0$ . The zero eigenvalue means there is a direction with no curvature at all. The loss is perfectly flat along that direction ( $w_{1} = - w_{2}$ ). Gradient descent on this surface would sprint along the steep direction and drift aimlessly on the flat one. The condition number $κ = λ_{m a x} / λ_{m i n}$ is infinite — the worst-case scenario.

A practical consequence: if two features are highly correlated, you can often drop one without losing information. Keeping both creates unnecessary curvature problems.

12.7.4 Student Q&A on Correlation and Loss Surfaces

Q: How does reducing correlation give us a bowl shape?

A: The off-diagonal entries of the Hessian ( $\frac{\partial ^{2} L}{\partial x \partial y}$ ) measure how loss changes when both variables shift in tandem. High correlation → large off-diagonals → the loss surface is stretched along the joint direction, like a tilted, elongated ravine. Low correlation → off-diagonals near zero → the surface is symmetric. Both axes curve the same amount. That is the bowl shape you want. Reducing correlation (by dropping redundant features or applying PCA) directly shrinks those off-diagonal entries.

Pitfalls:

Thinking the Hessian is only for two variables. The Hessian generalizes to any number of parameters. For $d$ parameters, $H \in R^{d \times d}$ with $d (d + 1) /2$ unique entries (it is symmetric). Computing and storing it becomes expensive for large $d$ .
Computing the full Hessian for deep learning. For a million-parameter network, the Hessian has $1 0^{12}$ entries — impossible to store. Practical optimizers (Adam, RMSprop) approximate second-order information using only diagonal or low-rank estimates.
Misinterpreting eigenvalues. A zero eigenvalue means the loss surface is flat in that eigenvector's direction — a ridge or plateau. This is common when features are perfectly correlated. A negative eigenvalue means the stationary point is a saddle, not a minimum.
Forgetting the Hessian informs the learning rate choice. The optimal learning rate for gradient descent is approximately $1/ λ_{m a x}$ , where $λ_{m a x}$ is the largest eigenvalue of the Hessian at the minimum. A large eigenvalue spread (ill-conditioned Hessian) forces a very small learning rate — slow convergence.

Recap: The Hessian is the matrix of second derivatives — it captures loss surface curvature. The diagonal entries give per-axis curvature. The off-diagonals encode correlation effects. The eigenvalues reveal shape: roughly equal means a bowl (good); one huge and one tiny means a narrow valley (bad). Highly correlated features inflate off-diagonals.

Exam note: The Hessian and its eigenvalues as a curvature diagnostic tool may appear as a conceptual exam question.

Bridge: Curvature problems from correlated features and uneven scaling produce a practical symptom: oscillation. Next we explore how to detect oscillation, how outliers confuse scaling, and why Bessel's correction is hiding in your standard deviation calculation.

The Hessian underpins Newton's method, used in R's `nlm` and`optim` functions. In deep learning the full Hessian is too big to compute. But the Fisher Information Matrix is approximated instead. It is the expected Hessian of the negative log-likelihood. Natural gradient descent and K-FAC use it — a core optimizer in some of DeepMind's efficient RL agents.

12.8 Practical Issues: Oscillation, Outliers, and Degrees of Freedom

Hook: Gradient descent has stopped. The loss is flat. You think you have converged. But how do you know you are not just oscillating in place? You might be bouncing between valley walls, going nowhere, while the real minimum sits untouched in a nearby basin.

12.8.1 Symbol Registry — Statistics

Symbol	Meaning	LaTeX	Type / Domain
$s$	sample standard deviation	$s$	scalar $\geq 0$
$n$	sample size	$n$	integer $\geq 2$
$x ˉ$	sample mean	$x ˉ$	scalar
$df$	degrees of freedom	$df = n - 1$	integer

12.8.2 Gradient Descent Oscillation

When the loss surface is a narrow valley (uneven curvature from Section 12.2), gradient descent can fall into an oscillation trap. The algorithm bounces from one wall of the valley to the other. It makes slow progress downward, but may never settle at the minimum.

Analogy: A ping-pong ball in a bathtub. You push it toward the drain. But the tub's walls are steep and the ball has momentum. It hits one wall, bounces, hits the opposite wall, bounces again. It gets closer to the drain, but never quite settles there — it keeps ricocheting.

Three scenarios for different learning rates on uneven terrain:

Learning rate $α$	Behavior	Outcome
Very high	Diverges — bounces out of the optimization region entirely	Training fails; loss → $\infty$
Moderately high	Oscillates — zigzags in the narrow valley, very slow progress	May look converged; loss plateaus
Very low	Creeps — gradient near zero on shallow slopes; stalls	Premature stopping far from true minimum

The fundamental problem: from a single run, you cannot distinguish convergence from oscillation. Both look like a flat loss curve.

Diagnosis strategy: You can never be sure from a single run. To get certainty:

Run with different learning rates. If $α = 0.01$ and $α = 0.001$ converge to drastically different loss values, one of them was oscillating.
Run from different random starting points. If starting points A, B, C all converge to the same loss, you have probably found a true basin. If they diverge, you were oscillating on ridges.
Compare optimizers: vanilla SGD vs. SGD with momentum vs. Adam. Momentum dampens oscillation — if it reaches a lower loss, the vanilla run was oscillating.
Track the loss curve shape. If the loss plateaus and then suddenly drops when you switch to a slightly different $α$ , you were oscillating.

12.8.3 Outliers and Their Effect on Scaling

Outliers are data points far from the bulk of the distribution. They affect scaling methods differently.

Min-max scaling is catastrophically sensitive to outliers. A single extreme value inflates the denominator $x_{m a x} - x_{m i n}$ . All non-outlier values get compressed into a tiny subrange near zero. The scaling becomes useless — almost all values look identical.
Z-score is more resilient. An outlier perturbs the estimates of $μ$ and $σ$ , but the effect is bounded. The transformed data stays usable because the outlier's extreme z-score is just one number — it does not compress everything else.
Robust scaling (IQR) is the strongest defense. It uses the median and IQR, which barely move when outliers appear. The extreme tails are ignored entirely.

Consider the values $[10, 12, 11, 13, 10, 500]$ where 500 is an outlier.

Min-max: Range = $500 - 10 = 490$ . Scaled values: $[0, 0.004, 0.002, 0.006, 0, 1.0]$ . Every normal point becomes near-zero — they all look identical.

Z-score: $x ˉ \approx 92.7$ , $s \approx 199$ . Z-scores: $[- 0.42, - 0.41, - 0.41, - 0.40, - 0.42, 2.05]$ . The normal points cluster together but remain distinguishable. The outlier stands out but does not destroy the scale.

Robust: Median = 11.5, IQR = $13 - 10.25 = 2.75$ . Scaled values: $[- 0.55, 0.18, - 0.18, 0.55, - 0.55, 177.6]$ . Normal points are well spread around zero. The outlier is flagged.

12.8.4 Bessel's Correction and Degrees of Freedom

The sample standard deviation uses $n - 1$ in the denominator, not $n$ :

$s = \frac{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2}}{n - 1}$

This is called Bessel's correction (statistical adjustment). Without it, $s$ systematically underestimates the true population standard deviation $σ$ .

Why $n - 1$ ? Because the sample mean $x ˉ$ is itself estimated from the same data. That estimation consumes one degree of freedom. You lose one independent piece of information because the deviations $x_{i} - x ˉ$ must sum to exactly zero — a constraint. Only $n - 1$ of those deviations are free to vary.

Analogy: A football (soccer) team with 11 players. You can freely position 10 of them anywhere on the field. The 11th player's position is forced — they must fill the last remaining spot for the formation to work. Only $n - 1 = 10$ players have freedom. That is degrees of freedom in a nutshell.

Or: you have three numbers that must average to 10. You freely choose 5 and 12. The third is forced to be 13. Two free choices, one forced. $n - 1 = 2$ .

Dividing by $n - 1$ rather than $n$ corrects for this lost degree of freedom. It makes $s^{2}$ an unbiased estimator of $σ^{2}$ — on average over many samples, it hits the true value. The correction matters most for small $n$ . For large $n$ , $n - 1 \approx n$ and the difference vanishes.

Pitfalls:

Using $n$ instead of $n - 1$ for small samples. With $n = 5$ , Bessel's correction changes the variance estimate by 25%. Ignoring it means underestimating the true variability of your data.
Confusing population and sample formulas. $σ = \frac{\sum ( x _{i} - μ ) ^{2}}{n}$ (divide by $n$ ) is for the population when you know the true mean. $s = \frac{\sum ( x _{i} - x ˉ ) ^{2}}{n - 1}$ (divide by $n - 1$ ) is for a sample when you estimate the mean. Using the wrong denominator in the wrong context is a classic statistics error.
Thinking Bessel's correction makes $s$ unbiased. Bessel's correction makes $s^{2}$ (the variance) unbiased, but $s$ (standard deviation) itself remains slightly biased — the square root of an unbiased estimator is not itself unbiased. For most ML purposes, this subtlety does not matter.

12.8.5 Student Q&A on Oscillation and Detection

Several students asked the same core question: how do you know if gradient descent has truly converged vs. just oscillating?

A: You cannot know from a single run. You only see the final parameters and loss value. To diagnose, you must compare. Run with different learning rates. Run with momentum. Run from different starting points. If a lower $α$ gives the same loss, you likely converged. If a slightly different setup yields a lower loss, your first run was oscillating. In high dimensions (hundreds of features), the loss surface is invisible — the only diagnostic is the loss value itself. That is exactly why practitioners try multiple optimization strategies and pick whichever reaches the lowest error on a validation set.

Recap: Gradient descent oscillation masquerades as convergence — you cannot tell them apart from a single run. Outlier sensitivity varies by scaling method: min-max is worst, z-score is moderate, robust (IQR) is best. Bessel's correction ( $n - 1$ ) accounts for the lost degree of freedom when estimating sample variance. It is a small-sample correction with deep statistical justification.

Bridge: These practical issues — oscillation diagnosis, scaling choices, statistical corrections — all connect back to the core principles of the lecture. They are gradient descent navigation on a loss surface, constrained optimization, and feature preparation. The final section surveys the tools and real-world practices that put these principles into action.

Real-world connection: Production systems automate oscillation detection with learning rate schedulers andearly stopping on validation loss. Tools like Weights & Biases and TensorBoard plot loss curves live. When Meta trains on thousands of GPUs for weeks, engineers watch for plateau-and-drop patterns and adjust $α$ or switch optimizers to escape.

12.9 Tools and Practical Workflows

12.9.1 Data Science Languages

Two primary ecosystems exist for optimization and ML work:

Python / scikit-learn / PyTorch / TensorFlow dominates modern deep learning. PyTorch and TensorFlow implement gradient descent, adaptive optimizers (Adam, RMSprop), and automatic differentiation. Scikit-learn provides `Ridge`, `Lasso`, `StandardScaler`, and `MinMaxScaler` — each a direct implementation of concepts from this lecture. Google OR-Tools (optimization library) handles constrained optimization at production scale.

R (statistical language) powers academic statistics and many industry analytics teams. Its optimization packages (`optim`, `nlm`, `glmnet`) are mature and battle-tested. R Shiny (interactive web framework) creates interactive web apps for visualization. These are exactly the kind used in this lecture to demonstrate loss surface geometry.

The choice between Python and R depends on your domain. Both implement the same math.

12.9.2 Visualization for Understanding Optimization

Interactive visualizations build intuition that equations alone cannot provide. The lecture demonstrations include:

Polynomial regression demo (degree 1–13): Watch the loss surface morph from a convex bowl into a jagged landscape as model complexity increases. Overfitting becomes visually obvious.
Gradient descent trajectory demo: Compare how the optimizer moves on unscaled vs. z-score-scaled data. The unscaled path zigzags; the scaled path is a smooth curve.
Hessian eigenvalue demo: See the eigenvalue spread widen as feature correlation increases — the numerical signature of a narrow-valley loss surface.

These are available as published Shiny applications and HTML widgets.

12.9.3 Learning with AI-Assisted Tools

NotebookLM (AI learning tool) can synthesize course materials — PDFs, lecture notes, transcripts — into structured knowledge graphs. You can generate mind maps showing how each concept connects, identify gaps, and build a navigable study map. Load all your sources; the tool maps the topology of the subject.

12.9.4 Why Some Model Weights Are Open-Sourced

LLaMA (Meta open-source model series) has weights from 7B to 70B parameters released as open source. GPT-2, GPT-4, GPT-5 (OpenAI language models) illustrate the pattern: GPT-2 weights are open; GPT-4 and GPT-5 are not. Why?

The weights are a product — they let you run inference and fine-tune. The training procedure — the optimization pipeline, the reinforcement learning from human feedback, the data mixture — is the secret sauce. If competitors had that, they could replicate the model from scratch. The weights alone are valuable but not the competitive advantage. The recipe is.

This matters for your understanding because the entire optimization pipeline you studied — gradient descent, learning rates, regularization, scaling — is exactly what remains proprietary. Companies expose the learned parameters; they do not expose the optimization choices that produced them.

Recap: Python (PyTorch, scikit-learn) and R are the primary tools for gradient descent and regularization. Interactive visualization builds geometric intuition for abstract optimization ideas. Open-source weights let you run and fine-tune models. But the optimization pipeline that trained them is the closely guarded competitive advantage.

Bridge: This completes the conceptual material for Lecture 12. See below for exam guidance and key industry applications — your study priorities and real-world context.

Real-world connection: When you download a model from Hugging Face, you get this lecture's optimization in action. Millions of weights, each learned through billions of gradient steps on huge clusters. They are regularized with weight decay and scaled inputs. The `model.safetensors` file is the result of everything covered here.

Exam Guidance Summary

Exam note: This lecture carries significant exam weight — 5 to 10 marks out of 40.

Highest-priority topics (problem-solving questions)

Constrained optimization using Lagrange's method is guaranteed — expect a problem where you are given $F (x, y)$ and $G (x, y) = C$ and asked to find the constrained extremum.

Worked example types to master:

Maximize $x y$ subject to $x + y = constant$ (see Section 12.4.6)
Minimize $x^{2} + y^{2}$ subject to $x y = C$ (see Section 12.4.7)
Similar two-variable problems with polynomial constraints

Key facts:

Exam problems involve one constraint only.
Both the gradient-parallelism method ( $\nabla F = λ \nabla G$ ) and the algebraic Lagrangian method ( $L = F - λ (G - C)$ ) are valid. Either earns full credit.
You must handle both sign cases for $λ$ and discard impossible solutions.

Conceptual topics (short-answer / multiple-choice)

Bias-variance tradeoff: definitions, tradeoff mechanism, which models are high/low bias/variance, the overfitting connection.
Overfitting mechanisms: how model complexity creates local minima, the weight explosion signature.
Scaling methods: z-score vs. min-max vs. robust; their effects on loss surface geometry; which to use when.
Hessian matrix: role in curvature measurement; eigenvalues as diagnostic tools; correlation effects on off-diagonal entries.
Regularization: Ridge (L2) vs. Lasso (L1); penalized form vs. constrained form; the role of $λ$ and $C$ .

Upcoming topics

Line search (adaptive learning rate selection): has not appeared in the past two years' exams but is likely to appear in a future session. The professor flagged it as material that will be covered.

Study strategy

Practice 3-4 Lagrange multiplier problems with different constraint shapes (linear, hyperbolic, quadratic).
Be able to draw and explain the bias-variance tradeoff curve.
Know the formulas for z-score, min-max, and robust scaling — and when each is appropriate.
Understand the Hessian's structure, what its eigenvalues mean, and why correlated features are problematic.
Be able to compare Ridge and Lasso on feature selection, zero-shrinkage, and the geometry of their constraint regions.

Key Industry Applications

Large Language Models (GPT-4, LLaMA): These models use high-dimensional embeddings (1024+ dimensions) with billions of parameters. The weight explosion problem is real — regularization and careful optimization are critical at that scale. Every training run is a gradient descent marathon on a non-convex surface in trillions of dimensions.
Google OR-Tools: Production-grade constrained optimization solvers for Python. Used in logistics (delivery routing), scheduling (shift planning), and resource allocation. Lagrange relaxation is one of its core techniques.
Hugging Face Model Hub: Distributes 500,000+ pre-trained model weights. Understanding regularization helps you know when and how to fine-tune these models on your own data without overfitting — especially when your dataset is small compared to the model's capacity.
R Shiny Applications: Used in industry for interactive optimization dashboards. Teams visualize loss surfaces, tune hyperparameters interactively, and debug gradient descent behavior in real time — the production version of the teaching demos from this lecture.
Web-Scale Feature Engineering: At Google, Meta, and Amazon, feature normalization is the first step in every training pipeline. Practitioners use log transformations and z-score normalization to keep computations stable across features that span many orders of magnitude (e.g., click counts from 0 to billions).
Open-Source Model Weights: Meta's LLaMA, Mistral, Falcon, and other open-weight models let the community run inference and fine-tune. The weights are public; the training recipe (optimization pipeline, data mixture, RLHF procedure) is proprietary. This distinction — weights as product vs. training as trade secret — shapes the entire open-source AI ecosystem.
Image Processing Pipelines: Convolutional neural networks (CNNs) default to min-max scaling because pixel values are naturally bounded (0-255 becomes 0-1). Text processing uses dimensionality-specific normalization: TF-IDF for bag-of-words, layer normalization for transformer embeddings.
AI Knowledge Tools: NotebookLM and similar tools ingest research papers and lecture materials to produce structured knowledge maps — applying the same information extraction and vector embedding principles used in building retrieval-augmented generation (RAG) systems and enterprise knowledge bases.

MFML Lecture 12 notes · Challenges of Gradient Descent and Constrained Optimization

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-11

Sections Breakdown

1Gradient Descent Recap

The update rule, the three variants (batch, mini-batch, SGD), and the role of the learning rate.

2Three Fundamental Challenges of Gradient Descent

Overfitting, local minima, and uneven curvature as the three obstacles that derail optimization.

3Bias, Variance, and the Overfitting Mechanism

Reducible vs irreducible error, the bias-variance tradeoff, and the weight explosion signature of overfitting.

4Constrained Optimization — Lagrange's Method

The gradient-parallelism condition, the Lagrangian, and two fully worked examples.

5Regularization as Constrained Optimization

Ridge (L2) and Lasso (L1) as weight constraints, the sign of lambda, and the penalized vs constrained forms.

6Feature Processing and Scaling

Why scaling matters, z-score vs min-max vs robust scaling, and their effect on loss surface geometry.

7The Hessian Matrix and Loss Surface Curvature

Second derivatives, eigenvalues as curvature diagnostics, and the effect of feature correlation.

8Practical Issues: Oscillation, Outliers, and Degrees of Freedom

Detecting oscillation, outlier sensitivity of scaling methods, and Bessel's correction.

9Tools and Practical Workflows

Python/R ecosystems, visualization for intuition, AI-assisted learning, and why weights are open-sourced.

10Exam Guidance Summary

Highest-priority problem-solving and conceptual topics, plus study strategy.

11Key Industry Applications

LLMs, OR-Tools, Hugging Face, web-scale feature engineering, and open-source weights.

Postgraduate students in Machine Learning and related quantitative fields

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Gradient Descent Update Rule

Must-know: All variants share the same update: $w_{new} = w_{old} - α \nabla L (w_{old})$ . The minus sign points downhill. Batch uses all data. Mini-batch uses a subset. SGD uses one point. The learning rate $α$ is a hyperparameter you tune before training.

$w_{new} = w_{old} - α \nabla L (w_{old})$

⚠️ Top pitfall: Treating $α$ as fixed forever. The best learning rate often changes during training; schedules and adaptive optimizers (Adam, RMSprop) adjust it. A high $α$ can make the loss bounce without settling — looking flat but not converged.

Self-check: Why does a learning rate that is too large cause divergence rather than convergence?

Connects to: Three Fundamental Challenges, Bias-Variance Tradeoff, Feature Scaling

Three Fundamental Challenges of Gradient Descent

Must-know: Gradient descent is derailed by three things. Overfitting fills the loss surface with local minima. Local minima trap you short of the global bottom when the gradient hits zero. Uneven curvature creates plateaus and narrow valleys. A convex surface has one global minimum; neural networks are usually non-convex.

$At a local min: \nabla L (w) = 0$

⚠️ Top pitfall: Assuming all loss surfaces are convex. Only linear models with squared loss are guaranteed convex. In high dimensions, most stationary points are saddle points, not local minima — and most local minima of neural nets are surprisingly good.

Self-check: Why does a zero gradient not guarantee you have reached the global minimum?

Connects to: Bias-Variance Tradeoff, The Hessian Matrix, Practical Issues: Oscillation

Bias-Variance Tradeoff

Must-know: Total error = reducible + irreducible error, and irreducible = bias + variance. Bias is simplification error from a too-simple model. Variance is sensitivity to training data from a too-complex model. Lowering one raises the other. Overfitting means low bias, high variance, and exploding weights.

$Bias + Variance = Constant$

⚠️ Top pitfall: Reading training error as model quality. Training error always falls with complexity; only test error reveals generalization. Also, you can lower both bias and variance by adding more high-quality training data.

Self-check: If your model has high variance (overfitting), which lever do you pull — more features or more training examples?

Connects to: Three Fundamental Challenges, Regularization as Constrained Optimization, Weight Explosion

Constrained Optimization — Lagrange's Method

Must-know: To optimize $F (x, y)$ subject to $G (x, y) = C$ , the gradients are parallel at the optimum: $\nabla F = λ \nabla G$ . Combine this with the constraint to solve for $x, y, λ$ . The algebraic Lagrangian form is equivalent. Both methods earn full exam credit.

$\nabla F = λ \nabla G with G (x, y) = C$

⚠️ Top pitfall: Forgetting to handle both sign cases for $λ$ and discard impossible (non-real) solutions. Lagrange's method finds candidate stationary points only — you must still check whether each is a max, min, or saddle.

Self-check: In the maximize-xy-subject-to-x+y=6 example, why must the optimum have x = y?

Connects to: Regularization as Constrained Optimization, The Hessian Matrix

Regularization as Constrained Optimization

Must-know: Regularization is a weight constraint. Ridge (L2) adds $λ \sum β_{j}^{2}$ — it shrinks weights but never to zero, keeping all features. Lasso (L1) adds $λ \sum ∣ β_{j} ∣$ — it can push weights exactly to zero. That gives automatic feature selection. The penalized and constrained forms are equivalent via Lagrange duality.

$Minimize: MSE + λ j = 1 \sum d β_{j}^{2}$

⚠️ Top pitfall: Using ridge when you need feature selection. Ridge keeps all 500 features with tiny coefficients; lasso zeros out the irrelevant ones. Also, $λ$ is chosen by cross-validation, not by gradient descent on the main loss.

Self-check: Why does Lasso produce exact zeros while Ridge only shrinks toward zero?

Connects to: Constrained Optimization, Bias-Variance Tradeoff, Feature Scaling

Feature Scaling

Must-know: Unequal feature scales create elongated, narrow-valley loss surfaces where gradient descent oscillates. Z-score $z = (x - μ) / σ$ gives every feature mean 0, variance 1 — the default for optimization. Min-max bounds to [0,1] and suits images. Robust (median/IQR) resists outliers.

$z = \frac{x - μ}{σ}$

⚠️ Top pitfall: Thinking min-max equals z-score. Min-max does not enforce unit variance, so the loss surface can stay tapered and gradient descent still oscillates. Models are biased toward high-variance features — salary can dominate age simply because its numbers are bigger.

Self-check: Why does an unscaled salary feature (200K-5M) dominate an age feature (22-65) during optimization?

Connects to: Three Fundamental Challenges, The Hessian Matrix, Practical Issues: Outliers

The Hessian Matrix

Must-know: The Hessian $H$ is the matrix of second partial derivatives — it measures curvature, not just slope. Its eigenvalues reveal shape. Roughly equal means a symmetric bowl (good). One huge and one tiny means a narrow valley (bad). One negative means a saddle. Correlated features inflate off-diagonal entries.

$H = [\frac{\partial ^{2} L}{\partial x ^{2}} & \frac{\partial ^{2} L}{\partial x \partial y} \frac{\partial ^{2} L}{\partial y \partial x} & \frac{\partial ^{2} L}{\partial y ^{2}}]$

⚠️ Top pitfall: Thinking the Hessian is only for two variables. It generalizes to $d$ parameters ( $d (d + 1) /2$ unique entries). A zero eigenvalue means a flat direction (ridge/plateau); a negative one means a saddle, not a minimum. The optimal learning rate is about $1/ λ_{m a x}$ .

Self-check: For two perfectly correlated features, why does the Hessian have a zero eigenvalue?

Connects to: Constrained Optimization, Feature Scaling, Three Fundamental Challenges

Practical Issues: Oscillation and Bessel's Correction

Must-know: Oscillation looks like convergence from a single run. Diagnose it by trying different learning rates, starts, and optimizers. Outlier sensitivity: min-max worst, z-score moderate, robust (IQR) best. The sample standard deviation divides by $n - 1$ (Bessel's correction) because estimating the mean consumes one degree of freedom.

$s = \frac{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2}}{n - 1}$

⚠️ Top pitfall: Using $n$ instead of $n - 1$ for small samples — with $n = 5$ this changes the variance estimate by 25%. Bessel's correction makes $s^{2}$ unbiased, but $s$ itself stays slightly biased.

Self-check: Why does dividing by $n - 1$ rather than $n$ correct for estimating the mean from the same data?

Connects to: Three Fundamental Challenges, Feature Scaling, Gradient Descent Recap

Previous LectureLecture 11 · Gradient Descent

Next LectureLecture 13 · Gradient Descent Optimization

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

12.1 Gradient Descent Recap

12.1.1 Review of Gradient Descent Variants

12.1.2 The Learning Rate Parameter

12.1.3 Symbol Registry

12.2 Three Fundamental Challenges of Gradient Descent

12.2.1 Overview

12.2.2 Challenge 1: Overfitting

12.2.3 Challenge 2: Local Minima

12.2.4 Challenge 3: Uneven Curvature

12.3 Bias, Variance, and the Overfitting Mechanism

12.3.1 Reducible and Irreducible Error

12.3.2 Bias-Variance Tradeoff

12.3.3 How Model Complexity Creates Local Minima

12.3.4 The Weight Explosion Problem

12.4 Constrained Optimization — Lagrange's Method

12.4.1 The Core Idea: Optimization with a Budget

12.4.2 Geometric Interpretation

12.4.3 The Gradient Parallelism Condition

12.4.4 Symbol Registry — Lagrange's Method

12.4.5 Lagrange's Algebraic Form

12.4.6 Worked Example 1: Maximize XY Subject to X+Y=6

12.4.7 Worked Example 2: Minimize X2+Y2 Subject to XY=1

12.4.8 Student Q&A on Lagrange's Method

12.5 Regularization as Constrained Optimization

12.5.1 Symbol Registry — Regularization

12.5.2 L2 Regularization (Ridge)

12.5.3 L1 Regularization (Lasso)

12.5.4 The Sign of Lambda

12.5.5 Student Q&A on Regularization

12.6 Feature Processing and Scaling

12.6.1 Symbol Registry — Scaling

12.6.2 Why Scaling Matters for Optimization

12.6.3 Scaling Methods Overview

12.6.4 Z-Score Standardization

12.6.5 Min-Max Scaling

12.6.6 Other Scaling Methods

12.6.7 Scaling and the Loss Surface Geometry

12.6.8 Student Q&A on Scaling

12.7 The Hessian Matrix and Loss Surface Curvature

12.7.1 Symbol Registry — Hessian Matrix

12.7.2 Definition and Role

12.7.3 Eigenvalues and Correlation Effects

12.7.4 Student Q&A on Correlation and Loss Surfaces

12.8 Practical Issues: Oscillation, Outliers, and Degrees of Freedom

12.8.1 Symbol Registry — Statistics

12.8.2 Gradient Descent Oscillation

12.8.3 Outliers and Their Effect on Scaling

12.8.4 Bessel's Correction and Degrees of Freedom

12.8.5 Student Q&A on Oscillation and Detection

12.9 Tools and Practical Workflows

12.9.1 Data Science Languages

12.9.2 Visualization for Understanding Optimization

12.9.3 Learning with AI-Assisted Tools

12.9.4 Why Some Model Weights Are Open-Sourced

Exam Guidance Summary

Highest-priority topics (problem-solving questions)

Conceptual topics (short-answer / multiple-choice)

Upcoming topics

Study strategy

Key Industry Applications

MFML Lecture 12 notes · Challenges of Gradient Descent and Constrained Optimization

Sections Breakdown

Exam Revision Notes

Gradient Descent Update Rule

Three Fundamental Challenges of Gradient Descent

Bias-Variance Tradeoff

Constrained Optimization — Lagrange's Method

Regularization as Constrained Optimization

Feature Scaling

The Hessian Matrix

Practical Issues: Oscillation and Bessel's Correction

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat

12.4.6 Worked Example 1: Maximize $X Y$ Subject to $X + Y = 6$

12.4.7 Worked Example 2: Minimize $X^{2} + Y^{2}$ Subject to $X Y = 1$