Mathematical Foundations for Machine Learning

Taylor Series and Hessian-Based Optimization

Published: 2026-07-10

Level: postgraduate

Audience: Postgraduate students in Machine Learning

Taylor Series and Hessian-Based Optimization

10.1 Rolle's Theorem

10.1.1 Definition and Intuition

The "Flat Spot" Guarantee. You start a hike at 1000m elevation and end at 1000m. Your trail can go up, then down. It can dip into a valley, then climb back. But one thing is certain: at some point along the way, you stood on perfectly flat ground. Your slope was zero. Rolle's theorem is the mathematical statement of that guarantee.

There are two fundamental theorems that build up Taylor series. The first is Rolle's theorem. It is the starting point.

The Thrown-Ball Analogy. Throw a ball straight up. It rises, reaches a highest point

And falls back. At that highest point, for one instant, the ball's rate of change is zero. It stopped going up. It has not yet started coming down. The slope at that instant is exactly zero. The function's path has a guaranteed "flat spot" somewhere between the start and the end.

Where the analogy breaks: Rolle's theorem only guarantees at least one flat spot —

The ball might have bounced, creating multiple. Also, the theorem covers minima too (a valley), not just maxima (the peak).

Rolle's theorem captures this idea for continuous functions. It has three conditions:

The function $f$ is continuous on the closed interval $[a, b]$ . No breaks, no gaps, no jumps anywhere between $a$ and $b$ , endpoints included.
The function $f$ is differentiable on the open interval $(a, b)$ . The curve must be smooth — no sharp corners, no sudden kinks — at every point between $a$ and $b$ .
The function values at the endpoints are equal: $f (a) = f (b)$ .

Rolle's Theorem (Formal Statement). Let $f : [a, b] \to R$ be a function where:

$f$ is continuous on the closed interval $[a, b]$
$f$ is differentiable on the open interval $(a, b)$
$f (a) = f (b)$

Then there exists at least one point $c \in (a, b)$ such that:

f^{'} (c) = 0

The derivative at $c$ is zero. At that point, the tangent line is horizontal —

You have either a local maximum or a local minimum.

10.1.2 Why the Conditions Are Chosen That Way

Continuity on the closed interval $[a, b]$ —

The function must not break anywhere from start to finish. If there is a gap, you could jump past the critical point and never hit zero slope.

Differentiability on the open interval $(a, b)$ —

Differentiability means the curve is smooth. No sharp edges. But the condition uses the open interval, not the closed one. The endpoints $a$ and $b$ are excluded.

The reason is practical. You do not know whether the function has a sharp corner right at the starting point. Imagine a curve that is smooth between $a$ and $b$

But at exactly $x = a$ it has a kink. The derivative at that edge is not well defined. To stay safe, you only require smoothness strictly between the endpoints.

Equal endpoint values —

Starting and ending at the same height guarantees that, during the journey, you either went up and came back down (a maximum) or went down and came back up (a minimum). Either way, the slope had to hit zero somewhere.

10.1.3 Worked Example

Verify Rolle's theorem for $f (x) = x^{2} - 4 x + 3$ on $[1, 3]$ .

Step 1 — Check conditions:

$f$ is a polynomial, so it is continuous everywhere. True on $[1, 3]$ .
$f$ is a polynomial, so it is differentiable everywhere. True on $(1, 3)$ .
$f (1) = 1^{2} - 4 (1) + 3 = 1 - 4 + 3 = 0$
$f (3) = 3^{2} - 4 (3) + 3 = 9 - 12 + 3 = 0$
$f (1) = f (3) = 0$ . All three conditions hold.

Step 2 — Find $c$ :

f^{'} (x) = 2 x - 4

Set $f^{'} (c) = 0$ :

2 c - 4 = 0 \Rightarrow c = 2

Step 3 — Verify $c$ is in $(1, 3)$ : $c = 2$ lies between 1 and 3.

Conclusion: Rolle's theorem is verified. The flat spot occurs at $x = 2$

Which is the vertex of the parabola —

A global minimum with value $f (2) = - 1$ .

Sense-check: The parabola opens upward, starts at $y = 0$ , dips to $y = - 1$

And returns to $y = 0$ . The lowest point must have a horizontal tangent. ✓

10.1.4 Assumptions & Scope

Scope: When Rolle's theorem applies and when it fails.

Breaks on a gap: If $f$ has a jump discontinuity on $[a, b]$ , the theorem fails. A function can skip past a flat spot by teleporting through a gap.
Breaks on a corner: If $f$ has a sharp kink (e.g., $∣ x ∣$ at $x = 0$ ), the derivative does not exist at the corner. The curve changes direction instantly, but there is no point where the tangent is truly horizontal — the derivative simply does not exist at the kink.
Breaks on a constant function: A constant function like $f (x) = 5$ on any interval satisfies Rolle's theorem — but every point has $f^{'} (c) = 0$ . The theorem says "at least one," which still holds.
Breaks without equal endpoints: If $f (a) \neq = f (b)$ , the theorem offers no guarantee at all. You could go from height 0 to height 10 without ever having zero slope.

10.1.5 Visual Intuition

Picture a smooth curve drawn on graph paper. The x-axis runs from $a$ to $b$ . The y-axis shows $f (x)$ . The curve touches the same horizontal line at $x = a$ and $x = b$ . Now imagine sliding a horizontal ruler up and down until it barely kisses the curve from above or below. That point of contact is your $c$ —

Where the tangent is horizontal. If the curve goes above its starting level and comes back down, the ruler kisses the top (maximum). If it dips below and rises back, the ruler kisses the bottom (minimum). The one-sentence takeaway: equal heights at the ends force at least one horizontal tangent in between.

10.1.6 Pitfalls

Forgetting the endpoints equality check. Students often jump straight to solving $f^{'} (c) = 0$ without verifying $f (a) = f (b)$ . If the endpoints are unequal, no $c$ is guaranteed — and you might waste time searching for one that does not exist.
Confusing "continuous on $[a, b]$ " with "continuous on $(a, b)$ ". Rolle's requires continuity at the endpoints too. A removable hole at $x = a$ breaks the theorem even if the rest of the curve is perfect.
Thinking Rolle's guarantees exactly one $c$ . The theorem only guarantees at least one. A wavy curve like $sin x$ on $[0, 2 π]$ has $f (0) = f (2 π) = 0$ and produces three points where $f^{'} (c) = 0$ .
Assuming differentiability at the endpoints. The theorem explicitly avoids this by using the open interval $(a, b)$ . Do not try to compute $f^{'} (a)$ or $f^{'} (b)$ — they might not exist.

10.1.7 Student Questions and Answers

Q: Why do we require differentiability on an open interval but continuity on a closed interval? Could the starting point be a sharp edge?

A: Yes. At the starting point you do not know if the gradient changes abruptly. The curve might be smooth throughout $(a, b)$ but have a corner exactly at $a$ . Since you cannot guarantee the derivative exists at the edge, you only enforce differentiability inside.

10.1.8 Recap + Bridge

Rolle's theorem guarantees a horizontal tangent somewhere between equal-height endpoints. It is the building block —

The first two conditions of Rolle's (continuity on $[a, b]$ and differentiability on $(a, b)$ ) carry over exactly to the Mean Value Theorem. Only the third condition (equal endpoints) gets dropped in the next step.

Rolle's theorem is the seed from which the Taylor series grows. The full proof of Taylor's theorem cleverly constructs a special function that meets Rolle's three conditions, guaranteeing the existence of a point that defines the remainder term. This is why we start here.

Real-world connection: Civil engineers use the logic of Rolle's theorem when analyzing beam deflections. A beam supported at both ends with zero deflection at the supports must have at least one point of zero slope along its curve —

Which tells them where the beam bends the most. Any physical system that returns to its starting state (a pendulum swinging through one full period, a piston completing a cycle) must pass through at least one instant of zero rate of change.

---

10.2 Mean Value Theorem

10.2.1 Definition and Intuition

How can your instantaneous speed ever equal your average speed? You drive 200 km in 2 hours. Average speed: 100 km/h. You probably spent some time at 80 km/h, some at 120 km/h. But the MVT guarantees: at least once during the trip, your speedometer read exactly 100 km/h. The instantaneous matched the average.

The mean value theorem (MVT) inherits the first two conditions of Rolle's theorem and drops the third. It no longer requires $f (a) = f (b)$ . You can start at one height and end at another.

The tilted Rolle's. Rolle's theorem requires starting and ending at the same height —

Like a hiker returning to the same elevation. MVT is the tilted version. Now the trail starts at one elevation and ends at a different one. The guarantee shifts: instead of "somewhere the slope is zero," it becomes "somewhere the slope exactly matches the average slope of the whole trail." MVT tilts the horizontal tangent of Rolle's into a tangent parallel to the secant line.

The professor's highway driving analogy: you drive 200 km in 2 hours. Average = 100 km/h. The MVT says your speedometer must have hit exactly 100 km/h at some moment. Where the analogy breaks: the MVT assumes a smooth speed curve (no instantaneous jumps)

Which real driving has (braking, traffic lights)

But the core insight stands.

MVT says: if $f$ is continuous on $[a, b]$ and differentiable on $(a, b)$ , then there exists at least one point $c$ in $(a, b)$ where:

Mean Value Theorem (Formal Statement). Let $f : [a, b] \to R$ be:

continuous on the closed interval $[a, b]$
differentiable on the open interval $(a, b)$

Then there exists at least one point $c \in (a, b)$ such that:

f^{'} (c) = \frac{f ( b ) - f ( a )}{b - a}

The left side is the instantaneous rate of change at $c$ —

The slope of the tangent line at that one point. The right side is the average rate of change across the whole interval —

The slope of the secant line from $(a, f (a))$ to $(b, f (b))$ . MVT says these two slopes are equal at some interior point.

10.2.2 Comparison with Rolle's Theorem

Rolle's theorem is a special case of MVT. When $f (a) = f (b)$ , the right side becomes $\frac{0}{b - a} = 0$ , and MVT reduces to Rolle's: $f^{'} (c) = 0$ .

Aspect	Rolle's Theorem	Mean Value Theorem
Endpoint condition	$f (a) = f (b)$ required	Any $f (a), f (b)$ allowed
Slope guaranteed	Zero (horizontal tangent)	Average slope (tangent ∥ secant)
Geometrically	Flat spot	Tangent parallel to secant line
Role in the story	Seed theorem — the starting point	Bridge to Taylor series
When you'd pick it	Need to prove existence of a critical point when endpoints match	Need to relate instantaneous change to overall average change

10.2.3 Geometric Interpretation

Draw a smooth curve from $(a, f (a))$ to $(b, f (b))$ . The secant line connects the two endpoints —

That is the average rate of change. Now slide a tangent line along the curve. At some point $c$ , the tangent becomes exactly parallel to the secant. That point is where $f^{'} (c)$ equals the average rate.

Real-world: functions like exponentials grow smoothly and always satisfy this. Pick any two points on $e^{x}$ . Somewhere between them, the instantaneous growth rate equals the average growth rate.

10.2.4 Worked Example

Find a point $c$ that satisfies the MVT for $f (x) = x^{3}$ on $[1, 3]$ .

Step 1 — Check conditions:

$f (x) = x^{3}$ is a polynomial: continuous everywhere on $[1, 3]$ (including endpoints). ✓
Differentiable everywhere on $(1, 3)$ . ✓

Step 2 — Compute the average slope:

\frac{f ( 3 ) - f ( 1 )}{3 - 1} = \frac{27 - 1}{2} = \frac{26}{2} = 13

Step 3 — Find $c$ where $f^{'} (c) = 13$ :

f^{'} (x) = 3 x^{2}, 3 c^{2} = 13, c^{2} = \frac{13}{3}, c = \frac{13}{3} \approx 2.08

Step 4 — Verify $c$ is in $(1, 3)$ : $c \approx 2.08$ lies between 1 and 3. ✓

Conclusion: At $x \approx 2.08$ , the instantaneous slope $f^{'} (c) = 3 (2.08)^{2} = 13$ equals the average slope over $[1, 3]$ . The point is $(2.08, 9.0)$ .

Sense-check: The curve $x^{3}$ steepens as $x$ grows. Near $x = 1$ , the slope is $3 (1^{2}) = 3$ (below average). Near $x = 3$ , the slope is $3 (9) = 27$ (above average). Somewhere in between, it must hit exactly 13. ✓

10.2.5 From MVT to Taylor Series — The Bridge

Rearrange the MVT equation:

f (b) = f (a) + f^{'} (c) (b - a)

You know the value at $a$ . You want the value at $b$ . MVT says: start with $f (a)$ , then add the rate of change at some intermediate point $c$ , scaled by how far $b$ is from $a$ .

Now look at the first two terms of the Taylor series:

f (b) \approx f (a) + f^{'} (a) (b - a)

The only difference: Taylor uses the derivative at $a$ itself, not at some unknown $c$ in between. Taylor tweaks MVT by anchoring the slope at the starting point rather than searching for the magical point $c$ .

If $b$ is close to $a$ , this single-slope approximation works well. The slope at $a$ is a good proxy for the average slope of the short interval. But when $b$ is far from $a$ , the slope at $a$ no longer captures all the bending and turning that happens in between. You need more terms.

10.2.6 Assumptions & Scope

Scope: When MVT applies and when it fails.

Both conditions are required. Losing continuity on $[a, b]$ or differentiability on $(a, b)$ kills the guarantee. The function $f (x) = ∣ x ∣$ on $[- 1, 2]$ fails differentiability at $x = 0$ and MVT does not apply there.
The point $c$ is not the midpoint. Don't assume $c = \frac{a + b}{2}$ . It is wherever the tangent happens to match the secant — which could be near $a$ , near $b$ , or anywhere in between. The symmetic function $f (x) = x^{2}$ on $[- 2, 2]$ has $c = 0$ (the midpoint), but $f (x) = x^{3}$ on $[1, 3]$ puts $c \approx 2.08$ , closer to $b$ .
MVT is an existence theorem, not a construction method. It tells you a $c$ exists but does not tell you how to find it directly. You still have to solve $f^{'} (c) = \frac{f ( b ) - f ( a )}{b - a}$ .
Works for any real-valued function on a real interval. The theorem does not generalize cleanly to vector-valued functions — the vector MVT requires inequalities rather than equalities.

10.2.7 Visual Intuition

Plot your function on $[a, b]$ . Draw the secant line connecting $(a, f (a))$ to $(b, f (b))$ —

That is your reference slope. Now imagine a ruler that starts with the same slope as the secant and slides along the curve, always staying parallel to the secant. MVT guarantees there is at least one point where the ruler kisses the curve as a tangent line. That point of contact is $c$ . For $f (x) = x^{3}$ on $[1, 3]$ , the secant rises steeply from $(1, 1)$ to $(3, 27)$ . The tangent at $c \approx 2.08$ runs parallel to it —

Touching the curve at a single point while matching the secant's slope. One-sentence takeaway: somewhere between $a$ and $b$ , the instantaneous slope matches the overall average slope.

10.2.8 Pitfalls

Thinking $c$ is always the midpoint. It is not. The location depends on how the curve bends. For highly asymmetric functions, $c$ can be arbitrarily close to one endpoint.
Applying MVT to vector-valued functions directly. The standard MVT applies to functions $f : R \to R$ . For curves in $R^{n}$ , you need a different form involving norms and inequalities.
Forgetting to check continuity at the endpoints. MVT requires continuity on the closed interval. A removable discontinuity at $x = b$ — even a single missing point — breaks the guarantee.
Confusing MVT with the Intermediate Value Theorem. IVT guarantees a function takes every value between $f (a)$ and $f (b)$ . MVT guarantees a derivative value equals the average slope. These are different theorems with different conclusions.
**Assuming the tangent is unique.** MVT guarantees at least one $c$ . A function that oscillates may have many parallel tangents.

10.2.9 Recap + Bridge

MVT guarantees that the instantaneous slope equals the average slope somewhere inside any smooth interval. Rearranging MVT gives $f (b) = f (a) + f^{'} (c) (b - a)$ —

A formula that directly prefigures the Taylor series. The Taylor series simply swaps the unknown midpoint slope $f^{'} (c)$ for the known starting-point slope $f^{'} (a)$ , then adds higher-order corrections for curvature.

The connection from MVT to Taylor is the conceptual bridge of this entire lecture. MVT says "there exists a $c$ giving the exact correction." Taylor says "instead of hunting for $c$ , I'll use $a$ as a proxy and add more terms to fix the error." This is the shift from an existence theorem (guaranteeing some point works) to a constructive approximation (building a polynomial that gets arbitrarily close).

Real-world connection: The Mean Value Theorem underpins error bounds in numerical methods. When a physics simulation approximates a continuous trajectory with discrete time steps, MVT provides the theoretical guarantee that the real velocity at some moment matches the computed average. Speed cameras on highways rely on the same logic: measure your time between two points, compute average speed

And —

By MVT —

You must have hit that speed at some instant, making the ticket mathematically sound.

---

10.3 Taylor Series — The Full Intuition

10.3.1 Position, Slope, Curvature, and Beyond

How can you clone a function using only information from a single point? Imagine you are a master forger trying to replicate a complex signature. You cannot see the whole signature at once. But you can analyze it deeply at its starting point —

The exact position, the angle of the pen, how the curve is bending, how the bend itself is changing. The Taylor series is what lets you build a polynomial imitation that gets closer and closer to the real thing with every extra detail you match.

A Taylor series approximation builds up in layers. Each new term adds a geometric feature to the polynomial clone:

Degree 0 — Position: $f (a)$ . Where the function sits. The forger places a dot at the starting point.
Degree 1 — Slope: $f^{'} (a) (x - a)$ . How fast the function is changing at $a$ , times the step you take. The forger draws a straight line matching the starting angle.
Degree 2 — Curvature: $\frac{f ^{''} ( a )}{2 !} (x - a)^{2}$ . How the slope itself is bending. The forger bends the line into a parabola matching the initial curve.
Degree 3+ — Higher-order curvatures: finer and finer bends. The forger matches how the curvature changes, then how that change changes, and so on.

Each added term gives the approximation a new way to bend. First you place the point. Then you tilt a straight line. Then you bend that line into a parabola. Then you add more nuanced curves on top. The Taylor series is the ultimate forger —

Using all the derivative information at a single point to build a perfect polynomial clone in that neighborhood.

Where the master forger analogy breaks. A real forger eventually runs out of steady-hand precision. The Taylor series can keep adding terms forever (the infinite Taylor series) to get an arbitrarily close match, as long as the function is infinitely differentiable. Also, the clone is only perfect near the starting point —

Walk too far away and the approximation diverges.

10.3.2 The Full 1D Formula

Taylor Series (1D). For a function $f$ that is $n$ times differentiable at a point $a$ , the Taylor polynomial of degree $n$ about $a$ is:

f (x) = k = 0 \sum n \frac{f ^{(k)} ( a )}{k !} (x - a)^{k}

Expanded:

f (x) = f (a) + f^{'} (a) (x - a) + \frac{f ^{''} ( a )}{2 !} (x - a)^{2} + \frac{f ^{'''} ( a )}{3 !} (x - a)^{3} + \dots + \frac{f ^{(n)} ( a )}{n !} (x - a)^{n}

Each piece:

$f^{(k)} (a)$ : the $k$ -th derivative evaluated at $a$ . The 0-th derivative is the function itself.
$k!$ : $k$ factorial. $0! = 1$ , $1! = 1$ , $2! = 2$ , $3! = 6$ , $4! = 24$ , and so on.
$(x - a)^{k}$ : the step from $a$ , raised to the $k$ -th power. The farther you step, the larger this term.
$\frac{f ^{(k)} ( a )}{k !}$ : the coefficient for the $k$ -th degree term — the pure $k$ -th derivative value, corrected for the factorial accumulation.

When $a = 0$ , the series is called a Maclaurin series —

A special name for a Taylor series centered at zero. The formula simplifies because $x - 0 = x$ for all terms.

10.3.3 Where the Factorials Come From

Taylor series approximates a non-polynomial function (like $lo g x$ , $e^{x}$ , $sin x$ ) with a polynomial. A non-polynomial function has a Taylor expansion as a polynomial series.

Set up a generic polynomial:

P (x) = c_{0} + c_{1} x + c_{2} x^{2} + c_{3} x^{3} + \dots

You want the coefficients $c_{k}$ so that $P (x)$ matches $f (x)$ and all its derivatives at $x = 0$ .

Evaluate at $x = 0$ :

$P (0) = c_{0}$ , so $c_{0} = f (0)$ .
$P^{'} (x) = c_{1} + 2 c_{2} x + 3 c_{3} x^{2} + \dots$ . At $x = 0$ : $P^{'} (0) = c_{1}$ , so $c_{1} = f^{'} (0)$ .
$P^{''} (x) = 2 c_{2} + 6 c_{3} x + \dots$ . At $x = 0$ : $P^{''} (0) = 2 c_{2}$ , so $c_{2} = f^{''} (0) /2 = f^{''} (0) /2!$ .
$P^{'''} (x) = 6 c_{3} + \dots$ . At $x = 0$ : $P^{'''} (0) = 6 c_{3}$ , so $c_{3} = f^{'''} (0) /6 = f^{'''} (0) /3!$ .

The factorial appears because repeated differentiation pulls down the exponent as a multiplicative factor. Each time you differentiate $x^{k}$ , you multiply by $k$ , then $k - 1$ , then $k - 2$

And so on. The accumulated product is exactly $k!$ . Dividing by $k!$ cancels that buildup, leaving the pure derivative value as the coefficient.

10.3.4 Why Higher-Order Terms Matter

The first derivative tells you how the function changes right at $a$ . That is a local property. If $b$ is close to $a$ , the local slope is enough —

The function has not had room to bend much.

If $b$ is far from $a$ , the local slope at $a$ does not know about all the curving that happened along the way. The second derivative captures how the slope itself changes —

It adds curvature. The third derivative captures how the curvature changes. Each higher order gives the approximation more room to follow the real function's bends.

10.3.5 Worked Examples from the Companion Document

Example 1: Taylor series of $e^{x}$ about $a = 0$ .

The function $f (x) = e^{x}$ is special because its derivative is always itself. All derivatives at $a = 0$ equal 1.

f (x) f^{'} (x) f^{''} (x) f^{'''} (x) = e^{x} = e^{x} = e^{x} = e^{x} f (0) f^{'} (0) f^{''} (0) f^{'''} (0) = 1 = 1 = 1 = 1

Plug into the Taylor formula:

e^{x} \approx \frac{f ( 0 )}{0 !} x^{0} + \frac{f ^{'} ( 0 )}{1 !} x^{1} + \frac{f ^{''} ( 0 )}{2 !} x^{2} + \frac{f ^{'''} ( 0 )}{3 !} x^{3} + \dots = \frac{1}{1} + \frac{1}{1} x + \frac{1}{2} x^{2} + \frac{1}{6} x^{3} + \dots = 1 + x + \frac{x ^{2}}{2} + \frac{x ^{3}}{6} + \dots

Sense-check: At $x = 0$ , the approximation gives $1$ , which matches $e^{0} = 1$ . At $x = 0.1$ , the true value $e^{0.1} \approx 1.10517$ . The degree-3 approximation gives $1 + 0.1 + 0.005 + 0.000167 = 1.105167$ — excellent agreement.

Example 2: Taylor series of $cos (x)$ about $a = 0$ .

Derivatives of cosine cycle every four steps and alternate between 0 and ±1:

f (x) f^{'} (x) f^{''} (x) f^{'''} (x) f^{(4)} (x) = cos x = - sin x = - cos x = sin x = cos x f (0) f^{'} (0) f^{''} (0) f^{'''} (0) f^{(4)} (0) = 1 = 0 = - 1 = 0 = 1

Only the even-order terms survive (odd-order coefficients are zero):

cos x = 1 - \frac{x ^{2}}{2 !} + \frac{x ^{4}}{4 !} - \frac{x ^{6}}{6 !} + \dots = 1 - \frac{x ^{2}}{2} + \frac{x ^{4}}{24} - \frac{x ^{6}}{720} + \dots

Sense-check: For small angles, $cos x \approx 1 - x^{2} /2$ is the standard small-angle approximation used in physics and engineering. At $x = 0.1$ rad, the true value is $cos (0.1) \approx 0.995004$ . The approximation $1 - 0.01/2 = 0.995000$ —

Accurate to five decimal places from just two terms.

10.3.6 Assumptions & Scope

Scope: What Taylor series needs to work.

Infinite differentiability for the infinite series. For the finite Taylor polynomial (degree $n$ ), you need $f$ to be $n$ times differentiable at $a$ . For the infinite series, $f$ must be infinitely differentiable — and even that is not enough. Some infinitely differentiable functions have a Taylor series that converges to the wrong value.
Convergence radius. Even when the Taylor series converges, it only matches the function within a certain radius around $a$ . For $1/ (1 - x)$ about $a = 0$ , the series $1 + x + x^{2} + \dots$ only works for $∣ x ∣ < 1$ , even though the function is smooth at $x = 2$ .
The remainder is real. A degree- $n$ Taylor polynomial has an error term — the remainder $R_{n} (x)$ . This remainder is what the full Taylor's theorem (the one with Rolle's theorem in its proof) quantifies. For practical work, you either ignore the remainder (staying close to $a$ ) or bound it using the Lagrange or integral form of the remainder.
Not all functions equal their Taylor series. The classic counterexample is $f (x) = e^{- 1/ x^{2}}$ for $x \neq = 0$ with $f (0) = 0$ . All derivatives at 0 are zero, so the Taylor series is identically zero — but the function is positive for $x \neq = 0$ . The series converges to the wrong value.

10.3.7 Visual Intuition

Plot $y = e^{x}$ on the x-axis from $- 2$ to $2$ . At $x = 0$ , plot the degree-0 approximation: a horizontal line $y = 1$ . Then overlay the degree-1 approximation: the tangent line $y = 1 + x$ . Then degree 2: $y = 1 + x + x^{2} /2$ —

A parabola that hugs the exponential upward. Then degree 3: $y = 1 + x + x^{2} /2 + x^{3} /6$

Which bends upward more aggressively for positive $x$ . As you increase the degree, the polynomial curves more and more like the exponential, staying close to it over a wider band around $x = 0$ . Landmarks: all approximations pass through $(0, 1)$ —

That is the position match. The tangent line (degree 1) touches with the right slope. The parabola (degree 2) has the right bend. One-sentence takeaway: each added term lets the polynomial curve in one more way, buying you accuracy over a larger neighborhood.

10.3.8 Pitfalls

Forgetting the factorial. Writing $\frac{f ^{''} ( a )}{2}$ instead of $\frac{f ^{''} ( a )}{2 !}$ is a common slip. The denominator is $k!$ , not $k$ . For $k = 3$ , the coefficient uses $3! = 6$ , not 3.
Mixing up the center. If the expansion is around $a = 1$ , all terms use $(x - 1)$ , $(x - 1)^{2}$ , $(x - 1)^{3}$ , not $x$ , $x^{2}$ , $x^{3}$ . Every $(x - a)^{k}$ must have the right $a$ .
Thinking more terms always means better approximation. More terms give better accuracy near $a$ , but far from $a$ , a high-degree polynomial can oscillate wildly (Runge's phenomenon). The Taylor series is a local tool, not a global one.
Assuming the Taylor series converges to the function. As noted in scope, some functions have Taylor series that converge to something else entirely. Always know your function's convergence radius.
**Confusing the Taylor polynomial with the Taylor series.** The polynomial truncates at degree $n$ . The series is the infinite sum. In practice you always truncate, so the remainder matters.

10.3.9 Student Questions and Answers

Q: I understand the 2nd derivative adds curvature. What is the intuition for the 3rd and higher derivatives?

A: In one variable, it is hard to visualize beyond the 2nd. The 1st derivative is the rate of change. The 2nd is the rate of change of the rate of change —

How the slope bends. The 3rd is how the curvature itself changes. Beyond that, you are adding finer and finer corrections.

Q: Is this like distance, velocity, acceleration, jerk?

A: Yes

That is a good parallel. Position is the 0th derivative. Velocity is the 1st. Acceleration is the 2nd. Jerk is the 3rd. Each tells a different story about the motion.

Q: In multiple dimensions, do higher-order terms make more sense?

A: Yes. In a multivariable function like a 3D surface, the function changes in the $x$ -direction, the $y$ -direction

And along the $x y$ -plane. The 3rd-order terms capture mixed rates of change —

How the curvature in $x$ changes as you move in $y$

And so on. The cross-derivative terms multiply once you go multivariable.

10.3.10 Pedagogical Insight

When you fit a Taylor polynomial, think of adding one geometric feature at a time. The 0th term places a horizontal line at the right height. The 1st term tilts it. The 2nd term bends it into a parabola. The 3rd term adds an S-shape. Each term refines the shape. This is the geometric meaning of Taylor series.

10.3.11 Recap + Bridge

The Taylor series clones a function at a point by matching its position, slope, curvature

And all higher-order bends using only derivative information from that single point. Each added derivative buys one more degree of shape-matching freedom. The factorial denominators undo the chain-rule accumulation of repeated differentiation.

The Taylor series is the central theorem that motivates everything else in this lecture. Rolle's and MVT built the logical foundation. Taylor provides the constructive tool. The next section extends this tool from one variable to two —

Opening the door to the gradient and the Hessian.

Real-world connection: Taylor series approximate solutions to differential equations that have no closed form. Weather models, orbital mechanics simulations

And electronic circuit analysis all use truncated Taylor expansions to turn impossible nonlinear equations into solvable polynomial approximations. In quantitative finance, the Black-Scholes option pricing formula is often approximated by a second-order Taylor expansion (delta-gamma approximation) for real-time risk calculations. Any time you see code that computes $sin x$ on a computer, it is almost certainly using a truncated Taylor (or Chebyshev) polynomial —

A degree-5 or degree-7 approximation that delivers double-precision accuracy for all inputs.

---

10.4 Two-Variable Taylor Series and the Gradient

10.4.1 The Hilly Landscape

What if your function is not a curve

But a whole surface? So far we approximated single-variable functions —

A road winding through hills. Now we step onto a full 2D landscape where elevation $z = f (x, y)$ depends on two coordinates. How do you approximate the height around any point? The answer is the same idea —

Match the position, the slope

And the curvature —

But now "slope" means two numbers (east-west and north-south) and "curvature" means three.

A function $f (x, y)$ defines a surface over the $x y$ -plane. The Taylor expansion in two variables builds a polynomial surface —

A tangent plane at degree 1, a paraboloid at degree 2 —

That hugs the real surface near the anchor point.

10.4.2 Symbol Registry

$f (x, y)$ — the function being expanded — real-valued function on $R^{2}$
$(a, b)$ — the anchor point where you know the function's value — constants in $R$
$h = x - a$ — step size in the $x$ -direction — scalar
$k = y - b$ — step size in the $y$ -direction — scalar
$f_{x} = \frac{\partial f}{\partial x}$ — partial derivative with respect to $x$ — scalar-valued function
$f_{y} = \frac{\partial f}{\partial y}$ — partial derivative with respect to $y$ — scalar-valued function
$\nabla f$ — the gradient vector: $(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y})$ — vector in $R^{2}$
$f_{xx} = \frac{\partial ^{2} f}{\partial x ^{2}}$ — second partial derivative in $x$ — scalar
$f_{y y} = \frac{\partial ^{2} f}{\partial y ^{2}}$ — second partial derivative in $y$ — scalar
$f_{x y} = \frac{\partial ^{2} f}{\partial x \partial y}$ — mixed second partial derivative — scalar

10.4.3 The First-Order Expansion

The tangent plane. When you stand on a hillside, the ground immediately under your feet is approximately flat. That flat approximation —

The tangent plane —

Is exactly the first-order Taylor expansion. You know the height at your feet. To estimate the height one step east and one step north, you add: (east slope) × (east step) + (north slope) × (north step). That is all the first-order expansion does.

For a function of two variables, you want to approximate $f (a + h, < p > B + k)$ . You already know the function's value at $(a, b)$ . You take $h$ steps in $x$ and $k$ steps in $y$ .

The first-order Taylor polynomial in two variables:

f (a + h, b + k) \approx f (a, b) + h f_{x} (a, b) + k f_{y} (a, b)

Verbally: the value at the new point equals the value at the old point, plus the slope in $x$ times the $x$ -step, plus the slope in $y$ times the $y$ -step.

10.4.4 Gradient Form

The two first-order terms can be written as a dot product:

h f_{x} + k f_{y} = \nabla f (a, b) \cdot (h, k) = [f_{x} f_{y}] [h k]

The gradient $\nabla f$ is the vector of all partial derivatives. The step vector $(h, k)$ is how far you move. Taking their dot product gives the total first-order change.

This is like saying: if you walk $h$ meters east, your elevation changes by (slope east) $\times h$ . If you walk $k$ meters north, your elevation changes by (slope north) $\times k$ . Add both contributions.

The gradient $\nabla f = (f_{x}, f_{y})$ is the vector pointing in the direction of steepest ascent. The first-order term $\nabla f \cdot (h, k)$ is the directional derivative —

It tells you how much the function changes when you step in direction $(h, k)$ . When $(h, k)$ points along the gradient, the dot product is maximal. When $(h, k)$ is perpendicular to the gradient, you walk along a contour line and the first-order change is zero.

10.4.5 The Second-Order Expansion

The degree-2 term adds curvature information:

\frac{1}{2 !} (h^{2} f_{xx} (a, b) + 2 hk f_{x y} (a, b) + k^{2} f_{y y} (a, b))

You need three pieces:

$f_{xx}$ : how the slope in $x$ changes as you move in $x$ — the pure- $x$ curvature.
$f_{y y}$ : how the slope in $y$ changes as you move in $y$ — the pure- $y$ curvature.
$f_{x y}$ : how the slope in $x$ changes as you move in $y$ — the cross-curvature, or how the two directions interact.

The factor 2 appears before the $hk f_{x y}$ term because the mixed derivative shows up twice in the expansion —

Once from $(h \partial_{x}) (k \partial_{y})$ and once from $(k \partial_{y}) (h \partial_{x})$ . Since $f_{x y} = f_{y x}$ (Clairaut's theorem), these two contributions are equal

So they combine to $2 hk f_{x y}$ .

10.4.6 The Full Two-Variable Taylor Formula (to Degree 3)

f (a + h, b + k) = f (a, b) + (h f_{x} + k f_{y}) + \frac{1}{2 !} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y}) + \frac{1}{3 !} (h^{3} f_{xxx} + 3 h^{2} k f_{xx y} + 3 h k^{2} f_{x y y} + k^{3} f_{y y y})

The third-order coefficients $(1, 3, 3, 1)$ are the binomial expansion of $(h + k)^{3}$ . This pattern repeats for all degrees: the coefficients are binomial coefficients. At degree $n$ , the coefficient of $h^{n - m} k^{m}$ is $(m n)$ .

For the degree- $n$ term, the general pattern is:

\frac{1}{n !} m = 0 \sum n (m n) h^{n - m} k^{m} \frac{\partial ^{n} f}{\partial x ^{n - m} \partial y ^{m}}

Where $(m n) = \frac{n !}{m ! ( n - m )!}$ is the binomial coefficient.

10.4.7 At a Critical Point — What the Taylor Expansion Reveals

At a critical point $(a, b)$ where the gradient is zero ( $\nabla f (a, b) = 0$ ), the first-order term vanishes. The Taylor expansion simplifies to:

f (a + h, b + k) - f (a, b) \approx \frac{1}{2 !} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y})

The left side is the change in height when you move away from the critical point. The right side is a quadratic form in $(h, k)$ . The sign of this quadratic form —

Positive, negative

Or mixed —

Determines whether the critical point is a minimum, maximum

Or saddle. This is why the Hessian matrix

Which organizes these second derivatives, is the tool for classifying extrema.

10.4.8 Assumptions & Scope

Scope: What the two-variable Taylor expansion needs.

Existence of partial derivatives. The expansion to degree $n$ requires all partial derivatives up to order $n$ to exist at $(a, b)$ . For most smooth functions ( $e^{x} sin y$ , polynomials, etc.), this holds everywhere.
Local tool. Like the 1D case, the 2D Taylor polynomial is accurate near $(a, b)$ and diverges for large steps. The error grows with the distance from the anchor point.
At a critical point, the first-order term vanishes. This is by design — you solve $\nabla f = 0$ to find candidate extrema, then use the second-order term (Hessian) to classify them. If the Hessian test is inconclusive, higher-order terms may be needed.
The number of terms grows combinatorially. At degree $d$ in $n$ variables, you have $(d n + d - 1)$ distinct partial derivatives to compute. For $n = 2, d = 3$ : $(3 4) = 4$ terms. For $n = 10, d = 3$ : $(3 12) = 220$ terms. This combinatorial explosion is why high-dimensional Taylor expansions are rarely used past degree 2.

10.4.9 Visual Intuition

Picture a 3D surface $z = f (x, y)$ . At the anchor point $(a, b)$ , the height is $f (a, b)$ —

A single dot in 3D space. The first-order expansion adds a tangent plane —

The flat surface that just kisses the real surface at $(a, b)$ , matching its slope in both the $x$ and $y$ directions. The second-order expansion bends that plane into a paraboloid —

It can curve upward (bowl), downward (dome)

Or twist into a saddle shape, depending on the signs of the Hessian entries. Landmarks: at a minimum, the tangent plane is horizontal (gradient zero) and the paraboloid opens upward. At a maximum, the plane is horizontal and the paraboloid opens downward. At a saddle, the plane is horizontal but the paraboloid curves up in one direction and down in another. One-sentence takeaway: the 2D Taylor expansion builds a polynomial surface that matches the real surface in height, tilt

And bend.

10.4.10 Pitfalls

Forgetting the factor of 2 before $hk f_{x y}$ . The middle term of the degree-2 expansion is $2 hk f_{x y}$ , not $hk f_{x y}$ . You get the factor of 2 because the mixed derivative contributes from both $(h \partial_{x}) (k \partial_{y})$ and $(k \partial_{y}) (h \partial_{x})$ .
Writing $x$ instead of $x - a$ when the anchor is not zero. If the expansion is around $(1, 2)$ , every term uses $(x - 1)$ and $(y - 2)$ . The formula template stays the same; only the numbers and variable expressions change.
Thinking the gradient is a scalar. The gradient is a vector of partial derivatives. The first-order change is the dot product of the gradient with the step — a scalar.
Confusing $f_{x y}$ with $f_{x} f_{y}$ . $f_{x y}$ is a single second-derivative — first differentiate in $x$ , then in $y$ . It is not the product of the two first derivatives.
Mixing up the factorial. At degree 2, the denominator is $2! = 2$ , not 1. At degree 3, it is $3! = 6$ , not 3.

10.4.11 Student Questions and Answers

Q: In a 1D Taylor series, you do not worry about direction. In 2D, how does the 2nd derivative tell you which direction to go?

A: The mixed term $f_{x y}$ captures how the function changes when both $x$ and $y$ move together. The gradient tells you the steepest direction of change. The Hessian tells you how that steepest direction itself curves. You combine the gradient (direction) and Hessian (curvature) to make a better step.

Q: If I have a single-variable function like $e^{x^{2}}$ , is that still a Taylor series in one variable?

A: Yes. It has only one variable, $x$ . The exponent being $x^{2}$ just makes the coefficients work out differently —

You can expand it using the chain rule. It is still a 1-variable Taylor expansion. A function becomes multivariable only when you have two or more distinct variables, like $e^{x} sin y$ or $x^{2} y$ .

Q: In the question format, how do we know whether to write $x$ or $x - a$ ?

A: If the expansion is around $(a, b)$ , every term uses $h = x - a$ and $k = y - b$ . So the first-order term becomes $(x - a) f_{x} + (y - b) f_{y}$ , the second-order terms become $(x - a)^{2} f_{xx}$

And so on. If the expansion is around $(0, 0)$ , then $h = x$ and $k = y$ directly —

No subtraction needed.

10.4.12 Gradient Descent Connection

The gradient $\nabla f$ always points in the direction where the function increases fastest. In machine learning, you want the minimum. You take the negative gradient —

The opposite direction —

And step downhill. Each step: compute the gradient, move opposite to it, repeat. This iterative process is gradient descent. It is covered fully in later topics

But the first-order Taylor term is its mathematical foundation.

10.4.13 Recap + Bridge

The two-variable Taylor series extends the 1D idea: match position $f (a, b)$ , then slope in each direction $(f_{x}, f_{y})$ , then three curvature terms $(f_{xx}, f_{x y}, f_{y y})$ , with binomial coefficients governing the higher-order terms. The gradient $\nabla f = (f_{x}, f_{y})$ points uphill

And its dot product with the step vector gives the first-order change.

The second-order term —

The quadratic form in $(h, k)$ —

Organizes the curvature information that the next two sections will formalize as the Hessian matrix. Once you have the Hessian, you can classify critical points and take informed steps in optimization.

Real-world connection: Gradient descent on a loss function is the workhorse of every neural network training loop. The first-order Taylor term is exactly what tells you which direction to update each weight. Weather models use 2D Taylor expansions to interpolate pressure and temperature readings from scattered weather stations onto a uniform grid for forecasting. In computational fluid dynamics, the Navier-Stokes equations are solved by discretizing space into a grid and using truncated Taylor expansions (finite difference methods) to approximate derivatives —

The very same idea, applied to partial differential equations over entire flow fields.

---

10.5 The Hessian Matrix

10.5.1 From Taylor Expansion to Matrix Form

The Hessian is the organizer. In the 1D world, the second derivative at a critical point is a single number: positive means minimum, negative means maximum. In 2D, you have three second derivatives ( $f_{xx}, f_{x y}, f_{y y}$ ) and you need to know the curvature in every direction —

Not just the pure $x$ and $y$ axes. The Hessian matrix packs those three numbers into a symmetric $2 \times 2$ matrix

And the quadratic form $x^{⊤} H x$ tells you the curvature along any direction $x$ .

The second-order term of the two-variable Taylor expansion is:

\frac{1}{2} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y})

This rearranges into a matrix-vector product. Define the step vector $x = [h k]$ and the matrix $H$ as:

The Hessian Matrix. For a function $f (x, y)$ with continuous second partial derivatives:

H = [f_{xx} f_{x y} f_{x y} f_{y y}]

The second-order Taylor term then equals:

\frac{1}{2} x^{⊤} H x

$H \in R^{2 \times 2}$ is the Hessian matrix — the matrix of all second partial derivatives. It is always symmetric because $f_{x y} = f_{y x}$ (Clairaut's theorem).

Why the quadratic form works. Multiply it out:

x^{⊤} H x = [h k] [f_{xx} f_{x y} f_{x y} f_{y y}] [h k] = h^{2} f_{xx} + hk f_{x y} + k h f_{x y} + k^{2} f_{y y} = h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y}

The off-diagonal $f_{x y}$ appears twice (once from $h f_{x y} k$ and once from $k f_{x y} h$ ), automatically producing the factor of 2 in the original formula. The quadratic form is the way to express the curvature of a multivariable function in any direction.

10.5.2 Symbol Registry

$H$ — the Hessian matrix — symmetric $2 \times 2$ (or $n \times n$ ) matrix of second partial derivatives
$x = (h, k)$ — the step vector — vector in $R^{2}$
$f_{xx} = \frac{\partial ^{2} f}{\partial x ^{2}}$ — second partial derivative in $x$ — scalar (rate of change of the slope in $x$ )
$f_{y y} = \frac{\partial ^{2} f}{\partial y ^{2}}$ — second partial derivative in $y$ — scalar
$f_{x y} = \frac{\partial ^{2} f}{\partial x \partial y}$ — mixed second partial derivative — scalar (how slope in $x$ changes with $y$ )
$x^{⊤} H x$ — quadratic form — scalar (curvature in the direction $x$ )
positive definite — $x^{⊤} H x > 0$ for all non-zero $x$ — signals a minimum
negative definite — $x^{⊤} H x < 0$ for all non-zero $x$ — signals a maximum

10.5.3 What the Hessian Represents

The Hessian captures how the gradient itself changes. Each entry has a geometric meaning:

$f_{xx}$ : how the slope in the $x$ -direction changes when you move along $x$ .
$f_{y y}$ : how the slope in the $y$ -direction changes when you move along $y$ .
$f_{x y}$ : how the slope in $x$ changes when you move along $y$ — the cross-interaction.

When you vary both $x$ and $y$ , the function's slope changes in all these ways. The Hessian bundles them into one matrix.

10.5.4 Connection to Definiteness

You have seen the form $x^{⊤} M x$ before —

In the context of positive definite matrices. A matrix $M$ is positive definite if $x^{⊤} M x > 0$ for every non-zero $x$ . It is negative definite if the product is always negative.

The Hessian being positive definite means the function is curving upward in every direction —

You are at a local minimum. Negative definite means the function is curving downward in every direction —

A local maximum. Mixed signs (some directions up, some down) give a saddle point.

10.5.5 Derivation: How the Hessian Connects to the Taylor Second-Order Term

Starting from the 2D Taylor degree-2 term:

\frac{1}{2} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y})

Rewrite as a matrix-vector product:

\frac{1}{2} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y}) = \frac{1}{2} [h k] [f_{xx} f_{x y} f_{x y} f_{y y}] [h k] = \frac{1}{2} x^{⊤} H x

This is the standard form. At a critical point where $\nabla f = 0$ , the full Taylor expansion simplifies to:

f (a + h, b + k) - f (a, b) \approx \frac{1}{2} x^{⊤} H x

The sign of $x^{⊤} H x$ for every possible direction $x$ determines whether $(a, b)$ is a minimum (always positive), maximum (always negative), or saddle (mixed).

10.5.6 The Hessian in $n$ Variables

For a function $f : R^{n} \to R$ , the Hessian generalizes to an $n \times n$ symmetric matrix:

H = \frac{\partial ^{2} f}{\partial x _{1}^{2}} \frac{\partial ^{2} f}{\partial x _{2} \partial x _{1}} ⋮ \frac{\partial ^{2} f}{\partial x _{n} \partial x _{1}} \frac{\partial ^{2} f}{\partial x _{1} \partial x _{2}} \frac{\partial ^{2} f}{\partial x _{2}^{2}} ⋮ \frac{\partial ^{2} f}{\partial x _{n} \partial x _{2}} \dots \dots ⋱ \dots \frac{\partial ^{2} f}{\partial x _{1} \partial x _{n}} \frac{\partial ^{2} f}{\partial x _{2} \partial x _{n}} ⋮ \frac{\partial ^{2} f}{\partial x _{n}^{2}}

$H_{ij} = \frac{\partial ^{2} f}{\partial x _{i} \partial x _{j}}$ . The Hessian has $\frac{n ( n + 1 )}{2}$ distinct entries (due to symmetry). For $n = 100$

That is 5050 entries —

Already computationally expensive. For $n = 1 0^{6}$ (a modest neural network), a dense Hessian would be $1 0^{12}$ entries

Which is why second-order optimization methods require approximations like L-BFGS rather than computing the full Hessian.

10.5.7 Alternative Notation

Some texts write the Hessian with Leibniz notation:

H = [\frac{\partial ^{2} f}{\partial x ^{2}} \frac{\partial ^{2} f}{\partial y \partial x} \frac{\partial ^{2} f}{\partial x \partial y} \frac{\partial ^{2} f}{\partial y ^{2}}]

Both notations mean the same thing. The subscript notation $f_{xx}, f_{x y}, f_{y y}$ is shorter. The Hessian is also sometimes written as $\nabla^{2} f$ —

The "Laplacian of gradient" notation —

Emphasizing it is the gradient of the gradient.

10.5.8 Hessian is Always Symmetric

For functions with continuous second derivatives, $f_{x y} = f_{y x}$ . The order of differentiation does not matter. This is Clairaut's theorem (also called Schwarz's theorem). It makes the Hessian symmetric. Symmetry is important —

It guarantees real eigenvalues, orthogonal eigenvectors

And lets you classify critical points cleanly.

10.5.9 Worked Example — Build a Hessian

Compute the Hessian of $f (x, y) = x^{3} + 3 x^{2} y + y^{2}$ at the point $(1, - 1)$ .

Step 1 — First derivatives (gradient):

f_{x} = 3 x^{2} + 6 x y, f_{y} = 3 x^{2} + 2 y

Step 2 — Second derivatives:

f_{xx} f_{y y} f_{x y} f_{y x} = \frac{\partial}{\partial x} (3 x^{2} + 6 x y) = 6 x + 6 y = \frac{\partial}{\partial y} (3 x^{2} + 2 y) = 2 = \frac{\partial}{\partial y} (3 x^{2} + 6 x y) = 6 x = \frac{\partial}{\partial x} (3 x^{2} + 2 y) = 6 x (confirms f_{x y} = f_{y x})

Step 3 — Evaluate at $(1, - 1)$ :

f_{xx} (1, - 1) = 6 (1) + 6 (- 1) = 0, f_{y y} (1, - 1) = 2, f_{x y} (1, - 1) = 6 (1) = 6

Step 4 — Hessian:

H (1, - 1) = [0662]

Sense-check: The matrix is symmetric ✓. $det (H) = 0 \cdot 2 - 6^{2} = - 36 < 0$ —

Indicating this point would be a saddle if it were a critical point. (Check: the gradient at $(1, - 1)$ is $(3 - 6, 3 - 2) = (- 3, 1) \neq = (0, 0)$

So it is not a critical point;

The Hessian still exists and describes the local curvature.)

10.5.10 Assumptions & Scope

Scope: When the Hessian is defined and what it can do.

Continuous second derivatives required. The Hessian needs $f$ to be twice differentiable. For functions with discontinuities in second derivatives (e.g., ReLU at zero, where $f^{'} (x) = 0$ for $x < 0$ and $f^{'} (x) = 1$ for $x > 0$ ), the Hessian is undefined at the kink.
Symmetric only when Clairaut's theorem holds. For pathological functions where mixed partials differ, the Hessian is not symmetric. These functions rarely appear in applied ML problems.
**The Hessian is a local descriptor.** Like the gradient, the Hessian at a point describes behavior in an infinitesimal neighborhood. A function can have positive-definite Hessian at one point and negative-definite at another.
Computational cost scales as $O (n^{2})$ . Computing the full Hessian for $n$ variables requires $O (n^{2})$ second derivative evaluations. For large $n$ , this is prohibitive. In deep learning, you use first-order methods (SGD, Adam) that never touch the Hessian.

10.5.11 Visual Intuition

At a critical point, imagine standing on a surface. The ground under you is flat (gradient zero). To figure out whether you are in a bowl, on a dome

Or on a saddle, you need to know what happens when you take a tiny step in any direction. The Hessian describes this: $x^{⊤} H x$ gives the curvature along direction $x$ . If $H = [2002]$ , then $x^{⊤} H x = 2 (h^{2} + k^{2})$ —

Always positive, a circular bowl. If $H = [20 0 - 2]$ , then $x^{⊤} H x = 2 h^{2} - 2 k^{2}$ —

Positive along the x-axis, negative along the y-axis, a saddle. The eigenvalues of $H$ capture the principal curvatures;

The eigenvectors capture the principal curvature directions. One-sentence takeaway: the Hessian converts "how does the surface bend" into a matrix whose definiteness classifies the type of critical point.

10.5.12 Pitfalls

Confusing the Hessian with the Jacobian. The Hessian is for scalar-valued functions ( $f : R^{n} \to R$ ) — it is a square matrix of second derivatives. The Jacobian is for vector-valued functions ( $f : R^{n} \to R^{m}$ ) — it is an $m \times n$ matrix of first derivatives.
Hessian entries might not be constants. In the worked example for Section 10.8, the Hessian entries were constant ( $f_{xx} = - 2$ ). But for most functions (like $e^{x} sin y$ ), the Hessian entries depend on $(x, y)$ and must be evaluated at the specific critical point.
Forgetting the Hessian exists even at non-critical points. You can compute $H$ anywhere. At a critical point, it classifies the extremum. Elsewhere, it describes local curvature for steps — useful in Newton's method.
Assuming definiteness implies global behavior. A positive-definite Hessian at a critical point only guarantees a local minimum. The global minimum could be elsewhere.

10.5.13 Recap + Bridge

The Hessian matrix $H$ organizes all second partial derivatives into a symmetric matrix, converting the second-order Taylor term into the quadratic form $\frac{1}{2} x^{⊤} H x$ . The definiteness of $H$ (all-positive, all-negative

Or mixed) reveals whether a critical point is a minimum, maximum

Or saddle.

The Hessian is the judge at the critical point. Once you have found where the gradient vanishes, you form $H$ , compute its determinant and $f_{xx}$

And deliver the verdict. The next section formalizes the classification rules.

Real-world connection: In Newton's method for optimization, the update step uses the inverse Hessian: $x_{k + 1} = x_{k} - H^{- 1} \nabla f (x_{k})$ . The Hessian tells you not just which direction to go (gradient)

But how far —

By accounting for curvature. This is what lets Newton's method converge quadratically near the optimum, compared to gradient descent's linear convergence. In structural engineering, the Hessian of the potential energy function determines the stability of a structure —

A positive-definite Hessian means the structure is stable under load;

A non-positive-definite Hessian signals buckling or collapse. In Gaussian process regression, the Hessian of the log-likelihood (the Fisher information matrix) quantifies the uncertainty in hyperparameter estimates.

---

10.6 Classifying Critical Points Using the Hessian

10.6.1 The Classification Rules

You found flat ground —

Now what? You stand at a point where the gradient is zero. The ground is flat under your feet. But are you at the bottom of a bowl, the top of a dome

Or on a mountain pass that goes up in one direction and down in another? The Hessian ends the suspense. It tells you which kind of flat ground you are standing on.

Given a function $f (x, y)$ and a critical point $(a, b)$ where $\nabla f (a, b) = (0, 0)$ , form the Hessian matrix $H$ at that point. Compute its determinant:

The Second Derivative Test (2D). Let $D = det (H)$ at a critical point where $\nabla f = 0$ :

D = det (H) = f_{xx} f_{y y} - (f_{x y})^{2}

Then classify:

Condition	Result
$D > 0$ and $f_{xx} > 0$	Local minimum
$D > 0$ and $f_{xx} < 0$	Local maximum
$D < 0$	Saddle point
$D = 0$	Test inconclusive

10.6.2 Symbol Registry for Classification

$D = det (H)$ — determinant of the Hessian — scalar
$f_{xx}$ — second partial derivative in $x$ at the critical point — scalar
$f_{y y}$ — second partial derivative in $y$ at the critical point — scalar
$f_{x y}$ — mixed partial derivative at the critical point — scalar

10.6.3 Intuition for Each Case

The professor's landscape metaphor. At a critical point, the tangent plane is perfectly horizontal. The Hessian tells you what the landscape looks like immediately around you.

Minimum ( $D > 0$ , $f_{xx} > 0$ ): The bowl. Any step you take leads uphill. The surface is concave-up in all directions. Think of the bottom of a cereal bowl.
Maximum ( $D > 0$ , $f_{xx} < 0$ ): The dome. Any step you take leads downhill. The surface is concave-down in all directions. Think of the top of a hill.
Saddle point ( $D < 0$ ): The Pringles chip. In one direction you go up; in the perpendicular direction you go down. The surface looks like a horse saddle or a mountain pass.
Inconclusive ( $D = 0$ ): The test cannot tell. The landscape might be completely flat (a plateau), or have a trough, or a higher-order shape. You need to look at directional derivatives or the next-order Taylor terms.

10.6.4 Intuitive Explanation of the Determinant Rule

Look at the diagonal entries $f_{xx}$ and $f_{y y}$ . If both are positive, the function is curving upward along both pure directions. But that alone is not enough —

The cross-term $f_{x y}$ could overpower them and flip the sign when you move diagonally. The determinant test accounts for that cross-interaction.

When $f_{xx} > 0$ and $D > 0$ , the positive curvature along $x$ is strong enough that no mixed direction can make it go negative. You are at a minimum. Conversely, when $f_{xx} < 0$ and $D > 0$ , no mixed direction can make it go positive —

A maximum.

When $D < 0$ , the cross-term dominates. Some directions curve up, others down. Saddle point.

10.6.5 One-Variable Analogy

In single-variable calculus, you found critical points by setting $f^{'} (x) = 0$ . You classified them with $f^{''} (x)$ :

$f^{''} (x) > 0$ → minimum.
$f^{''} (x) < 0$ → maximum.

The Hessian test is the multivariable version of this. The 2nd derivative becomes a matrix. "Positive" becomes "positive definite." The determinant test is a shortcut for checking definiteness of a $2 \times 2$ symmetric matrix.

10.6.6 Why $f_{xx}$ Suffices When $D > 0$

For a $2 \times 2$ symmetric matrix, $D > 0$ means the two diagonal entries have the same sign. If $f_{xx} > 0$ , then $f_{y y}$ must also be positive (otherwise $D = f_{xx} f_{y y} - f_{x y}^{2}$ could not be positive —

A negative $f_{y y}$ would make the product negative and the determinant would only be saved if $f_{x y}^{2}$ were even more negative

Which is impossible since squares are non-negative).

Proof sketch: If $D > 0$ , then $f_{xx} f_{y y} > f_{x y}^{2} \geq 0$

So $f_{xx} f_{y y} > 0$ . This means $f_{xx}$ and $f_{y y}$ share the same sign. Checking just $f_{xx}$ is enough —

You get $f_{y y}$ 's sign for free.

For $n > 2$ , you check all leading principal minors, not just the determinant and one diagonal entry.

10.6.7 Worked Example — Saddle Point Detection

Find and classify the critical point of $f (x, y) = x^{2} - y^{2}$ .

Step 1 — Gradient:

f_{x} = 2 x = 0 \Rightarrow x = 0, f_{y} = - 2 y = 0 \Rightarrow y = 0

Critical point: $(0, 0)$ .

Step 2 — Hessian entries:

f_{xx} = 2, f_{y y} = - 2, f_{x y} = 0

Step 3 — Determinant:

D = (2) (- 2) - 0^{2} = - 4 < 0

Step 4 — Classify: $D < 0$ → saddle point at $(0, 0)$ .

Sense-check: $f (x, 0) = x^{2}$ —

Along the x-axis, the function curves upward (minimum at 0). $f (0, y) = - y^{2}$ —

Along the y-axis, the function curves downward (maximum at 0). A classic saddle. The function value at the critical point is $f (0, 0) = 0$ ;

Nearby it is positive in some directions and negative in others.

10.6.8 Assumptions & Scope

Scope: Limitations of the second derivative test.

Critical point required. The test only classifies points where $\nabla f = 0$ . If the gradient is non-zero, you are on a slope — there is no extremum there in the first place.
$D = 0$ is a dead end. When the determinant is zero, the Hessian test gives no answer. The point could be anything — a minimum, maximum, saddle, or something more exotic. You need to examine the function directly (graph it, check directional derivatives along suspected trouble directions, or compute higher-order Taylor terms).
Local, not global. A "local minimum" via Hessian tells you nothing about other critical points elsewhere. The function could dip lower at some faraway point.
The $n > 2$ generalization uses leading principal minors, not a single determinant. For a function of $n$ variables, the definiteness test checks the signs of $D_{1} = f_{x_{1} x_{1}}$ , $D_{2} = det [f_{x_{1} x_{1}} f_{x_{2} x_{1}} f_{x_{1} x_{2}} f_{x_{2} x_{2}}]$ , …, $D_{n} = det (H)$ . All positive → positive definite (minimum). Signs alternate starting negative → negative definite (maximum).

10.6.9 Visual Intuition

Plot a 3D surface with x and y axes on the horizontal plane and z = f(x,y) on the vertical axis. For a minimum, the surface near the critical point looks like an upright bowl —

Contour lines are concentric ellipses. For a maximum, the surface looks like an inverted bowl —

Contour lines are concentric ellipses but the z-values decrease outward. For a saddle point, the surface twists —

Along one axis it rises;

Along the perpendicular axis it falls. Contour lines form hyperbolas crossing at the saddle. The determinant $D = f_{xx} f_{y y} - f_{x y}^{2}$ is the Gaussian curvature (up to a factor) of the surface at that point. Positive curvature = local extremum. Negative curvature = saddle. One-sentence takeaway: the sign of $D$ distinguishes a bowl/dome from a saddle;

The sign of $f_{xx}$ distinguishes a bowl from a dome when $D > 0$ .

10.6.10 Pitfalls

Forgetting to verify $\nabla f = 0$ first. The Hessian test only makes sense at critical points. Applying it elsewhere gives curvature information but does not classify extrema.
Checking $f_{y y}$ instead of $f_{xx}$ is fine when $D > 0$ . Both have the same sign. But the canonical rule says $f_{xx}$ . Stick to it for consistency.
**Assuming $D > 0$ and $f_{xx} > 0$ means global minimum.** It only means local. There could be a deeper valley elsewhere.
Treating $D = 0$ as equivalent to a saddle. $D = 0$ means the test is inconclusive. A saddle requires $D < 0$ . These are different categories.
Forgetting the Hessian entries must be evaluated at the critical point. If $f_{xx} = 6 x$ , you must plug in the $x$ -coordinate of the critical point before checking its sign. The sign of $6 x$ changes with $x$ .

10.6.11 Comparison: 1D vs 2D Classification

Aspect	1D ( $f : R \to R$ )	2D ( $f : R^{2} \to R$ )
Critical point condition	$f^{'} (x) = 0$	$\nabla f = (f_{x}, f_{y}) = (0, 0)$
Curvature measure	Single number $f^{''} (x)$	Hessian matrix $H$ (3 numbers)
Minimum test	$f^{''} (x) > 0$	$D > 0$ and $f_{xx} > 0$
Maximum test	$f^{''} (x) < 0$	$D > 0$ and $f_{xx} < 0$
Saddle/inflection	$f^{''} (x) = 0$ (inconclusive/possible inflection)	$D < 0$ (definite saddle)
Possible outcomes	Min, max, inflection	Min, max, saddle, inconclusive
Key difference	No saddle possible in 1D	Saddle points are a new category in 2D+

10.6.12 Student Questions and Answers

Q: Why do the minima/maxima conditions only check $f_{xx}$ ? What if the function is decreasing in $y$ ?

A: If $D > 0$ and $f_{xx} > 0$ , the positive definiteness guarantees that $f_{y y}$ must also be positive. The determinant condition bundles the cross-check. A positive definite $2 \times 2$ symmetric matrix has both diagonal entries positive. Similarly, negative definite means both diagonals are negative.

Q: Is there a way to tell if a critical point is a global minimum or maximum just from the equation?

A: No. There is no mathematical method to guarantee a global extremum in general. You can only verify locally. If a function has only one critical point and the Hessian says it is a minimum, then it is the global minimum. But with multiple critical points, each must be evaluated separately. This is exactly the problem that makes machine learning hard —

With hundreds of variables, you cannot solve for all critical points directly. Iterative methods like gradient descent are used instead

And they can get stuck in local minima.

Q: From an exam perspective, what exactly do we do?

A: Find where $\nabla f = (0, 0)$ . Solve the system. Compute the Hessian entries at each critical point. Compute $D = f_{xx} f_{y y} - f_{x y}^{2}$ . Also note $f_{xx}$ . Classify using the table. If there are multiple critical points, evaluate each one separately.

10.6.13 The Saddle Point Problem in Deep Learning

In high-dimensional optimization (neural networks), saddle points are far more common than local minima. The Hessian becomes huge. Most directions are flat or mixed. The saddle point —

Where some directions curve up and others down —

Is the norm. This is why pure second-order methods are rarely used directly. The Hessian is too expensive to compute and too mixed to trust.

The probability that a random critical point in $n$ dimensions is a local minimum decays exponentially with $n$ . The ratio of saddle points to minima grows combinatorially. A 1000-dimensional loss surface is overwhelmingly saddle —

Which is why gradient descent with momentum (which can "roll through" shallow saddles) works so well in practice

And why Newton-type methods (which get attracted to saddles) require careful damping.

10.6.14 Recap + Bridge

At a critical point, compute $D = f_{xx} f_{y y} - f_{x y}^{2}$ . If $D > 0$ , the sign of $f_{xx}$ decides min vs max. If $D < 0$ , it is a saddle. If $D = 0$ , the test cannot decide. This is the multivariable upgrade of the 1D second derivative test

And it will be applied in the next worked example.

The Hessian is the judge. You have the crime scene (the critical point where the gradient vanished). Now you need to deliver the verdict. The next two sections walk through full worked examples —

One for Taylor expansion, one for extrema classification.

Real-world connection: Hessian-based saddle point detection is central to understanding why deep learning works at all. The loss surfaces of neural networks are dominated by saddle points, not poor local minima —

Meaning gradient descent rarely gets permanently trapped in bad spots. It mostly slows down near saddle points (where the gradient is small but not zero in all directions). Techniques like second-order optimization (Newton, L-BFGS), natural gradient descent

And trust-region methods explicitly use the Hessian or its approximations to account for curvature when deciding update steps. In computational chemistry, Hessian eigenvalue analysis determines whether a molecular geometry is a stable equilibrium (all positive eigenvalues) or a transition state (exactly one negative eigenvalue —

A saddle point on the potential energy surface).

---

10.7 Worked Example — Two-Variable Taylor Series

10.7.1 Problem Statement

This is the canonical Taylor exam problem. You are handed a two-variable function, told an anchor point and a degree

And asked to crank out the expansion. The process is mechanical —

List derivatives, evaluate, plug in —

But it rewards systematic work. Every skipped derivative is a lost mark.

Expand $f (x, y) = e^{x} sin y$ as a Taylor polynomial around $(0, 0)$ up to degree 3.

The anchor point is $(a, b) = (0, 0)$ , so $h = x - 0 = x$ and $k = y - 0 = y$ .

10.7.2 Step 1 — List All Required Derivatives

The general degree-3 formula from Section 10.4:

f (x, y) = f (0, 0) + (x f_{x} + y f_{y}) + \frac{1}{2 !} (x^{2} f_{xx} + 2 x y f_{x y} + y^{2} f_{y y}) + \frac{1}{3 !} (x^{3} f_{xxx} + 3 x^{2} y f_{xx y} + 3 x y^{2} f_{x y y} + y^{3} f_{y y y})

You need: $f$ , $f_{x}$ , $f_{y}$ , $f_{xx}$ , $f_{y y}$ , $f_{x y}$ , $f_{xxx}$ , $f_{y y y}$ , $f_{xx y}$ , $f_{x y y}$ . That is 10 partial derivatives in total (counting $f$ itself as the 0-th derivative). All evaluated at $(0, 0)$ .

10.7.3 Step 2 — Compute Each Derivative and Evaluate at (0, 0)

Degree 0 — The function itself:

f (x, y) = e^{x} sin y, f (0, 0) = e^{0} sin 0 = 1 \cdot 0 = 0

Degree 1 — First partial derivatives:

$f_{x}$ : differentiate $e^{x} sin y$ with respect to $x$ . The factor $sin y$ is constant with respect to $x$ . The derivative of $e^{x}$ is $e^{x}$ .

f_{x} = e^{x} sin y, f_{x} (0, 0) = e^{0} sin 0 = 0

$f_{y}$ : differentiate $e^{x} sin y$ with respect to $y$ . The factor $e^{x}$ is constant with respect to $y$ . The derivative of $sin y$ is $cos y$ .

f_{y} = e^{x} cos y, f_{y} (0, 0) = e^{0} cos 0 = 1 \cdot 1 = 1

Degree 2 — Second partial derivatives:

$f_{xx}$ : differentiate $f_{x} = e^{x} sin y$ with respect to $x$ again. Same as $f_{x}$ .

f_{xx} = e^{x} sin y, f_{xx} (0, 0) = 0

$f_{y y}$ : differentiate $f_{y} = e^{x} cos y$ with respect to $y$ . The derivative of $cos y$ is $- sin y$ .

f_{y y} = - e^{x} sin y, f_{y y} (0, 0) = - e^{0} sin 0 = 0

$f_{x y}$ : start from $f_{x} = e^{x} sin y$ . Differentiate with respect to $y$ . $e^{x}$ is constant. The derivative of $sin y$ is $cos y$ .

f_{x y} = e^{x} cos y, f_{x y} (0, 0) = e^{0} cos 0 = 1 \cdot 1 = 1

Degree 3 — Third partial derivatives:

$f_{xxx}$ : differentiate $f_{xx} = e^{x} sin y$ with respect to $x$ . Same result.

f_{xxx} = e^{x} sin y, f_{xxx} (0, 0) = 0

$f_{y y y}$ : differentiate $f_{y y} = - e^{x} sin y$ with respect to $y$ . The derivative of $sin y$ is $cos y$ .

f_{y y y} = - e^{x} cos y, f_{y y y} (0, 0) = - e^{0} cos 0 = - 1 \cdot 1 = - 1

$f_{xx y}$ : start from $f_{xx} = e^{x} sin y$ . Differentiate with respect to $y$ . Derivative of $sin y$ is $cos y$ .

f_{xx y} = e^{x} cos y, f_{xx y} (0, 0) = e^{0} cos 0 = 1 \cdot 1 = 1

$f_{x y y}$ : start from $f_{x y} = e^{x} cos y$ . Differentiate with respect to $y$ . Derivative of $cos y$ is $- sin y$ .

f_{x y y} = - e^{x} sin y, f_{x y y} (0, 0) = - e^{0} sin 0 = 0

10.7.4 Step 3 — Assemble

Plug all computed values into the formula:

Degree 0 term:

f (0, 0) = 0

Degree 1 term:

x \cdot f_{x} (0, 0) + y \cdot f_{y} (0, 0) = x \cdot 0 + y \cdot 1 = y

Degree 2 term:

\frac{1}{2 !} (x^{2} \cdot f_{xx} (0, 0) + 2 x y \cdot f_{x y} (0, 0) + y^{2} \cdot f_{y y} (0, 0)) = \frac{1}{2} (x^{2} \cdot 0 + 2 x y \cdot 1 + y^{2} \cdot 0) = \frac{1}{2} (2 x y) = x y

Degree 3 term:

\frac{1}{3 !} (x^{3} \cdot 0 + 3 x^{2} y \cdot 1 + 3 x y^{2} \cdot 0 + y^{3} \cdot (- 1)) = \frac{1}{6} (3 x^{2} y - y^{3}) = \frac{1}{2} x^{2} y - \frac{1}{6} y^{3}

10.7.5 Final Answer

e^{x} sin y \approx y + xy + \frac{1}{2} x^{2} y - \frac{1}{6} y^{3}

The approximation is valid near $(0, 0)$ and includes terms up to degree 3.

10.7.6 Numerical Sense-Check

Verify the approximation at a small point, say $x = 0.1$ , $y = 0.2$ :

True value: $e^{0.1} sin (0.2) \approx 1.10517 \times 0.19867 = 0.21956$

Approximation:

y + x y + \frac{1}{2} x^{2} y - \frac{1}{6} y^{3} = 0.2 + (0.1) (0.2) + \frac{1}{2} (0.01) (0.2) - \frac{1}{6} (0.008) = 0.2 + 0.02 + 0.001 - 0.00133 = 0.21967

Error: $∣0.21956 - 0.21967∣ \approx 0.00011$ — accurate to four decimal places. The degree-3 polynomial is practically exact for small $(x, y)$ near the origin. ✓

10.7.7 Assumptions & Scope

Scope: Limitations of this expansion.

Local only. The approximation degrades as $x$ or $y$ grow. At $x = 1$ , $y = 1$ , the true value is $e sin 1 \approx 2.718 \times 0.841 = 2.287$ . The degree-3 approximation gives $1 + 1 + 0.5 - 0.167 = 2.333$ — still decent (~2% error). At $x = 2$ , $y = 2$ , the error blows up because the exponential factor dominates and the truncated polynomial cannot keep up.
The function must be smooth. $e^{x} sin y$ is infinitely differentiable everywhere, so no problems. Functions like $∣ x ∣$ or $x$ have derivative issues that limit or prevent Taylor expansion at certain points.
Higher-degree terms are not zero. The degree-3 expansion stopped at 3 — it ignored degree-4 terms like $x^{4} /24 \cdot e^{x} sin y$ evaluated at $(0, 0)$ , which are zero for this particular function at the origin. But for other anchor points, or for functions like $e^{x} cos y$ , the higher terms matter.

10.7.8 Visual Intuition

The surface $z = e^{x} sin y$ near $(0, 0)$ looks like a gently twisted sheet. At the origin, the function value is zero —

You are at sea level. Moving in the pure $x$ -direction ( $sin y \approx y$ is near-zero for tiny $y$ ), the surface is nearly flat because $f_{x} (0, 0) = 0$ . Moving in the pure $y$ -direction, the surface rises linearly with slope 1 (the term $y$ ). Moving in the $x y$ -diagonal, the surface rises with an extra push from the mixed term $x y$ . The cubic term $- \frac{1}{6} y^{3}$ bends the surface slightly downward in the $y$ -direction as you move farther out. One-sentence takeaway: the polynomial $y + x y + \frac{1}{2} x^{2} y - \frac{1}{6} y^{3}$ is the degree-3 polynomial that best matches $e^{x} sin y$ near the origin in position, slope

And all curvatures.

10.7.9 Pitfalls

Missing the factor of 2 before $x y f_{x y}$ . In the degree-2 term, it is $2 x y f_{x y}$ , not $x y f_{x y}$ . The factor of 2 comes from the binomial coefficient $(1 2) = 2$ .
Mixing up $f_{x y}$ with $f_{x} f_{y}$ . $f_{x y}$ is a second derivative — differentiate once in $x$ , then once in $y$ . It is not the product of the two first derivatives.
Forgetting to evaluate at the anchor point. $f_{x y y} = - e^{x} sin y$ is the general formula. You must plug in $(0, 0)$ to get the coefficient. The fact that $f_{x y y} (0, 0) = 0$ means that specific term vanishes — not that you skip computing it.
Omitting the factorial denominators. Degree 2 uses $2! = 2$ . Degree 3 uses $3! = 6$ . These are not optional — they undo the chain of factors that repeated differentiation accumulates.
Writing $x$ instead of $x - 0$ . At $(0, 0)$ , $h = x$ and $k = y$ , so the shortcut works. At any other anchor, you must write $(x - a)$ and $(y - b)$ everywhere.

10.7.10 Student Questions and Answers

Q: Why is $f_{x y}$ computed by taking $f_{x}$ and differentiating with respect to $y$ ?

A: The notation $f_{x y}$ means: first differentiate with respect to $x$ , then differentiate the result with respect to $y$ . You compute $f_{x}$ first (getting $e^{x} sin y$ ), then treat that as a new function and take its $y$ -derivative. Since $e^{x}$ is constant with respect to $y$ , only $sin y$ gets differentiated, giving $e^{x} cos y$ .

Q: Does the order matter? Is $f_{x y}$ the same as $f_{y x}$ ?

A: For smooth functions (continuous second derivatives), yes. $f_{x y} = f_{y x}$ . You can take the derivatives in either order and get the same result. This is Clairaut's theorem. In this example, computing $f_{y x}$ (first $y$ , then $x$ ): $f_{y} = e^{x} cos y$ , then $f_{y x} = e^{x} cos y$ —

Identical to $f_{x y}$ .

Q: What if the anchor point is not $(0, 0)$ ? Say it is $(1, 1)$ ?

A: Then every term uses $x - 1$ instead of $x$

And $y - 1$ instead of $y$ . Compute all derivatives at $(1, 1)$ instead of $(0, 0)$ . For example, $f (1, 1) = e^{1} sin 1$ , $f_{x} (1, 1) = e^{1} sin 1$ , $f_{y} (1, 1) = e^{1} cos 1$

And so on. The structure of the formula stays the same;

Only the numbers change.

10.7.11 Exam Guidance

Exam note: The question format is predictable —

You get a function like $e^{x} sin y$ , $e^{x} (x^{2} + 1)$

Or $x^{2} lo g y$ . You are told an anchor point and a degree (usually 2 or 3). Compute all partial derivatives to that degree, evaluate at the point

And plug into the formula. Show each derivative step. Do not skip intermediate derivatives. The marking scheme rewards systematic work —

Even if you make an arithmetic error, a clear table of derivatives earns partial credit. Practice at least three of these before the exam.

10.7.12 Recap + Bridge

The degree-3 Taylor expansion of $e^{x} sin y$ about $(0, 0)$ is $y + x y + \frac{1}{2} x^{2} y - \frac{1}{6} y^{3}$ . The process is mechanical but requires care —

List all derivatives, compute each one, evaluate at the anchor, then assemble term by term. The Bernoulli coefficients at each degree are binomial coefficients divided by the factorial of the degree.

Now you have seen the full Taylor expansion pipeline. The next section flips the lens —

Instead of building a Taylor expansion, we use the gradient and Hessian to find and classify extrema.

Real-world connection: Every physics engine in a game or simulation uses truncated Taylor expansions for numerical integration. The Verlet integration algorithm updates positions using a second-order Taylor expansion of the equations of motion —

Matching position and velocity exactly, with error in the jerk term. When you see cloth ripple or water splash in a video game, each particle's trajectory is computed with exactly this kind of local Taylor approximation.

---

10.8 Worked Example — Extrema Using the Hessian

10.8.1 Problem Statement

This is the canonical extrema exam problem. You are given a two-variable function, told to find and classify its critical points. The four-step recipe —

Gradient zero, Hessian entries, determinant, classify —

Works every time. This is the problem that tests whether you understand what the Hessian is for.

Find and classify the extrema of:

f (x, y) = x y - x^{2} - y^{2} - 2 x - 2 y + 4

10.8.2 Step 1 — Find Critical Points

Set the gradient to zero:

f_{x} f_{y} = \frac{\partial}{\partial x} (x y - x^{2} - y^{2} - 2 x - 2 y + 4) = y - 2 x - 2 = 0 = \frac{\partial}{\partial y} (x y - x^{2} - y^{2} - 2 x - 2 y + 4) = x - 2 y - 2 = 0

Solve the system of two linear equations:

{y - 2 x = 2 x - 2 y = 2

From the first equation: $y = 2 x + 2$ . Substitute into the second:

x - 2 (2 x + 2) = 2 \Rightarrow x - 4 x - 4 = 2 \Rightarrow - 3 x = 6 \Rightarrow x = - 2

Then $y = 2 (- 2) + 2 = - 4 + 2 = - 2$ .

The critical point is $(- 2, - 2)$ .

10.8.3 Step 2 — Compute the Hessian Entries

f_{xx} f_{y y} f_{x y} = \frac{\partial}{\partial x} (y - 2 x - 2) = - 2 = \frac{\partial}{\partial y} (x - 2 y - 2) = - 2 = \frac{\partial}{\partial y} (y - 2 x - 2) = 1

All three second derivatives are constants —

They do not depend on $x$ or $y$ . This means the curvature is the same everywhere on the surface.

The Hessian at $(- 2, - 2)$ :

H = [- 2 1 1 - 2]

10.8.4 Step 3 — Classify

Compute the determinant:

det (H) = f_{xx} f_{y y} - (f_{x y})^{2} = (- 2) (- 2) - (1)^{2} = 4 - 1 = 3

$det (H) = 3 > 0$ (positive).
$f_{xx} = - 2 < 0$ (negative).

Apply the classification table from Section 10.6:

$det (H) > 0$ and $f_{xx} < 0$ → local maximum.

Conclusion: the critical point $(- 2, - 2)$ is a local maximum.

10.8.5 Step 4 — Compute the Maximum Value

Plug $x = - 2, y = - 2$ into $f$ :

f (- 2, - 2) = (- 2) (- 2) - (- 2)^{2} - (- 2)^{2} - 2 (- 2) - 2 (- 2) + 4 = 4 - 4 - 4 + 4 + 4 + 4 = 8

The maximum value is $f (- 2, - 2) = 8$ .

10.8.6 Step 5 — Verify by Intuition

Both $f_{xx}$ and $f_{y y}$ are negative $(- 2)$ . The function curves downward in both the $x$ and $y$ directions. The determinant is positive, confirming the curvature is consistently downward in every direction —

The quadratic form $x^{⊤} H x = - 2 h^{2} + 2 hk - 2 k^{2}$ is always negative for non-zero $(h, k)$ because the cross-term $2 hk$ is never large enough to overcome $- 2 h^{2} - 2 k^{2}$ (since $∣2 hk ∣ \leq h^{2} + k^{2}$ by the AM-GM inequality). You are at the top of a hill.

10.8.7 Second Worked Example — Multiple Critical Points

Find and classify all critical points of $f (x, y) = x^{3} + y^{3} - 3 x y + 1$ . (From the companion document.)

Step 1 — Gradient:

f_{x} = 3 x^{2} - 3 y = 0 \Rightarrow y = x^{2}

f_{y} = 3 y^{2} - 3 x = 0 \Rightarrow x = y^{2}

Substitute $y = x^{2}$ into $x = y^{2}$ :

x = (x^{2})^{2} = x^{4} \Rightarrow x^{4} - x = 0 \Rightarrow x (x^{3} - 1) = 0

So $x = 0$ or $x = 1$ .

If $x = 0$ , then $y = 0^{2} = 0$ . Critical point: $(0, 0)$ .
If $x = 1$ , then $y = 1^{2} = 1$ . Critical point: $(1, 1)$ .

Step 2 — Hessian entries:

f_{xx} = 6 x, f_{y y} = 6 y, f_{x y} = - 3

Step 3 — Classify each point:

At $(0, 0)$ :

f_{xx} = 0, f_{y y} = 0, f_{x y} = - 3

D = (0) (0) - (- 3)^{2} = - 9 < 0

$D < 0$ → Saddle point at $(0, 0)$ .

At $(1, 1)$ :

f_{xx} = 6, f_{y y} = 6, f_{x y} = - 3

D = (6) (6) - (- 3)^{2} = 36 - 9 = 27 > 0

$D > 0$ and $f_{xx} = 6 > 0$ → Local minimum at $(1, 1)$ .

Values: $f (0, 0) = 1$ . $f (1, 1) = 1 + 1 - 3 + 1 = 0$ .

Sense-check: The function is cubic in both variables. Near $(0, 0)$ , in the direction $y = x$ , $f (x, x) = 2 x^{3} - 3 x^{2} + 1 \approx 1 - 3 x^{2}$ —

Curving downward. In the direction $y = - x$ , $f (x, - x) = x^{3} - x^{3} + 3 x^{2} + 1 = 1 + 3 x^{2}$ —

Curving upward. Mixed curvature confirms the saddle. Near $(1, 1)$ , the function dips to a minimum value of 0.

10.8.8 Assumptions & Scope

Scope: What makes Hessian classification easy or hard.

Constant Hessian entries are rare. For this function, $f_{xx} = f_{y y} = - 2$ are constants. For most functions (like $x^{3} + y^{3} - 3 x y$ ), the Hessian entries vary with $x$ and $y$ . Always evaluate them at the critical point.
Linear gradient equations are also rare. The system $y - 2 x = 2$ , $x - 2 y = 2$ was two linear equations — easy to solve. For nonlinear gradients (like $3 x^{2} - 3 y = 0$ ), you may get zero, one, or many critical points.
A single critical point with constant negative-definite Hessian means global maximum. The function is a concave quadratic. No other critical point can exist, because the gradient is a linear system with a unique solution. The Hessian negativity means the function is concave everywhere, so the unique stationary point is the global maximizer.
With multiple critical points, no global guarantee without comparing values. You must evaluate $f$ at each candidate and at the boundaries (if the domain is bounded) to find the global extremum.

10.8.9 Visual Intuition

The function $f (x, y) = x y - x^{2} - y^{2} - 2 x - 2 y + 4$ is a concave quadratic —

An inverted paraboloid. Completing the square shows it is $f (x, y) = 8 - (x + 2)^{2} - (y + 2)^{2} + x y + 4 x + 4 y$

Which after rotation amounts to a downward-opening bowl centered at $(- 2, - 2)$ with maximum value 8. The contour lines are concentric ellipses, getting smaller and higher as they approach the center. The gradient arrows all point toward the peak. The Hessian $H = [- 2 1 1 - 2]$ has eigenvalues $- 1$ and $- 3$ —

Both negative, confirming the surface bends downward in all directions, more steeply along one axis (eigenvalue $- 3$ ) than the other ( $- 1$ ). One-sentence takeaway: this function is a textbook concave quadratic —

One peak, no surprises, perfectly classified by the Hessian.

10.8.10 Pitfalls

Solving the gradient equations sloppily. For linear systems, double-check your algebra. A sign error in $x = - 2$ vs $x = 2$ flips the entire classification.
Forgetting to compute $f (x, y)$ at the critical point. The question asks you to find the extrema — that means reporting both the point AND the function value there. The Hessian only classifies the type; it does not give the value.
Assuming constant Hessian implies constant classification everywhere. For this function, the Hessian is constant, so one classification suffices. For non-constant Hessians (like with $f_{xx} = 6 x$ ), the sign depends on where you are. Classify each critical point independently.
Stopping at the determinant without checking $f_{xx}$ . $D > 0$ narrows it to min or max, but you need $f_{xx}$ 's sign to finish. For symmetric $2 \times 2$ matrices, $D > 0$ means both diagonal entries have the same sign, so either one works — but the convention is to state $f_{xx}$ .

10.8.11 Student Questions and Answers

Q: Can we say this is a global maximum?

A: Yes, in this case. The Hessian entries are constants —

The curvature is the same everywhere. There is only one critical point. Since the Hessian is negative definite everywhere

That single critical point is the global maximum. The function value there is 8

And $f (x, y) \leq 8$ for all $(x, y)$ .

Q: What if we get multiple critical points?

A: Evaluate the Hessian at each one separately. Some might be minima, some maxima, some saddle points. You cannot declare a global extremum without checking all of them and comparing their function values. Also check the function's behavior as $(x, y) \to \infty$ —

For unbounded domains, the global extremum might not exist at all.

Q: From an exam perspective, what exactly do we do?

A: Four steps: (1) Set $\nabla f = (0, 0)$ and solve for critical point(s). (2) Compute $f_{xx}, f_{y y}, f_{x y}$ and evaluate at each critical point. (3) Compute $D = f_{xx} f_{y y} - f_{x y}^{2}$ . (4) Classify using the table: $D > 0, f_{xx} > 0$ → min;

$D > 0, f_{xx} < 0$ → max;

$D < 0$ → saddle;

$D = 0$ → inconclusive. Also report the function value at each extremum.

10.8.12 Exam Guidance

Exam note: The extrema question follows this exact template. Expect one function to classify —

Possibly with multiple critical points (like $x^{3} + y^{3} - 3 x y$ ). Practice the four-step recipe until it is automatic. Common variants: the function might be $x^{3} + 3 x y^{2} - 15 x - 12 y$ or $x^{4} + y^{4} - 4 x y + 1$ . For the second example in this section, you got two critical points —

One saddle at $(0, 0)$ , one minimum at $(1, 1)$ . The professor expects you to classify all critical points found. Missing one is a lost mark.

10.8.13 Recap + Bridge

$f (x, y) = x y - x^{2} - y^{2} - 2 x - 2 y + 4$ has a global maximum at $(- 2, - 2)$ with value 8. The classification pipeline —

Gradient zero → Hessian entries → determinant → classify —

Is the same for every two-variable function. The only variation is whether the Hessian entries are constants (easy) or functions of $(x, y)$ (requires plugging in the critical point).

Now you have seen both the Taylor expansion pipeline (Section 10.7) and the Hessian classification pipeline (this section). The next two sections close the lecture with two more directly applicable topics —

Linearization (the simplest Taylor approximation) and backpropagation (the chain rule on graphs).

Real-world connection: This exact problem type —

Find and classify the extrema of a multivariate function —

Is the central task in maximum likelihood estimation. When you fit a Gaussian distribution to data, you maximize the log-likelihood function. You set its gradient to zero, compute the Hessian of the log-likelihood (which gives you the Fisher information matrix)

And verify the point is a maximum. In portfolio optimization (Markowitz mean-variance), you minimize portfolio variance subject to return constraints —

The objective is a quadratic function whose Hessian is the covariance matrix of asset returns. Classifying the critical point confirms you have the global minimum-variance portfolio.

---

10.9 Linearization

10.9.1 Definition

What is the simplest possible Taylor approximation? Take the full Taylor series and stop after degree 1. That's linearization —

Just the position and the slope, no curvature. It turns a curved function into a straight line (or flat plane) that touches it at one point. It's the math equivalent of saying "for small changes, everything looks linear."

Linearization is the first-order Taylor approximation. It approximates $f (x)$ near a point $x_{0}$ using only the position and slope:

1D Linearization:

L (x) = f (x_{0}) + f^{'} (x_{0}) (x - x_{0})

2D Linearization:

L (x, y) = f (x_{0}, y_{0}) + f_{x} (x_{0}, y_{0}) (x - x_{0}) + f_{y} (x_{0}, y_{0}) (y - y_{0})

That is it. No second derivatives. No curvature. No factorials. Just the function's value at the anchor point plus the gradient dotted with the step.

In vector form, for a function $f : R^{n} \to R$ :

L (x) = f (x_{0}) + \nabla f (x_{0})^{⊤} (x - x_{0})

10.9.2 Symbol Registry

$L (x)$ — the linearization (linear approximation) of $f$ at $x_{0}$ — real-valued function of $x$
$x_{0}$ — the anchor point where you know the function's value — constant in $R$
$f (x_{0})$ — the function value at $x_{0}$ — scalar
$f^{'} (x_{0})$ — the first derivative (slope) at $x_{0}$ — scalar
$x - x_{0}$ — the step size from the anchor — scalar

10.9.3 When Linearization Works

Linearization is accurate when $x$ is close to $x_{0}$ . The slope at $x_{0}$ is a good proxy for the average slope over a tiny interval. As the step size grows, the error from ignored curvature grows quadratically with the step size —

That quadratic error is exactly the second-order Taylor term you dropped:

f (x) - L (x) \approx \frac{f ^{''} ( x _{0} )}{2} (x - x_{0})^{2} (for small steps)

This is why linearization is a "local" tool. Near $x_{0}$ , the error is tiny. Far from $x_{0}$ , the curvature terms you ignored dominate and the approximation becomes useless.

10.9.4 Worked Example 1 — 1D

Find the linearization of $f (x) = x^{2} + 1$ at $x_{0} = - 4$ . (From the lecture.)

Step 1 — Value of the function at $x_{0}$ :

f (- 4) = (- 4)^{2} + 1 = 16 + 1 = 17

Step 2 — First derivative:

f^{'} (x) = \frac{1}{2 x ^{2} + 1} \cdot 2 x = \frac{x}{x ^{2} + 1}

Evaluate at $x_{0} = - 4$ :

f^{'} (- 4) = \frac{- 4}{17}

Step 3 — Assemble:

L (x) = 17 + \frac{- 4}{17} (x + 4)

Simplify if needed:

L (x) = 17 - \frac{4}{17} (x + 4)

Numerical sense-check: $17 \approx 4.123$ . At $x = - 4$ , $L (- 4) = 17 \approx 4.123$

Which matches $f (- 4)$ . At $x = - 3.9$ (a small nudge of 0.1), the linearization predicts $L (- 3.9) = 4.123 - 0.970 \times 0.1 = 4.026$ . The true value $f (- 3.9) = 15.21 + 1 = 16.21 \approx 4.026$ —

The linearization is excellent at close range. ✓

10.9.5 Worked Example 2 — 2D

Find the linearization of $f (x, y) = e^{x} cos y$ at $(x_{0}, y_{0}) = (0, 0)$ . (From the companion document.)

Step 1 — Value of the function:

f (0, 0) = e^{0} cos 0 = 1 \cdot 1 = 1

Step 2 — Gradient:

f_{x} = e^{x} cos y, f_{y} = - e^{x} sin y

Evaluate at $(0, 0)$ :

f_{x} (0, 0) = e^{0} cos 0 = 1, f_{y} (0, 0) = - e^{0} sin 0 = 0

Step 3 — Assemble:

L (x, y) = f (0, 0) + f_{x} (0, 0) (x - 0) + f_{y} (0, 0) (y - 0) = 1 + 1 \cdot x + 0 \cdot y = 1 + x

Conclusion: Near the origin, $e^{x} cos y \approx 1 + x$ . The function behaves like a plane tilted in the $x$ -direction only —

The first-order change in $y$ is zero because the cosine is flat at $y = 0$ .

Sense-check: At $(0.1, 0.1)$ , the true value is $e^{0.1} cos (0.1) \approx 1.1052 \times 0.9950 = 1.0997$ . The linear approximation gives $L = 1 + 0.1 = 1.1$ . Error: $∣1.0997 - 1.1∣ \approx 0.0003$ — incredibly accurate at this small step. ✓

10.9.6 Assumptions & Scope

Scope: When linearization is good enough and when it is not.

Step size matters. If $∣ x - x_{0} ∣$ is tiny, linearization is excellent. If the step is large, the error from dropped curvature and higher-order terms dominates.
The function must be differentiable at $x_{0}$ . If $f^{'} (x_{0})$ does not exist (sharp corner, cusp), there is no tangent line to build the linearization around.
Linearization replaces the function with its tangent. For a concave function like $x$ , the linearization always overestimates. For a convex function like $e^{x}$ , it always underestimates. Knowing the sign of $f^{''}$ tells you which direction the error goes.
Linearization is the foundation of gradient descent. Each step of gradient descent moves along the direction of steepest descent as predicted by linearization. The first-order Taylor approximation justifies taking a small step opposite the gradient.
Linearization cannot capture curvature-dependent behavior. If the function bends sharply near $x_{0}$ (large $∣ f^{''} (x_{0}) ∣$ ), linearization breaks quickly. The second-order Taylor term would be needed for even moderate step sizes.

10.9.7 Visual Intuition

Plot $y = f (x) = x^{2} + 1$ over $x \in [- 6, - 2]$ . At $x_{0} = - 4$ , the function has value $17 \approx 4.12$ and is sloping downward (since $f^{'} (- 4) = - 4/ 17 \approx - 0.97$ ). The linearization $L (x)$ is the tangent line at that point —

A straight line passing through $(- 4, 17)$ with slope $- 4/ 17$ . Near $x = - 4$ , the tangent line hugs the curve closely. Move out to $x = - 6$ or $x = - 2$

And the curve pulls away from the straight line —

The linearization underestimates the function at both ends because $x^{2} + 1$ is convex (its second derivative is positive). One-sentence takeaway: linearization replaces a curve with its tangent line —

Accurate near the touch point, diverging as you move away.

10.9.8 Pitfalls

Forgetting the chain rule in the derivative. For $f (x) = x^{2} + 1$ , the derivative is $\frac{x}{x ^{2} + 1}$ , not $\frac{1}{2 x ^{2} + 1}$ alone. The chain rule brings the factor $2 x$ .
Dropping the sign of the derivative. At $x_{0} = - 4$ , $f^{'} (- 4) = - 4/ 17$ — it is negative. The tangent line slopes downward. Forgetting the negative sign gives an upward slope, which is completely wrong.
Using $x$ instead of $x - x_{0}$ . The formula is $L (x) = f (x_{0}) + f^{'} (x_{0}) (x - x_{0})$ . If $x_{0} = - 4$ , then $x - (- 4) = x + 4$ . Writing $x - 4$ instead is a common slip.
Confusing linearization with interpolation. Linearization matches the function value and derivative at one point. Interpolation matches function values at multiple points. These serve different purposes.

10.9.9 Exam Guidance

Exam note: A linearization question is a Taylor series question cut off at degree 1. Compute $f (x_{0})$ and $f^{'} (x_{0})$

Or $f (x_{0}, y_{0})$ and the gradient for 2D. Plug into $L = f (x_{0}) + f^{'} (x_{0}) (x - x_{0})$ . That is the whole answer. A common variant gives a 2D function and asks for the linearization at a specific point —

Compute both partial derivatives and assemble. Do not overthink it;

No factorials, no second derivatives.

10.9.10 Recap + Bridge

Linearization is the degree-1 Taylor approximation. It captures position and slope, ignoring curvature and all higher-order bends. It is accurate near the anchor point and forms the mathematical basis for gradient descent —

At each step, you use the linear approximation to decide where to go.

The last concept in this lecture, backpropagation, uses exactly the machinery built so far —

Partial derivatives, the chain rule

And the gradient —

To compute gradients through networks of functions efficiently.

Real-world connection: Linearization is the first step in virtually every nonlinear control system. When a drone stabilizes itself mid-flight, the flight controller linearizes the nonlinear aerodynamics around the current orientation and solves a linear quadratic regulator (LQR) problem 100 times per second. Self-driving cars linearize their motion models around the current state estimate in Extended Kalman Filters. In circuit design, transistors are nonlinear devices

But small-signal analysis linearizes them around a DC operating point, enabling frequency response analysis with simple linear techniques. Any time you hear "small-signal model" or "linear regime," you are hearing about linearization in the wild.

---

10.10 Backpropagation and Computational Graphs

10.10.1 What Backpropagation Is

You know the chain rule. So you already know backpropagation. If you can compute $\frac{d}{d x} sin (x^{2})$ by saying "derivative of sin is cos, times derivative of the inside which is $2 x$ ," you have done backpropagation. Backprop is nothing more than the chain rule applied to a network of operations —

Doing it backwards, one layer at a time

So you never repeat work.

Backpropagation is the chain rule applied to a network of connected computations. It is the algorithm that trains neural networks. But the core idea is simple.

Consider a function defined through intermediate steps. Say $A$ flows into $B$ and $C$ . Both $B$ and $C$ flow into $D$ .

Forward pass: compute all intermediate values, working left to right. Backward pass: compute how the output $D$ changes with respect to the input $A$ , working right to left.

10.10.2 Why Backpropagation Exists

Purpose. Manual differentiation of deeply nested functions —

Like neural networks with millions of parameters —

Is impossible. Backpropagation automates gradient computation by breaking a complex function into elementary operations whose derivatives are known, then applying the chain rule systematically in reverse. The result: the gradient of the output with respect to every input and every intermediate parameter, computed in one backward pass.

10.10.3 Inputs & Outputs

Inputs:

A computational graph (DAG) of elementary operations (+, ×, sin, exp, matrix multiply, etc.)
An input value for each leaf node
A final output node whose gradient we want to propagate

Outputs:

The gradient $\frac{\partial output}{\partial leaf}$ for every leaf node in the graph — efficiently, in a single backward pass

The forward pass computes all intermediate values. The backward pass computes all gradients, reusing the forward values where needed.

10.10.4 The Chain Rule on a Graph

To get $\frac{\partial D}{\partial A}$ , trace every path from $A$ to $D$ and multiply derivatives along each path, then sum:

\frac{\partial D}{\partial A} = \frac{\partial D}{\partial B} \cdot \frac{\partial B}{\partial A} + \frac{\partial D}{\partial C} \cdot \frac{\partial C}{\partial A}

If a node $u$ feeds into multiple downstream nodes $v_{1}, v_{2}, \dots$ , its total influence on the output is the sum over all paths:

\frac{\partial f}{\partial u} = i \sum \frac{\partial f}{\partial v _{i}} \cdot \frac{\partial v _{i}}{\partial u}

This is the multivariate chain rule. Each edge stores a local derivative. The backward pass accumulates them along all paths.

10.10.5 Steps — The Forward/Backward Algorithm

Forward Pass (left to right):

Start with the input node(s), with their actual numeric values.
At each node, apply its operation using the values from its parents, producing a new value.
Store each intermediate result — these will be needed during the backward pass.
Continue until you reach the output node.

Backward Pass (right to left):

Initialize the gradient of the output with respect to itself: $f ˉ = \frac{\partial f}{\partial f} = 1$ .
At each node $v$ producing output that feeds into downstream nodes:

Compute each local derivative $\frac{\partial v}{\partial u}$ for every input $u$ .
Multiply the downstream gradient $v ˉ$ by each local derivative to get $u ˉ$ .
If $u$ feeds into multiple nodes, sum all incoming gradient contributions.

Continue until all leaf nodes have accumulated gradients.

The notation $u ˉ = \frac{\partial f}{\partial u}$ (read "u-bar") is the total gradient of the final output with respect to node $u$ .

10.10.6 Trace — Worked Example from the Companion Document

Compute $\frac{df}{d x}$ at $x = 2$ for:

f (x) = x^{2} + e^{x^{2}} + cos (x^{2} + e^{x^{2}})

Step 1 — Decompose into elementary operations:

a b c d e f = x^{2} = e^{a} = a + b = c = cos (c) = d + e

Step 2 — Forward pass (compute all values with $x = 2$ ):

x a b c d e f = 2.000 = 2^{2} = 4.000 = e^{4} \approx 54.598 = 4 + 54.598 = 58.598 = 58.598 \approx 7.655 = cos (58.598) \approx 0.830 = 7.655 + 0.830 = 8.485

Step 3 — Backward pass (compute gradients right to left):

Start: $f ˉ = \frac{\partial f}{\partial f} = 1$

Node f = d + e : Node c \to d, e : Node b = e^{a} : Node a \to b, c : Node x : d ˉ = f ˉ \cdot \frac{\partial f}{\partial d} = 1 \cdot 1 = 1.000 e ˉ = f ˉ \cdot \frac{\partial f}{\partial e} = 1 \cdot 1 = 1.000 c ˉ = d ˉ \cdot \frac{\partial d}{\partial c} + e ˉ \cdot \frac{\partial e}{\partial c} = 1 \cdot \frac{1}{2 c} + 1 \cdot (- sin c) = \frac{1}{2 58.598} - sin (58.598) \approx \frac{1}{15.311} - 0.075 \approx 0.0653 - 0.0749 = - 0.0096 b ˉ = c ˉ \cdot \frac{\partial c}{\partial b} = - 0.0096 \cdot 1 = - 0.0096 a ˉ = b ˉ \cdot \frac{\partial b}{\partial a} + c ˉ \cdot \frac{\partial c}{\partial a} = (- 0.0096) \cdot e^{a} + (- 0.0096) \cdot 1 = (- 0.0096) \cdot 54.598 + (- 0.0096) \approx - 0.524 - 0.010 = - 0.534 x ˉ = a ˉ \cdot \frac{\partial a}{\partial x} = (- 0.534) \cdot 2 x = (- 0.534) \cdot 4 = - 2.136

Final answer: $\frac{df}{dx}_{x = 2} \approx - 2.136$

Sense-check: The function $f (x)$ decreases as $x$ increases at $x = 2$ —

The negative gradient confirms this. If we nudge $x$ upward by a tiny amount $ϵ$ , $f$ should drop by roughly $2.136 ϵ$ . The magnitude is reasonable for a function whose value is ~8.5 and whose dominant term is $x^{2} + e^{x^{2}}$ with $e^{x^{2}} \approx 54.6$ dominating the square root.

10.10.7 Complexity & Cost

The forward pass requires one evaluation per node. The backward pass requires one gradient computation per edge. The total cost of computing the gradient with respect to all inputs via backpropagation is roughly 2× the cost of a single forward pass —

One forward compute, one backward compute. This is what makes it feasible to train networks with millions of parameters. Without backprop, computing each parameter's gradient independently would cost $O (p)$ forward passes for $p$ parameters. With backprop: one forward + one backward, regardless of $p$ .

The memory cost scales with the graph size —

You must store all intermediate forward-pass values because the backward pass needs them to compute local derivatives. For deep learning, this is the leading memory bottleneck. Techniques like activation checkpointing trade compute for memory by recomputing some forward values during the backward pass.

10.10.8 When to Use / Alternatives

Use backpropagation when: you need gradients through a differentiable computational graph — neural network training, variational inference, neural ODEs, differentiable physics simulators.
Use numerical differentiation (finite differences) when: you only need to verify a handful of gradients, or the function is not analytically differentiable. $O (p)$ cost makes it impractical for training.
Use symbolic differentiation when: the function is small and you want an exact analytical expression. This blows up exponentially on large graphs (expression swell).
Use automatic differentiation (backprop is the reverse-mode variant) when: you have many inputs and few outputs — the standard case in ML (millions of parameters → one scalar loss). Forward-mode AD is better when you have few inputs and many outputs.

10.10.9 Symbol Registry

$A, B, C, D$ — nodes in a computational graph — can be scalars, vectors, or tensors
$\frac{\partial D}{\partial A}$ — the total derivative of output $D$ with respect to input $A$ — scalar
$u ˉ$ — "u-bar": shorthand for $\frac{\partial f}{\partial u}$ , the gradient of final output $f$ w.r.t. node $u$
$\frac{\partial D}{\partial B}$ — local derivative of $D$ with respect to intermediate node $B$ — scalar
$\frac{\partial B}{\partial A}$ — local derivative of $B$ with respect to input $A$ — scalar

10.10.10 How It Connects to Neural Networks

A neural network is one giant computational graph. Each layer computes:

a_{i} = σ (W_{i} a_{i - 1} + b_{i})

Where $a_{i - 1}$ is the input from the previous layer, $W_{i}$ and $b_{i}$ are the parameters (weights and biases), and $σ$ is an activation function (sigmoid, tanh, ReLU).

The goal: find parameters $θ = {W_{1}, b_{1}, \dots, W_{L}, b_{L}}$ minimizing a loss, e.g., squared error:

L (θ) = ∥ y_{true} - a_{L} ∥^{2}

Backpropagation computes $\frac{\partial L}{\partial W _{i}}$ and $\frac{\partial L}{\partial b _{i}}$ for every layer in one backward pass. Update rule (gradient descent):

W_{i} \leftarrow W_{i} - η \frac{\partial L}{\partial W _{i}}, b_{i} \leftarrow b_{i} - η \frac{\partial L}{\partial b _{i}}

Backpropagation is not a new mathematical idea. It is the chain rule, applied systematically to a densely connected graph. Once you understand partial derivatives and the chain rule, backpropagation follows naturally.

10.10.11 Assumptions & Scope

Scope: When backpropagation works and when it doesn't.

The graph must be differentiable. Every operation in the graph must have a well-defined local derivative. Operations like "if x > 0 then y = 1 else y = 0" break differentiability. ReLU is differentiable except at $x = 0$ , where the subgradient is used.
Directed acyclic graph (DAG). Backprop works on feedforward graphs. For recurrent networks with cycles, you unroll the graph in time (backpropagation through time, BPTT), creating a DAG.
Vanishing/exploding gradients. For very deep networks, the chain of multiplied derivatives can shrink to zero (vanishing) or blow up to infinity (exploding). This is a practical problem, not a mathematical failure — it is addressed by residual connections, careful initialization, gradient clipping, and normalization layers.
Memory scales with graph depth. Storing all intermediate activations for the backward pass is expensive. For a 100-layer network with batch size 64 and activation size $1024 \times 1024$ , the activation memory alone can exceed GPU RAM.

10.10.12 Visual Learning Resource

Interactive visualizations help make backpropagation intuitive. MLU Explain (Machine Learning University Explain) has an interactive neural network demo that steps through the forward and backward passes. Watching each weight update as the error flows backward makes the chain of derivatives concrete.

10.10.13 Student Questions and Answers

Q: How does this relate to the gradient and Taylor series we covered?

A: The gradient tells you the direction of steepest change for a function. Backpropagation is how you actually compute that gradient for a neural network —

A function defined by many composed operations. You use the chain rule (backprop) to get the gradient

And then you use the gradient to update weights (gradient descent). The first-order Taylor approximation is the theoretical foundation: you move weights opposite the gradient to reduce the loss. The full chain is: Taylor → gradient → gradient descent

And backprop → efficiently computing that gradient for huge functions.

10.10.14 Pitfalls

Forgetting to sum gradients when a node has multiple children. If node $u$ flows into $v_{1}$ and $v_{2}$ , then $u ˉ = v ˉ_{1} \frac{\partial v _{1}}{\partial u} + v ˉ_{2} \frac{\partial v _{2}}{\partial u}$ . Missing one path gives the wrong gradient.
Accumulating gradients instead of resetting between batches. In frameworks like PyTorch, calling .backward() accumulates gradients in .grad. Forgetting to zero them (optimizer.zero_grad()) causes the gradient to be the sum over all previous batches — a common debugging nightmare.
Confusing the backward pass with backpropagation-through-time (BPTT). Standard backprop works on DAGs. Recurrent networks need BPTT, which unrolls the recurrence into a DAG over time steps.
Assuming backprop is always reverse-mode AD. Reverse-mode AD (what backprop is) is optimal for scalar outputs with many inputs. Forward-mode AD is optimal for few inputs with many outputs. The choice matters for applications beyond neural network training.

10.10.15 Recap + Bridge

Backpropagation is the chain rule applied backwards through a computational graph, computing the gradient of a scalar output with respect to every input in a single right-to-left pass. It is not magic —

It is the systematic application of the chain rule, reusing intermediate gradients to avoid redundant computation. The forward pass computes values;

The backward pass computes derivatives by multiplying local derivatives along each edge and summing at every node.

Backpropagation ties together every concept in this lecture. Rolle's theorem and MVT built the logical foundation for approximation. Taylor series gave the polynomial-building tool. The gradient tells you which direction to move. The Hessian tells you about curvature. And backpropagation is how you actually compute the gradient for real-world functions that are too complex to differentiate by hand.

Real-world connection: Every deep learning framework —

TensorFlow, PyTorch, JAX —

Is built on automatic differentiation (backprop). When you write loss.backward() in PyTorch, you are running exactly the backward pass algorithm described here, scaled to millions of parameters. The same technique powers differentiable physics engines (Brax, DiffTaichi) that optimize robot controllers by backpropagating through simulated physics. In computational finance, the "Greeks" (sensitivities of option prices to underlying parameters) are computed via AAD (adjoint algorithmic differentiation) —

The same reverse-mode AD that backs neural network training. Google's PageRank can be understood as a fixed-point computation whose sensitivity to link structure is computed via the same chain-rule accumulation.

---

Exam Guidance Summary

Exam note: The exam has 5 to 6 problems total. Lectures 6 and 7 carry the most weight —

Expect at least two or three problems from those sections. Lecture 8 (linearization, backpropagation) is lighter.

Question distribution (approximate):

Problem	Topic	Weight
1	REF / RREF — guaranteed	Standard
2	Linear combination, independence, or basis — one question from these related concepts	Standard
3	Vector spaces — proving a set is a vector space or subspace	Standard
4	Decomposition — SVD, eigendecomposition, or diagonalization	Standard
5	Taylor series — possibly combined with gradient concepts	Standard
6	Hessian matrix — classification of critical points	Standard

Topic priority for exam preparation:

Lectures 6 and 7 carry the most weight. Expect at least two or three problems from these sections.
Lecture 8 (linearization, backpropagation) is lighter. Give it less time if you are short.
Practice Taylor series and Hessian problems. These are reliably tested.

For Taylor series problems (2D):

Exam note —

Taylor recipe: (1) List all required partial derivatives up to the stated degree. (2) Compute each derivative and evaluate at the given anchor point. (3) Plug into the two-variable Taylor formula, respecting the binomial coefficients and factorial denominators. (4) Show every derivative step —

Skipping intermediate derivatives costs marks. The anchor point determines whether you write $x$ or $x - a$ . If the anchor is $(0, 0)$ , $h = x$ and $k = y$ . Otherwise, every term uses $(x - a)$ and $(y - b)$ .

For Hessian / extrema problems (2D):

Exam note —

Extrema recipe: (1) Set $\nabla f = (0, 0)$ and solve the system to find all critical points. (2) Compute $f_{xx}, f_{y y}, f_{x y}$ —

These may be functions of $(x, y)$

So evaluate them at each critical point. (3) Compute $D = f_{xx} f_{y y} - (f_{x y})^{2}$ . (4) Classify using the table: $D > 0, f_{xx} > 0$ → min;

$D > 0, f_{xx} < 0$ → max;

$D < 0$ → saddle;

$D = 0$ → inconclusive. (5) Report the function value at each extremum —

Not just the classification. If there are multiple critical points, classify all of them.

General exam advice:

Companion documents are the best study resource. They are concise — a few pages each. Recordings take much longer to review. Use the documents.
Practice from the exam preparation problem set uploaded to the course platform.
Work through the full worked examples in Sections 10.7 and 10.8 until the four-step recipes become automatic — these are the two most reliably tested problems from this lecture.

---

Key Industry Applications

The concepts in this lecture are not just exam material — they are the mathematical backbone of modern machine learning systems.

Gradient descent is the workhorse of all neural network training. Every time you call model.fit() in Keras or optimizer.step() in PyTorch, the framework computes a gradient via backpropagation and updates weights by stepping opposite to it. The first-order Taylor approximation is the mathematical guarantee that a small step opposite the gradient reduces the loss. Stochastic gradient descent (SGD) and its variants (Adam, RMSProp, Adagrad) are used in every production ML system from recommendation engines to large language models.

Hessian-based optimization powers second-order methods like Newton's method and quasi-Newton approximations (L-BFGS). These are standard in logistic regression, support vector machines

And any convex optimization problem where the Hessian is tractable. L-BFGS is the default optimizer in scikit-learn's LogisticRegression and is widely used in maximum entropy models, conditional random fields

And classical structure-from-motion in computer vision. The Hessian (or its approximation) informs both the direction and the step size, enabling quadratic convergence near the optimum —

Dramatically faster than gradient descent's linear convergence.

Backpropagation and computational graphs are the engine behind all deep learning frameworks —

TensorFlow, PyTorch, JAX, Flux.jl. These libraries build a computational graph from your code, then apply reverse-mode automatic differentiation (backprop) to compute gradients. The same technique powers differentiable renderers (for inverse graphics), differentiable physics simulators (Brax, DiffTaichi —

Used in robotics for optimizing controllers through simulated dynamics)

And neural ODEs. Financial institutions use adjoint algorithmic differentiation (AAD) —

The same reverse-mode AD —

To compute the "Greeks" (market risk sensitivities) for complex derivative portfolios.

Saddle points are a central research topic in deep learning theory. In high dimensions, the loss surface is overwhelmingly dominated by saddle points, not local minima. The probability that a random critical point is a local minimum decays exponentially with dimension. This is why gradient descent with momentum —

Which can "roll through" shallow saddles —

Works so well empirically

And why pure Newton methods fail in deep learning without significant damping or trust-region modifications. Understanding saddle points is the starting point for grasping why training large networks is possible at all.

Linearization is the foundational technique in control theory and state estimation. Extended Kalman Filters (used in every GPS receiver, drone autopilot

And self-driving car perception stack) linearize nonlinear motion and observation models around the current state estimate at each time step. Small-signal analysis in circuit design linearizes transistor behavior around a DC operating point. Model predictive control (MPC) for autonomous vehicles repeatedly linearizes the vehicle dynamics to solve a quadratic program in real time. Every "linear regime" or "small-signal model" in engineering is linearization in the wild —

The degree-1 Taylor approximation, applied at scale.

---

MFML Lecture 10 notes · Taylor Series and Hessian-Based Optimization

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-10

Sections Breakdown

110.1 Rolle's Theorem

The flat-spot guarantee and why its three conditions are chosen that way.

210.2 Mean Value Theorem

Average slope equals an instantaneous slope; the bridge to Taylor series.

310.3 Taylor Series - The Full Intuition

Position, slope, curvature, factorials, and higher-order terms in one variable.

410.4 Two-Variable Taylor Series and the Gradient

Extending Taylor to surfaces, the gradient, and the second-order expansion.

510.5 The Hessian Matrix

Collecting second derivatives and what the symmetric matrix represents.

610.6 Classifying Critical Points Using the Hessian

The determinant test that separates minima, maxima, and saddles.

710.7 Worked Example - Two-Variable Taylor Series

A full degree-3 expansion of e^x sin y about the origin.

810.8 Worked Example - Extrema Using the Hessian

Finding and classifying critical points step by step.

910.9 Linearization

The degree-1 approximation and where it stays accurate.

1010.10 Backpropagation and Computational Graphs

Reverse-mode differentiation and the chain rule on a graph.

11Exam Guidance Summary

Question distribution and the two reliable four-step recipes.

12Key Industry Applications

Gradient descent, Hessian methods, backpropagation, saddles, and linearization in practice.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Rolle's Theorem

Must-know: A function that starts and ends at the same height must flatten somewhere between. This is the stepping stone to the Mean Value Theorem.

$f^{'} (c) = 0 for some c \in (a, b)$

⚠️ Top pitfall: Forgetting that the endpoint equality $f (a) = f (b)$ is required. Without it the theorem does not apply.

Self-check: Give one example where $f (a) \neq = f (b)$ yet a zero slope still occurs in $(a, b)$ .

Connects to: Mean Value Theorem, Taylor series foundations.

Mean Value Theorem

Must-know: Somewhere between the endpoints, the instantaneous slope equals the average slope across the interval. This bridges Rolle's theorem to Taylor series.

$f^{'} (c) = \frac{f ( b ) - f ( a )}{b - a} for some c \in (a, b)$

⚠️ Top pitfall: Reading the conclusion as if it held at every point. The slope matches only at one point $c$ .

Self-check: For $f (x) = x^{2}$ on $[0, 2]$ , find the value of $c$ the theorem promises.

Connects to: Rolle's theorem, Taylor remainder term.

One-Variable Taylor Series

Must-know: Near a point, a smooth function is built from its value, slope, curvature, and higher derivatives. Each term carries a factorial in the denominator.

$f (a + h) = f (a) + h f^{'} (a) + \frac{h ^{2}}{2 !} f^{''} (a) + \frac{h ^{3}}{3 !} f^{(3)} (a) + \dots$

⚠️ Top pitfall: Dropping the factorial denominators. They undo the chain of factors that repeated differentiation produces.

Self-check: Write the degree-2 Taylor polynomial of $e^{x}$ about $x = 0$ and state where it stays accurate.

Connects to: Two-variable Taylor series, linearization.

Two-Variable Taylor Series and the Gradient

Must-know: In two variables the first-order change is the dot product of the gradient with the step. The degree-2 term adds three curvature pieces, with a factor of 2 on the mixed term.

$f (a + h, b + k) \approx f (a, b) + h f_{x} + k f_{y} + \frac{1}{2 !} (h^{2} f_{xx} + 2 hk f_{x y} + k^{2} f_{y y})$

⚠️ Top pitfall: Writing $hk f_{x y}$ instead of $2 hk f_{x y}$ . The mixed derivative contributes twice by Clairaut's theorem.

Self-check: Explain why the mixed term carries the coefficient 2 rather than 1.

Connects to: Hessian matrix, gradient descent, worked Taylor example.

Two-Variable Taylor Worked Example

Must-know: The exam recipe is mechanical. List every derivative to the stated degree, evaluate at the anchor, then plug into the formula. Show each derivative step.

$e^{x} sin y \approx y + x y + \frac{1}{2} x^{2} y - \frac{1}{6} y^{3}$

⚠️ Top pitfall: Forgetting to evaluate derivatives at the anchor. A zero coefficient means that term vanishes, not that you skip it.

Self-check: At the origin, which single term gives the leading linear rise in the $y$ -direction?

Connects to: Two-variable Taylor series, Hessian classification.

The Hessian Matrix

Must-know: The Hessian collects all second partial derivatives into a symmetric matrix. It organises the curvature that the Taylor series captures at a critical point.

$H = [f_{xx} f_{x y} f_{x y} f_{y y}]$

⚠️ Top pitfall: Thinking the Hessian is not symmetric. For smooth functions $f_{x y} = f_{y x}$ , so the off-diagonal entries match.

Self-check: Why must the Hessian be symmetric when second derivatives are continuous?

Connects to: Critical point classification, Newton's method.

Classifying Critical Points with the Hessian

Must-know: Set the gradient to zero to find candidates, then use the Hessian determinant $D$ . The sign of $D$ and of $f_{xx}$ decides the type.

$D = f_{xx} f_{y y} - (f_{x y})^{2}, D > 0, f_{xx} > 0 \Rightarrow min, D < 0 \Rightarrow saddle, D = 0 \Rightarrow inconclusive$

⚠️ Top pitfall: Treating $D = 0$ as a saddle. It is inconclusive, so use higher-order terms or test nearby points.

Self-check: For $f (x, y) = x^{2} - y^{2}$ , classify the critical point at the origin.

Connects to: Hessian matrix, two-variable Taylor series, extrema worked example.

Linearization

Must-know: Linearization is the degree-1 Taylor polynomial. It replaces a curved function with its tangent line near a point and underpins many engineering models.

$f (x) \approx f (a) + f^{'} (a) (x - a)$

⚠️ Top pitfall: Using the linearization far from the anchor. It is a local tool and the error grows with distance.

Self-check: Give the linearization of $x$ about $x = 1$ and state where it stays accurate.

Connects to: One-variable Taylor series, Extended Kalman Filters.

Backpropagation and Computational Graphs

Must-know: A neural network is a graph of operations. Backpropagation applies the chain rule backwards to compute each weight's gradient in one pass.

$\frac{\partial L}{\partial w} = paths \sum \frac{\partial L}{\partial y} \frac{\partial y}{\partial w}$

⚠️ Top pitfall: Believing backprop invents new math. It is the chain rule applied to a graph, nothing more.

Self-check: Why does reverse-mode automatic differentiation need only one backward pass for all parameters?

Connects to: Gradient descent, first-order Taylor term.

Previous LectureLecture 09 · Gradients, the Jacobian, and Taylor Series

Next LectureLecture 11 · Gradient Descent

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Taylor Series and Hessian-Based Optimization

10.1 Rolle's Theorem

10.1.1 Definition and Intuition

10.1.2 Why the Conditions Are Chosen That Way

10.1.3 Worked Example

10.1.4 Assumptions & Scope

10.1.5 Visual Intuition

10.1.6 Pitfalls

10.1.7 Student Questions and Answers

10.1.8 Recap + Bridge

10.2 Mean Value Theorem

10.2.1 Definition and Intuition

10.2.2 Comparison with Rolle's Theorem

10.2.3 Geometric Interpretation

10.2.4 Worked Example

10.2.5 From MVT to Taylor Series — The Bridge

10.2.6 Assumptions & Scope

10.2.7 Visual Intuition

10.2.8 Pitfalls

10.2.9 Recap + Bridge

10.3 Taylor Series — The Full Intuition

10.3.1 Position, Slope, Curvature, and Beyond

10.3.2 The Full 1D Formula

10.3.3 Where the Factorials Come From

10.3.4 Why Higher-Order Terms Matter

10.3.5 Worked Examples from the Companion Document

10.3.6 Assumptions & Scope

10.3.7 Visual Intuition

10.3.8 Pitfalls

10.3.9 Student Questions and Answers

10.3.10 Pedagogical Insight

10.3.11 Recap + Bridge

10.4 Two-Variable Taylor Series and the Gradient

10.4.1 The Hilly Landscape

10.4.2 Symbol Registry

10.4.3 The First-Order Expansion

10.4.4 Gradient Form

10.4.5 The Second-Order Expansion

10.4.6 The Full Two-Variable Taylor Formula (to Degree 3)

10.4.7 At a Critical Point — What the Taylor Expansion Reveals

10.4.8 Assumptions & Scope

10.4.9 Visual Intuition

10.4.10 Pitfalls

10.4.11 Student Questions and Answers

10.4.12 Gradient Descent Connection

10.4.13 Recap + Bridge

10.5 The Hessian Matrix

10.5.1 From Taylor Expansion to Matrix Form

10.5.2 Symbol Registry

10.5.3 What the Hessian Represents

10.5.4 Connection to Definiteness

10.5.5 Derivation: How the Hessian Connects to the Taylor Second-Order Term

10.5.6 The Hessian in n Variables

10.5.7 Alternative Notation

10.5.8 Hessian is Always Symmetric

10.5.9 Worked Example — Build a Hessian

10.5.10 Assumptions & Scope

10.5.11 Visual Intuition

10.5.12 Pitfalls

10.5.13 Recap + Bridge

10.6 Classifying Critical Points Using the Hessian

10.6.1 The Classification Rules

10.6.2 Symbol Registry for Classification

10.6.3 Intuition for Each Case

10.6.4 Intuitive Explanation of the Determinant Rule

10.6.5 One-Variable Analogy

10.6.6 Why fxx​ Suffices When D>0

10.6.7 Worked Example — Saddle Point Detection

10.6.8 Assumptions & Scope

10.6.9 Visual Intuition

10.6.10 Pitfalls

10.6.11 Comparison: 1D vs 2D Classification

10.6.12 Student Questions and Answers

10.6.13 The Saddle Point Problem in Deep Learning

10.6.14 Recap + Bridge

10.7 Worked Example — Two-Variable Taylor Series

10.7.1 Problem Statement

10.7.2 Step 1 — List All Required Derivatives

10.7.3 Step 2 — Compute Each Derivative and Evaluate at (0, 0)

10.7.4 Step 3 — Assemble

10.5.6 The Hessian in $n$ Variables

10.6.6 Why $f_{xx}$ Suffices When $D > 0$