Mathematical Foundations for Machine Learning

Singular Value Decomposition and Vector Calculus

Published: 2026-07-10

Level: postgraduate

Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Singular Value Decomposition (SVD) — introduced in Lecture 7 (section 7.7), which this lecture develops fully with a numerical worked example.
Eigen Decomposition / Diagonalization — covered in Lecture 6 (section 6.3) and Lecture 7 (section 7.6); SVD generalizes it to rectangular matrices.
Eigenvalues & Eigenvectors — Lecture 6 (section 6.3); needed because right singular vectors are eigenvectors of $A^{⊤} A$ .
Determinants & Characteristic Equation — Lecture 6/7; used to find eigenvalues via $det (A^{⊤} A - λ I) = 0$ .
Rank, Null Space & Rank-Nullity Theorem — Lecture 7 (sections 7.3); the cap $rank (A) \leq min (m, n)$ reappears here.
Matrix Decompositions (Cholesky, Eigen) — Lecture 7 (sections 7.4–7.6); SVD is the third major decomposition in that sequence.

Singular Value Decomposition and Vector Calculus

8.1 Singular Value Decomposition — The Rectangular Reality

8.1.1 Definition and Motivation

Hook. Eigen decomposition diagonalizes a matrix — but only if the matrix is square. What happens when your data has 10,000 users and 20 features? You need a tool that works on rectangles. That tool exists. It is the singular value decomposition.

Intuition. Think of a passport photo. It is a rectangle — taller than it is wide. If you try to force it through a square-cutter (eigen decomposition), it breaks. SVD is the rectangle-cutter. It slices any matrix into three clean pieces, no matter the shape.

The pieces are always the same kind: two rotation-like matrices and one scaling matrix. One rotation lines things up, the scaling stretches or squishes, and the second rotation puts everything back into the original coordinate frame.

A singular value decomposition (SVD) breaks any matrix — square or not — into three constrained matrices. You met eigen decomposition already. It has one hard limit. It only works on square matrices, where rows equal columns.

Real data is almost never square. A Netflix-style dataset has millions of users (rows) but only a few features per user (columns). A phone photo is a 1920 × 1080 grid of pixels — a non-square matrix. Tweet collections hold millions of messages but only a few hundred feature columns. In all these cases, eigen decomposition cannot apply. You need a tool that works on rectangles.

That tool is SVD. It is the natural sequel to eigen decomposition. It is the most important answer to one question: how do you decompose a non-square matrix?

8.1.2 Why Eigen Decomposition Falls Short

Eigen decomposition assumes the matrix is square: $rows (A) = cols (A)$ . Number of rows of $A$ must equal the number of columns. That is the hard assumption.

Most working data breaks this rule. Structured-data problems tend to be long. They have many more rows than columns. Example — predicting who subscribes next month. The user roster runs to millions. The feature set (age, gender, genre preferences) is a few dozen. The matrix is tall, not square. So eigen decomposition dies there. SVD does not.

8.1.3 Long Format vs Wide Format Data

SVD handles both long and wide non-square matrices. The computation strategy depends on which dimension is smaller.

Long format — number of rows is much greater than number of columns. This is the common case for tables of users, transactions, documents.
Wide format — number of columns is greater than number of rows. Rare, but it occurs.

For a long-format matrix (rows ≫ cols), start the SVD computation from the smaller side, $A^{⊤} A$ . For a wide-format matrix (cols ≫ rows), start from $A A^{⊤}$ . Always pick the smaller square to keep the algebra short.

8.1.4 Symbol Registry

$A \in R^{m \times n}$ — the data matrix being decomposed, with $m$ rows and $n$ columns. This is the standard notation: $m$ is always rows, $n$ is always columns.
$U \in R^{m \times m}$ — matrix of left singular vectors, square and orthogonal.
$Σ \in R^{m \times n}$ — rectangular diagonal matrix of singular values, same shape as $A$ .
$V^{⊤} \in R^{n \times n}$ — matrix of right singular vectors (transposed), square and orthogonal.
$σ_{i}$ — the $i$ -th singular value — scalar $\geq 0$ , ordered $σ_{1} \geq σ_{2} \geq \dots \geq 0$ .
$λ_{i}$ — eigenvalue of $A^{⊤} A$ (or $A A^{⊤}$ ) — scalar.

The notation is standard across textbooks (Strang, Aggarwal): $A$ is $m \times n$ , $U$ is $m \times m$ , $Σ$ is $m \times n$ , and $V^{⊤}$ is $n \times n$ .

8.1.5 Assumptions and Scope

Scope. SVD applies to any real-valued matrix $A \in R^{m \times n}$ — no square requirement, no full-rank requirement, no symmetry requirement. The decomposition always exists.

Assumptions. Singular values are always non-negative ( $σ_{i} \geq 0$ ). This holds because they come from the eigenvalues of $A^{⊤} A$ , which is positive semidefinite. If your matrix has negative eigenvalues in $A^{⊤} A$ , something is wrong with the computation, not with SVD itself.

When it breaks. SVD breaks only at the computational level. If $A$ is ill-conditioned, the numerical computation can become unstable. But mathematically, the decomposition always exists.

8.1.6 Visual Intuition

Picture a rectangular sheet of grid paper. The grid lines are the original coordinate axes. Now apply the matrix $A$ to every point on that sheet. The grid warps — lines bend, stretch, or rotate.

SVD says this warping always breaks into three clean stages. First, rotate the sheet so the main stretch directions line up with the axes ( $V^{⊤}$ ). Then stretch or squish along those axes ( $Σ$ ). Finally, rotate the warped sheet one more time into its final orientation ( $U$ ). Every matrix, no matter how messy, decomposes into rotate → scale → rotate. The singular values are exactly how much each axis stretches or shrinks.

8.1.7 Pitfalls

Assuming eigen decomposition works everywhere. When you see a non-square matrix, eigen decomposition is off the table. Many students try it anyway — it does not work.
Confusing the dimension convention. The notation $m \times n$ is universal. Some sources swap $m$ and $n$ . Always check which convention a problem uses by looking at the matrix dimensions first. In this course, $A \in R^{m \times n}$ means $m$ rows, $n$ columns.
Thinking SVD is only for compression. SVD is foundational to linear algebra itself. Compression is one application. It also reveals the four fundamental subspaces, solves least-squares problems, and computes matrix pseudoinverses.

8.1.8 Recap and Bridge

SVD decomposes any matrix — rectangular or square — into $A = U Σ V^{⊤}$ . Eigen decomposition fails on non-square matrices; SVD steps in. Next, you will learn what rank means and why it limits how many singular values can be non-zero.

8.1.9 Real-World and Domain Connection

SVD underpins dimensionality reduction everywhere in data science. When Netflix recommends movies, it does not store every user's every rating. It decomposes the giant user × movie matrix via SVD and keeps only the top singular values. That captures taste patterns without storing billions of entries. The same idea powers image compression and latent semantic analysis in NLP. It also powers modern PEFT methods like LoRA, which fine-tune large language models by decomposing weight matrices.

8.1.10 Student Questions and Answers

Q: Is reducing dimensions the same idea as multicollinearity?

A: Yes — highly related. Multicollinearity means features are near-linear combinations of each other. SVD exploits exactly that redundancy to shrink dimensions. When people say "low rank approximation," "decomposition," or "rank approximation," SVD is a first candidate.

8.2 Rank and Maximum Possible Rank

8.2.1 Definition

Hook. You have 1000 rows of data but only 2 columns. How many truly independent directions can you possibly have? Not 1000 — at most 2. That is the hard ceiling that rank enforces.

Intuition. Think of a bookshelf with 50 books (rows) but only 2 unique authors (columns). Even though you have 50 rows, the "content" boils down to 2 distinct styles. The rank is 2. No matter how many books you add, if they are all by the same two authors, the rank stays 2.

The analogy breaks when you have genuinely independent data. If each book had a unique author, the rank could go up. But here, columns are the limiting factor — just like in a data matrix.

The rank of a matrix is how many linearly independent rows or columns it has. The maximum possible rank is the smaller of the row count and the column count. Formally:

rank (A) \leq min (m, n) .

A 1000 × 2 matrix has maximum possible rank 2. Even with 1000 rows, only two columns exist. So you can have at most two independent directions.

8.2.2 Worked Intuition — Age vs Experience

Take a 1000-row dataset with two columns: age and years of experience.

Maximum possible rank: $min (1000, 2) = 2$ .

But age and experience often move together. Say experience $\approx$ age $-$ 22 from people starting work around 22. Then:

experience \approx 1 \cdot age - 22

This is a linear relationship — the second column is (roughly) the first column shifted by a constant. The two columns are linearly dependent, so the true rank drops to $1$ .

Sense check: Plot age on the x-axis, experience on the y-axis. The points hug a single line. The two features tell almost the same story. You can represent the same information with one column instead of two — reducing storage by half without losing signal.

8.2.3 When Correlation Reduces Rank

Experience may be a scaled version of age. Say experience $\approx k \cdot$ age, or experience $\approx$ age $-$ 22 from people starting work around 22. The two columns are then linearly related. The matrix rank falls from a possible 2 down to 1.

Now scale the idea up. With hundreds of columns, many overlap this way. Rank drops, and that is the door SVD walks through to compress data.

8.2.4 Assumptions and Scope

Scope. The rank bound $rank (A) \leq min (m, n)$ holds for every real matrix. It is a theoretical ceiling, not an empirical estimate.

What reduces actual rank. Linear dependence among rows or columns drives the actual rank below the maximum. One example is age nearly equaling experience minus a constant. Correlation does not guarantee reduced rank (nonlinear relationships do not count), but linear correlation does.

Full rank. If $rank (A) = min (m, n)$ , the matrix is said to have full rank. In the 1000 × 2 case, full rank means 2 independent columns. That is the best you can do.

8.2.5 Visual Intuition

Draw two axes: age on the x-axis (from 20 to 60) and experience on the y-axis (from 0 to 40). Every person is a dot. If age and experience are independent, dots scatter across the full 2D plane — rank is 2. If experience always equals age minus 22, every dot lands exactly on the line $y = x - 22$ . The dots collapse to a 1D line. The data still has 1000 dots, but the effective dimension — the rank — is 1. The second dimension carries no new information.

8.2.6 Pitfalls

Confusing rank with row count. A 1000 × 2 matrix can have rank at most 2, not 1000. Rank counts independent directions, not data points.
Thinking correlation always reduces rank. Only linear correlation drops rank. If age and experience have a perfect U-shaped relationship, the rank is still 2. Rank only cares about linear combinations.
Forgetting that rank is about both rows and columns. Row rank always equals column rank. You cannot have 3 independent rows and only 2 independent columns in the same matrix.

8.2.7 Recap and Bridge

Rank is the number of independent directions in your data, capped at $min (m, n)$ . Correlation drops rank — and that drop is exactly what SVD exploits for compression. Low rank approximation, next, is the art of safely throwing away the redundant directions.

8.2.8 Real-World and Domain Connection

Rank is the engine behind every recommender system. A movie-rating matrix with millions of users and thousands of films might have a true rank of 50. That means only 50 underlying "taste factors" explain all the ratings. SVD recovers those factors. Dimensionality reduction, feature selection, and compressed sensing all share one insight. Your data's rank is far smaller than its row count.

8.2.9 Student Questions and Answers

Q: Can experience be age minus a fixed quantity — like people starting work at 20 or 22?

A: Yes. That is a linear relationship too. When two features are linear combinations of each other (scaled, shifted, or combined), they reduce to one independent direction. The rank drops accordingly.

8.3 Low Rank Approximation

8.3.1 Intuition

Hook. You can throw away half your data and lose almost nothing. How? Because that half is redundant. Low rank approximation finds the redundancy and cuts it out.

Intuition. Think of a compressed JPEG photo. A 12-megapixel image boils down to a few hundred numbers that capture edges, colors, and textures. The rest is detail you cannot notice. A low rank approximation does the same thing for any matrix. Keep the big structural pieces, discard the noise, and the result looks nearly identical.

The mapping: the "big pieces" are the directions with large singular values. The "noise" is the directions with tiny singular values. You keep the former, zero out the latter, and the reconstructed matrix barely changes.

A low rank approximation takes a huge matrix and approximates it with a matrix of much lower rank. You drop redundant rows and columns and keep the essential structure. The data shrinks, but the signal survives.

The age/experience example made this concrete. Two columns collapsed to one because they were near-copies. SVD is one of the most important low-rank approximation methods. Wherever you see "decomposition," "rank approximation," or "approximation," SVD is a candidate.

8.3.2 Reducing Rows Loses Nothing When They Overlap

A natural worry: if you cut rows, do you lose data points? Only the redundant ones. If two users have identical age and salary rows, their plotted points sit on top of each other. The second row teaches the model nothing new. Removing it keeps every bit of information.

So low rank approximation trims redundant rows and redundant columns. It can also cut non-redundant rows. A small singular value proves a direction carries little information. That is the deeper point the worked example will show.

8.3.3 Formalizing: The SVD Connection

SVD gives the exact machinery for low rank approximation. After decomposing $A = U Σ V^{⊤}$ , the rank- $k$ approximation ( $k < r$ ) is:

A_{k} = i = 1 \sum k σ_{i} u_{i} v_{i}^{⊤}

You keep the first $k$ terms in the sum — the ones with the largest singular values — and drop the rest. The Eckart-Young theorem guarantees this is the best rank- $k$ approximation in the Frobenius norm: no other rank- $k$ matrix is closer to $A$ .

8.3.4 Assumptions and Scope

Scope. Low rank approximation works best when the singular values decay rapidly — a few large ones, then a sharp drop. If all singular values are comparable, no rank- $k$ approximation with $k < r$ will be good.

When it fails. If the data genuinely spans $r$ independent directions of equal importance, you cannot truncate without losing signal. The singular value spectrum (the list of $σ_{i}$ in descending order) tells you how much you can safely cut.

8.3.5 Visual Intuition

Plot the singular values of a typical data matrix as a bar chart. Put $σ_{1}$ through $σ_{r}$ on the x-axis, magnitude on the y-axis. A "good" spectrum looks like a ski slope: high at $σ_{1}$ , steep drop, then a long flat tail near zero. The tail is where you truncate. A "bad" spectrum looks like a gentle ramp — every bar is roughly the same height. Truncation there discards real signal. The shape of this bar chart is your decision tool.

8.3.6 Pitfalls

Blindly truncating to a fixed $k$ . The right $k$ depends on the singular value spectrum, not a magic number. One dataset may need $k = 2$ , another $k = 200$ .
Thinking "low rank = always safe." Low rank approximation is lossy compression. It trades fidelity for size. Whether the trade is worth it depends on how fast the singular values decay.
Confusing removal of rows with removal of data. When rows are redundant, removing them loses nothing. When rows carry independent signal, removing them loses information. The singular value tells you which case you are in.

8.3.7 Recap and Bridge

Low rank approximation keeps the big singular values and drops the small ones. It is the core of compression, noise removal, and parameter-efficient ML. SVD gives you the tools to decide exactly how much rank to keep.

8.3.8 Real-World: LoRA and Parameter-Efficient Fine-Tuning

Low rank approximation is not just for compressing old data. It now powers the latest large language models. LoRA (Low-Rank Adaptation) freezes a pre-trained model's giant weight matrices and inserts small, trainable low-rank matrices alongside them. These tiny add-ons capture the new task (e.g., medical Q&A) without touching the billions of original parameters. The result: fine-tuning that costs a fraction of the compute while matching full fine-tuning accuracy. Every article on "parameter-efficient fine-tuning" or "PEFT" rides on one idea. A weight update $Δ W$ can be re-expressed as a low-rank product $B A$ — pure SVD thinking applied to modern AI infrastructure.

8.3.9 Student Questions and Answers

Q: Does reducing dimensions mean we lose data without hurting model accuracy?

A: Yes — when the dropped dimensions are redundant. You remove rows that teach the model nothing new. The model's accuracy does not degrade. You are just cutting extra compute effort. The same information stays; the model runs faster and uses less memory. But this only holds when the dropped singular values are genuinely small. If they are comparable to the kept ones, accuracy will drop.

8.4 SVD Formal Definition and Dimensions

8.4.1 The Decomposition $A = U Σ V^{⊤}$

Hook. Every matrix — square, rectangular, tall, wide, full-rank, or rank-deficient — can be written as exactly three matrices multiplied together. One for rotation, one for scaling, one for rotation. Always.

Intuition. Take a lump of clay. You want to reshape it into a specific form. Any reshaping can be broken into steps. First, rotate the clay block so its main axes align with your hands ( $V^{⊤}$ ). Then stretch or squish along those axes to the right proportions ( $Σ$ ). Finally, rotate the result one more time to face the right direction ( $U$ ). SVD says every linear transformation — no matter how complicated — is exactly rotate → scale → rotate. The singular values are how hard each axis gets pulled.

Any rectangular matrix $A \in R^{m \times n}$ factors into three constrained matrices:

A = U Σ V^{⊤} .

$U \in R^{m \times m}$ — matrix of left singular vectors, square and orthogonal ( $U^{⊤} U = I$ ).
$Σ \in R^{m \times n}$ — rectangular diagonal matrix of singular values, same shape as $A$ . Only entries $Σ_{ii}$ with $i \leq min (m, n)$ can be non-zero. All singular values $σ_{i} \geq 0$ , ordered $σ_{1} \geq σ_{2} \geq \dots \geq 0$ .
$V^{⊤} \in R^{n \times n}$ — matrix of right singular vectors (transposed), square and orthogonal ( $V^{⊤} V = I$ ).

The notation $A \in R^{m \times n}$ is standard across textbooks (Strang, Aggarwal): $m$ always denotes rows, $n$ always denotes columns. The dimensions then follow mechanically: $U$ is $m \times m$ , $Σ$ is $m \times n$ , and $V^{⊤}$ is $n \times n$ .

Singular vectors generalize eigenvectors to rectangular matrices. The right singular vectors (columns of $V$ ) are the eigenvectors of $A^{⊤} A$ . The left singular vectors (columns of $U$ ) are the eigenvectors of $A A^{⊤}$ . The singular values are the square roots of the eigenvalues shared by both.

8.4.2 Geometric Roles of the Three Matrices

A matrix can scale, shear, or rotate its input space. SVD cleanly separates those effects:

$V^{⊤}$ — a rotation (or reflection) that aligns the input with the principal axes of $A$ .
$Σ$ — a scaling (stretch or squish) along those axes by the singular values.
$U$ — a rotation (or reflection) back into the output orientation.

This is the polar decomposition perspective: every matrix is an orthogonal transformation times a positive semidefinite scaling. SVD makes this explicit.

8.4.3 Worked Dimension Walk — A 1000 × 2 Matrix

Take 1000 people, each with age and experience (2 columns). So $A$ is 1000 × 2.

$U$ is 1000 × 1000 — left singular vectors. Only the first 2 columns matter (they pair with non-zero singular values). The remaining 998 columns span the nullspace of $A^{⊤}$ .
$Σ$ is 1000 × 2 — singular values. Only the first 2 entries can be non-zero. The rest are padding zeros.
$V^{⊤}$ is 2 × 2 — right singular vectors. All 4 entries can be non-zero.

The dimension you cannot ignore: $Σ$ has the same shape as $A$ . Even though $A$ has 1000 rows, only the first $min (1000, 2) = 2$ singular values can be non-zero.

8.4.4 Assumptions and Scope

Scope. SVD exists for every real matrix $A \in R^{m \times n}$ — no exceptions. The proof relies on the fact that $A^{⊤} A$ is always symmetric positive semidefinite, so it always has real, non-negative eigenvalues and orthonormal eigenvectors.

Complex case. SVD extends to complex matrices too, with conjugate transposes replacing regular transposes.

Uniqueness. Singular values are unique. Singular vectors are unique up to sign flips (multiplying a pair $u_{i}, v_{i}$ by $- 1$ simultaneously leaves the product unchanged).

8.4.5 Visual Intuition

Draw a unit circle in 2D. Applying a 2×2 matrix $A$ deforms it into an ellipse. The singular values are the lengths of the semi-axes of that ellipse. The right singular vectors (columns of $V$ ) are the directions of the original circle's radii that become the ellipse's axes. The left singular vectors (columns of $U$ ) are the directions those axes point to after transformation. The SVD is the geometric statement. Every linear map takes a circle to an ellipse, and the singular values describe that ellipse's shape.

8.4.6 Pitfalls

Swapping $U$ and $V$ . $U$ comes from $A A^{⊤}$ and has $m$ columns. $V$ comes from $A^{⊤} A$ and has $n$ columns. Their sizes differ when $m \neq = n$ . You cannot swap them.
Forgetting $Σ$ is rectangular. When $m \neq = n$ , $Σ$ is not square. It has the exact shape of $A$ . Filling a square diagonal matrix with singular values and extra zeros is wrong — only the leading $min (m, n)$ diagonal entries carry values.
Thinking singular vectors are eigenvectors of $A$ . They are eigenvectors of $A^{⊤} A$ and $A A^{⊤}$ , not of $A$ itself (unless $A$ is symmetric positive semidefinite).

8.4.7 Recap and Bridge

$A = U Σ V^{⊤}$ is the universal decomposition. $U$ and $V$ rotate, $Σ$ scales. Every matrix, every time. The geometric story — circle to ellipse — carries through to the next section, where you see the three steps in sequence.

8.4.8 Real-World and Domain Connection

The SVD's geometric interpretation — rotate, scale, rotate — is the foundation for 3D computer graphics and game engines. When a GPU renders a character, every vertex gets transformed by a series of matrix multiplications. SVD decomposes any such transform into its purest elements: orientation, stretch, orientation. Game physics engines also use SVD to compute the polar decomposition. This separates a deformation into pure rotation and pure stretch. It lets them simulate soft-body collisions realistically.

8.4.9 Symbol Registry

$A \in R^{m \times n}$ — the matrix being decomposed — tall or wide.
$U \in R^{m \times m}$ — left singular vectors — square, orthogonal.
$Σ \in R^{m \times n}$ — singular values — rectangular, diagonal, same shape as $A$ .
$V^{⊤} \in R^{n \times n}$ — right singular vectors (transposed) — square, orthogonal.
$m, n$ — row count and column count of $A$ — positive integers.

8.5 Geometric Intuition — Rotation, Scaling, Rotation

8.5.1 Align to the Direction of Most Variance

Hook. Your data has a shape — a direction where it spreads most. SVD rotates the whole dataset so that direction lines up with the x-axis. Now every axis carries independent information.

Intuition. Hold a pencil at a slant on a piece of paper. If you try to describe its position by how far it extends along the desk's edges, you need two numbers and a messy description. But if you first rotate the paper so the pencil lies flat along one edge, the description collapses to one simple number: its length. SVD finds the rotation that makes your data simplest to describe. The rotation is $V^{⊤}$ .

Picture age on the x-axis and experience on the y-axis after normalization. Both axes share a scale of roughly $- 1$ to $1$ . The data spreads most along a diagonal. That is the axis of maximum variance, also called maximum explainability.

The first step, $V^{⊤}$ , rotates the entire grid so the data's main spread lines up with the coordinate axes. You do not know in advance how much to rotate or where to stop. You find the rotation that exposes the axes of greatest variance. The rotation comes from the eigenvectors of $A^{⊤} A$ : each column of $V$ is an eigenvector of $A^{⊤} A$ , ordered by descending eigenvalue. The first column of $V$ points along the direction of maximum variance, the second along the next most, and so on.

8.5.2 Scaling Is Stretching — and Squishing

The second step, $Σ$ , scales each aligned axis. Scaling can stretch or squish. When one direction carries little variance, the corresponding singular value is small, and $Σ$ squishes that axis toward zero. A 3D point cloud that really lives in a flat 2D pancake gets squished to a 2D plane. The "depth" axis shrinks away.

The singular values tell you exactly how much each direction matters. $σ_{1}$ scales the direction of most variance, $σ_{2}$ scales the next, and so on. If $σ_{3} \approx 0$ , that direction carries almost no information — and can be dropped entirely, which is how PCA works.

8.5.3 Rotate Back, Land on Principal Axes

The third step, $U$ , rotates the squished data back to a natural orientation. You started with a 3D spread. You rotated, scaled, dropped the thin direction, and rotated back. You land on a lower-dimensional representation that keeps the high-variance structure.

The columns of $U$ are the principal axes in the output space — the directions along which the transformed data varies. Together, $U$ and $V$ tell you the relationship between input and output orientations.

8.5.4 Assumptions and Scope

Scope. The rotation-scaling-rotation interpretation is exact when the singular values are non-negative and the vectors are orthonormal. This holds for all real matrices.

Reflections possible. "Rotation" can include reflections (determinant $- 1$ ). $U$ and $V$ are orthogonal, not necessarily proper rotations. SVD does not distinguish the two — it separates the transformation into orthogonal × diagonal × orthogonal regardless.

8.5.5 Pitfalls

Thinking scaling means only stretching. Scaling includes compression. A singular value of 0.01 means "squish this axis to 1% of its size." The operation is still scaling — it just shrinks the axis instead of growing it.
Confusing "maximum variance" with "important." High variance directions carry lots of signal for many tasks, but not all. In anomaly detection, the low variance directions — the ones SVD would drop — sometimes contain the interesting outliers.
Forgetting that rotation angles come from eigenvectors. $V$ is not an arbitrary rotation. Its columns are the eigenvectors of $A^{⊤} A$ . You compute them, not guess them.

8.5.6 Recap and Bridge

SVD's three steps — rotate to align with variance ( $V^{⊤}$ ), scale ( $Σ$ ), rotate to output ( $U$ ) — are the geometric story behind every linear transformation. This story makes the SVD computation concrete in the next section.

8.5.7 Real-World and Domain Connection

The rotation-scaling-rotation decomposition is the mathematical engine behind Principal Component Analysis (PCA). When you run PCA(n_components=2) on a high-dimensional dataset, the algorithm computes the SVD of the centered data matrix. The first two right singular vectors (columns of $V$ ) give you the directions of maximum variance — exactly the principal components. Every dimensionality reduction pipeline built on PCA is, under the hood, running SVD and keeping the top singular vectors. This includes everything from gene expression analysis to customer segmentation.

8.5.8 Student Questions and Answers

Q: Is the second step scaling or shrinking?

A: It is scaling. The data shrinks as a side effect — axes with little information get squished. But the operation itself is scaling, which includes compressing some axes. The term "scaling" covers both stretching ( $σ > 1$ ) and squishing ( $0 < σ < 1$ ).

Q: Variance gives explainability of the data. SVD aligns data where variance is most — does that mean better explainability?

A: Yes. Where there is more variance, there is more explainability. Explainability here means how much of the data's total spread each direction captures. SVD rotates the data toward the direction of most change. The result carries more signal per retained dimension.

Q: Why is the initial rotation needed?

A: To align the data to the axis of maximum variance before scaling. You cannot usefully scale axes that do not line up with where the data spreads. If you scale first without rotating, the scaling affects the original features arbitrarily rather than the directions that actually matter.

8.6 Worked Example — SVD of a 3 × 2 Matrix

8.6.1 Setup and Strategy

Hook. Computing an SVD by hand sounds intimidating. But the process is actually a recipe: build the small square, eigen-decompose it, then recover the other half with a one-line identity.

Intuition. You have a rectangular door (3 feet by 2 feet) and need to measure its diagonal. Rather than measuring across the rectangle directly, you measure the smaller 2×2 square corner. Then use the Pythagorean theorem to get the rest. SVD works the same way. Decompose the smaller of $A^{⊤} A$ (2×2) or $A A^{⊤}$ (3×3). Then derive the other from the identity $A V = U Σ$ .

Given:

A = 110011, A \in R^{3 \times 2} .

The two columns, $(1, 1, 0)^{⊤}$ and $(0, 1, 1)^{⊤}$ , are not multiples of each other. So the columns are linearly independent. The first and third rows do relate — $row_{3}$ is the reverse of $row_{1}$ — but columns stay independent. This $A$ cannot be eigen decomposed (it is not square). You must SVD it.

Strategy: build a square matrix from $A$ , eigen decompose that, and recover the SVD factors. The two square options are $A^{⊤} A$ (2 × 2) and $A A^{⊤}$ (3 × 3). Always start from the smaller one.

Exam note: For a long-format matrix (rows ≫ cols), compute $A^{⊤} A$ first — it is the smaller square. For wide-format, compute $A A^{⊤}$ first. You can derive the larger from the smaller afterward.

8.6.2 Compute $A^{⊤} A$ (the Smaller Square)

A^{⊤} = [101101], A^{⊤} A = [101101] 110011 .

$(A^{⊤} A)_{11} = 1 \cdot 1 + 1 \cdot 1 + 0 \cdot 0 = 2.$
$(A^{⊤} A)_{12} = 1 \cdot 0 + 1 \cdot 1 + 0 \cdot 1 = 1.$
$(A^{⊤} A)_{21} = 0 \cdot 1 + 1 \cdot 1 + 1 \cdot 0 = 1.$
$(A^{⊤} A)_{22} = 0 \cdot 0 + 1 \cdot 1 + 1 \cdot 1 = 2.$

So:

A^{⊤} A = [2112] .

This matrix is symmetric. Eigenvectors of a symmetric matrix are orthogonal. You will use that shortly.

8.6.3 Eigenvalues of $A^{⊤} A$

Solve $det (A^{⊤} A - λ I) = 0$ :

det [2 - λ 1 1 2 - λ] = (2 - λ)^{2} - 1 = 0.

Expand:

4 - 4 λ + λ^{2} - 1 = 0 ⟹ λ^{2} - 4 λ + 3 = 0.

Factor:

(λ - 3) (λ - 1) = 0 ⟹ λ_{1} = 3, λ_{2} = 1.

These are the two eigenvalues of $A^{⊤} A$ . Both are non-negative, as expected for a symmetric positive definite matrix. The singular values will come from these.

8.6.4 Eigenvectors of $A^{⊤} A$ — the $V$ Matrix

For $λ_{1} = 3$ , solve $(A^{⊤} A - λ I) x = 0$ :

[2 - 3 1 1 2 - 3] [x_{1} x_{2}] = [- 1 1 1 - 1] [x_{1} x_{2}] = [00] .

Both rows reduce to $- x_{1} + x_{2} = 0$ , so $x_{1} = x_{2}$ . Pick $x_{2} = 1$ ; the eigenvector is $(1, 1)^{⊤}$ .

Normalize: length $= 1^{2} + 1^{2} = 2$ . Unit eigenvector:

v_{1} = [1/ 2 1/ 2] .

For $λ_{2} = 1$ :

[2 - 1 1 1 2 - 1] [x_{1} x_{2}] = [1111] [x_{1} x_{2}] = [00] .

Both rows give $x_{1} = - x_{2}$ . Pick $x_{2} = 1$ ; the eigenvector is $(1, - 1)^{⊤}$ . Normalize:

v_{2} = [1/ 2 - 1/ 2] .

The right singular vectors form $V^{⊤}$ . Largest singular value first:

V^{⊤} = [v_{1}^{⊤} v_{2}^{⊤}] = [1/ 2 1/ 2 1/ 2 - 1/ 2] .

Check orthogonality: $v_{1} \cdot v_{2} = \frac{1}{2} - \frac{1}{2} = 0$ . The matrix is orthogonal.

8.6.5 The Singular Value Matrix $Σ$

The singular values are the square roots of the eigenvalues of $A^{⊤} A$ :

σ_{i} = λ_{i} .

Since $A^{⊤} A$ is positive semidefinite, all eigenvalues are $\geq 0$ , so all singular values are real and non-negative. For this problem:

σ_{1} = 3, σ_{2} = 1 = 1.

The third singular value $σ_{3}$ is 0 because $A$ has rank 2 and only $min (3, 2) = 2$ singular values can be non-zero. $Σ \in R^{3 \times 2}$ holds only 2 non-zero diagonal entries. Place them in descending order:

Σ = 300010 .

8.6.6 Computing $U$ — the $A V = U Σ$ Identity

You could build $A A^{⊤}$ (3 × 3), find its three eigenvalues and three eigenvectors, and assemble $U$ from them. That is long and error prone. There is a shortcut.

Start from $A = U Σ V^{⊤}$ and multiply both sides by $V$ . Since $V^{⊤} V = I$ :

A V = U Σ.

Column by column, for the $i$ -th column:

A v_{i} = σ_{i} u_{i} ⟹ u_{i} = \frac{A v _{i}}{σ _{i}} .

First column $u_{1}$ :

A v_{1} = 110011 [1/ 2 1/ 2] = 1/ 2 2/ 2 1/ 2 = 1/ 2 2 1/ 2 .

Divide by $σ_{1} = 3$ :

u_{1} = \frac{1}{6} \frac{2}{3} \frac{1}{6} .

Verify: $\frac{2}{3} = \frac{2}{3}$ . All three forms are equivalent.

Second column $u_{2}$ :

A v_{2} = 110011 [1/ 2 - 1/ 2] = 1/ 2 0 - 1/ 2 .

Since $σ_{2} = 1$ :

u_{2} = 1/ 2 0 - 1/ 2 .

Third column $u_{3}$ :

The shortcut $u_{i} = A v_{i} / σ_{i}$ fails when $σ_{i} = 0$ (division by zero). But $U$ is orthogonal, so its columns are mutually perpendicular. Find $u_{3}$ as the vector perpendicular to $u_{1}$ and $u_{2}$ using the cross product:

u_{3} = u_{1} \times u_{2} .

Compute:

u_{1} u_{1} \times u_{2} = 1/ 6 2 / 3 1/ 6, u_{2} = 1/ 2 0 - 1/ 2, = \frac{2}{3} \cdot (- \frac{1}{2}) - \frac{1}{6} \cdot 0 \frac{1}{6} \cdot \frac{1}{2} - \frac{1}{6} \cdot (- \frac{1}{2}) \frac{1}{6} \cdot 0 - \frac{2}{3} \cdot \frac{1}{2} = - \frac{1}{3} \frac{1}{3} - \frac{1}{3} .

Check the norm: $\frac{1}{3} + \frac{1}{3} + \frac{1}{3} = 1$ . It is already a unit vector.

u_{3} = - 1/ 3 1/ 3 - 1/ 3 .

Sign convention note. The source said "U3 = U2 × U1". Computing $u_{2} \times u_{1}$ gives $[1/ 3, - 1/ 3, 1/ 3]^{⊤} = - u_{3}$ . Both are valid since singular vectors are unique only up to sign. The product $U Σ V^{⊤}$ is unchanged if you flip the sign of both $u_{i}$ and $v_{i}$ together. In this course, use $u_{1} \times u_{2}$ for consistency with the right-hand rule.

The full SVD of $A = [110011]$ is:

A = U Σ V^{⊤} = \frac{1}{6} \frac{2}{3} \frac{1}{6} \frac{1}{2} 0 - \frac{1}{2} - \frac{1}{3} \frac{1}{3} - \frac{1}{3} 300010 [\frac{1}{2} \frac{1}{2} \frac{1}{2} - \frac{1}{2}] .

Sense check: Multiply the three matrices. $V^{⊤}$ rotates, $Σ$ scales and pads, $U$ rotates back. The product recovers $A$ exactly — and indeed $U Σ V^{⊤} = [110011]$ . The rank is 2, matching the two non-zero singular values $3$ and $1$ .

8.6.7 Symbol Registry

$v_{1}, v_{2}$ — right singular vectors (eigenvectors of $A^{⊤} A$ ) — $R^{2}$ , unit length.
$u_{1}, u_{2}, u_{3}$ — left singular vectors (eigenvectors of $A A^{⊤}$ ) — $R^{3}$ , unit length, mutually orthogonal.
$σ_{1} = 3, σ_{2} = 1, σ_{3} = 0$ — singular values — scalars, ordered descending.
$λ_{1} = 3, λ_{2} = 1$ — eigenvalues of $A^{⊤} A$ — scalars.
$I$ — identity matrix (here 2 × 2) — $R^{2 \times 2}$ .

8.6.8 Assumptions and Scope

Scope. This worked example shows the standard SVD path for a tall matrix ( $m > n$ ). Compute $A^{⊤} A$ , find its eigenvalues and eigenvectors, and set $σ_{i} = λ_{i}$ . Then recover $U$ from $A v_{i} = σ_{i} u_{i}$ . The cross-product trick for the zero-singular-value column works in $R^{3}$ . It does not generalize to higher dimensions. There you need Gram-Schmidt or the full eigendecomposition of $A A^{⊤}$ .

When eigenvalues are negative. $A^{⊤} A$ is always positive semidefinite, so eigenvalues are always $\geq 0$ theoretically. Numerical errors can produce tiny negative values. Treat them as zero.

8.6.9 Visual Intuition

The matrix $A$ maps $R^{2}$ into $R^{3}$ . Draw the unit circle in the input $R^{2}$ plane. $A$ sends that circle to an ellipse lying in a 2D subspace of $R^{3}$ . The semi-major axis of that ellipse has length $3$ and points along $u_{1}$ . The semi-minor axis has length 1 and points along $u_{2}$ . The ellipse is perfectly flat — the third direction $u_{3}$ gets zero stretch. That is why A has rank 2: the output lives on a 2D plane inside 3D space.

8.6.10 Pitfalls

Starting from the larger square. Computing $A A^{⊤}$ (3 × 3) first wastes time. Always pick $min (m, n)$ — compute the smaller of $A^{⊤} A$ and $A A^{⊤}$ .
Forgetting to normalize eigenvectors. Unnormalized eigenvectors produce unnormalized singular vectors. The decomposition still works algebraically, but the orthogonality property ( $U^{⊤} U = I$ ) fails, which breaks downstream uses.
Dividing by zero for $σ_{i} = 0$ . When a singular value is zero, $u_{i} = A v_{i} / σ_{i}$ fails. Use orthogonality (cross product, Gram-Schmidt) to find the missing column instead.
Using the wrong cross product order. $u_{1} \times u_{2}$ and $u_{2} \times u_{1}$ differ by a sign. Either is fine as long as you are consistent. The exam will accept either direction.

8.6.11 Recap and Bridge

The SVD recipe for a tall matrix: compute $A^{⊤} A$ , then find $λ_{i}$ and $v_{i}$ . Set $σ_{i} = λ_{i}$ , and recover $u_{i} = A v_{i} / σ_{i}$ . Fill zero-singular-value columns via cross product. This same recipe generalizes to any rectangular matrix.

8.6.12 Student Questions and Answers

Q: Can you explain how to calculate $Σ$ again from scratch?

A: Take the eigenvalues of $A^{⊤} A$ (3 and 1). Take the square root of each ( $3$ and 1). Arrange them along the diagonal of a matrix the same shape as $A$ (3 × 2), descending. Fill remaining slots with zeros.

Q: What if eigenvalues come out negative?

A: The square root would be imaginary. But $A^{⊤} A$ is positive semidefinite, so eigenvalues are always $\geq 0$ . If you get a negative eigenvalue, it is a rounding error — treat it as 0.

Q: What if there is only one eigenvalue?

A: Then $A^{⊤} A$ is 1 × 1 and $A$ has only one column. $V^{⊤}$ is 1 × 1, $Σ$ gets one non-zero entry, and the remaining singular value slots are zero. SVD still works.

Q: How do you compute $u_{3}$ ?

A: The formula $u_{i} = A v_{i} / σ_{i}$ breaks when $σ_{3} = 0$ . Since $U$ must be orthogonal, its columns are mutually perpendicular. Compute $u_{3} = u_{1} \times u_{2}$ (the cross product) and normalize if needed.

Q: Why are we doing $u_{1} \times u_{2}$ ?

A: From the identity $A V = U Σ$ , each column $u_{i} = A v_{i} / σ_{i}$ . But division by $σ_{3} = 0$ is undefined. The columns of $U$ are perpendicular by definition of SVD, so the cross product of the first two gives the third direction directly.

Q: Why normalize eigenvectors? Can we leave them unnormalized?

A: You can, but the solution is not unique — your $(1, 1)$ is someone else's $(4, 4)$ . Only unit vectors in an eigenvector's direction give a canonical decomposition. Normalize to get $U$ and $V$ orthogonal, which is required for the SVD to satisfy $U^{⊤} U = I$ and $V^{⊤} V = I$ .

8.6.13 Exam Notes

Exam note: A numerical SVD problem is highly likely on the mid-semester exam. Practice the full walkthrough. Compute $A^{⊤} A$ , find eigenvalues and eigenvectors, and build $Σ$ . Recover $U$ via $A v_{i} / σ_{i}$ , and use the cross product for zero-singular-value columns. Always start from the smaller square.

8.7 Low Rank Approximation by Truncating Singular Values

8.7.1 The Truncated Sum

Hook. A 3 × 2 matrix takes 6 numbers to store. Its rank-1 approximation takes only 5 numbers — and in many datasets, it captures 90% of the information. That is the power of truncation.

Intuition. Imagine you are packing for a trip. Your suitcase (the original matrix) is overstuffed. You decide to only bring the three heaviest items — they account for most of the weight anyway. Truncating singular values is the same idea. Keep the terms with the biggest $σ_{i}$ (heavy items) and drop the rest. The bag is lighter and still has everything that matters.

Any matrix factorized by SVD can be rewritten as a sum of rank-1 pieces:

A = i = 1 \sum r σ_{i} u_{i} v_{i}^{⊤} = σ_{1} u_{1} v_{1}^{⊤} + σ_{2} u_{2} v_{2}^{⊤} + \dots + σ_{r} u_{r} v_{r}^{⊤},

where $r$ is the rank of $A$ (the number of non-zero singular values). Each term $σ_{i} u_{i} v_{i}^{⊤}$ is a rank-1 matrix. Every row is a scalar multiple of $v_{i}^{⊤}$ , and every column is a scalar multiple of $u_{i}$ . A rank- $k$ approximation keeps the first $k$ terms and discards the rest.

8.7.2 Worked Example — Rank-1 Approximation

Keep only the first (largest) singular value. Drop everything else:

A \approx A_{1} = σ_{1} u_{1} v_{1}^{⊤} = 3 1/ 6 2 / 3 1/ 6 [1/ 2 1/ 2] .

Computing the outer product entry by entry. For row 1, column 1:

3 \cdot \frac{1}{6} \cdot \frac{1}{2} = 3 \cdot \frac{1}{12} = 3 \cdot \frac{1}{2 3} = \frac{1}{2} .

Row 1, column 2: same computation, gives $\frac{1}{2}$ .

Row 2, column 1:

3 \cdot \frac{2}{3} \cdot \frac{1}{2} = 3 \cdot \frac{2}{3} \cdot \frac{1}{2} = 1.

Row 2, column 2: same, gives $1$ .

Row 3, column 1 and 2: identical to row 1 — both are $\frac{1}{2}$ .

The full rank-1 approximation is:

A_{1} = \frac{1}{2} 1 \frac{1}{2} \frac{1}{2} 1 \frac{1}{2} .

Compare with the original:

A = 110011 .

The rank-1 approximation captures the dominant pattern: the middle row is twice the outer rows, and both columns have similar values. But it misses the asymmetry. The original has zeros in opposite corners (top-right and bottom-left), which the rank-1 approximation smears into uniform halves. That information lives in the dropped $σ_{2} u_{2} v_{2}^{⊤}$ term.

Sense check: adding back $σ_{2} u_{2} v_{2}^{⊤}$ should recover the original exactly. Compute $u_{2} v_{2}^{⊤}$ :

1/ 2 0 - 1/ 2 [1/ 2 - 1/ 2] = \frac{1}{2} 0 - \frac{1}{2} - \frac{1}{2} 0 \frac{1}{2} .

Add to $A_{1}$ :

\frac{1}{2} 1 \frac{1}{2} \frac{1}{2} 1 \frac{1}{2} + \frac{1}{2} 0 - \frac{1}{2} - \frac{1}{2} 0 \frac{1}{2} = 110011 = A .

The reconstruction is exact — exactly as the sum formula guarantees.

8.7.3 When Truncation Is Safe

When the first singular value dominates, the second term barely changes the matrix. Say $σ_{1}$ sits in the hundreds while $σ_{2}$ lands in tenths. Dropping the second term loses almost no information. The classic case: two columns are near-exact copies (extreme multicollinearity). Then $σ_{1}$ is huge and $σ_{2}$ is near 0, so a rank-1 cut keeps nearly all the data.

When $σ_{1}$ and $σ_{2}$ are comparable, dropping $σ_{2}$ hurts. Here $3 \approx 1.73$ versus $1$ — similar magnitude — so accuracy suffers visibly. The rank-1 approximation here is a poor one, as the worked example shows.

The singular value spectrum — the list of $σ_{i}$ in descending order — tells you how much information each term carries. If $σ_{1} ≫ σ_{2} ≫ σ_{3} \approx 0$ , truncation is safe. If all $σ_{i}$ are similar, truncation is destructive.

8.7.4 Assumptions and Scope

Scope. Low rank approximation via SVD truncation is exact when keeping all $r$ terms. It is optimal in the Frobenius norm for any fixed $k < r$ (Eckart-Young theorem). This means no other rank- $k$ matrix gets closer to $A$ .

When it fails. If singular values decay slowly, truncation is poor. If the data has no redundancy (each column is independent), rank reduction loses real signal, not just noise.

8.7.5 Visual Intuition

Picture the original matrix $A$ as a 2D grid of colored squares — 3 rows, 2 columns. The rank-1 approximation replaces that grid with one where row 1 and row 3 are identical light gray, and row 2 is a darker gray. The pattern is simpler but the original's distinctive zero entries are gone. Now add the rank-2 term back — the zeros pop into place. Each singular value adds one more layer of detail, like a printer adding one more color pass to a photo.

8.7.6 Pitfalls

Treating truncation as always safe. Only safe when the dropped singular values are negligible. Blindly truncating a matrix with a flat spectrum ruins the data.
Forgetting that rank-k approximation IS a matrix of rank k. Each term adds one more rank. Keep k terms, get exactly rank k. The rank of the approximation equals the number of singular values you keep.
Confusing the rank-1 term with the whole SVD. The sum of ALL r terms reconstructs A exactly. Truncation is optional — you only drop terms when you want compression or denoising.

8.7.7 Recap and Bridge

Truncating singular values gives the best rank- $k$ approximation of any matrix. The sum of rank-1 pieces reconstructs the original exactly when you keep all terms. Remove terms from the tail — the ones with the smallest $σ_{i}$ — and you compress without losing the dominant structure.

8.7.8 Real-World: Image Compression

The SVD truncation is the engine behind image compression. A 1920 × 1080 grayscale photo is a 1920 × 1080 matrix. Its SVD ranks the singular values — typically a steep drop after the first few hundred. Keep the top ~30% of singular values and the image looks nearly identical. The storage drops dramatically, from 1920×1080 numbers to $k \times (1920 + 1080 + 1)$ numbers. WhatsApp and similar apps use this to send photos that are kilobytes instead of megabytes, with no visible quality loss.

8.7.9 Student Questions and Answers

Q: Here $σ_{1} = 3$ and $σ_{2} = 1$ — comparable, not 1000 vs 2. So dropping $σ_{2}$ is risky here, right?

A: Yes — for this example truncating $σ_{2}$ would noticeably hurt, because $σ_{1}$ and $σ_{2}$ are not far apart. The truncation intuition is strongest when features are highly correlated. Then $σ_{1}$ dominates hugely and $σ_{2}$ is tiny, so the second term is nearly noise. That is the case you meet in principal component analysis.

Q: Reducing dimensions doesn't affect accuracy — that holds always?

A: Only when the dropped directions carry redundant or low-variance information. If the dropped singular value is comparable to the kept ones, accuracy degrades. The approximation trades accuracy for size. The singular value spectrum tells you how much you can safely trade.

8.8 Derivatives of Univariate Functions

8.8.1 Setup — A Single Variable Predicts Another

Hook. You know your salary at age 30 and at age 50. But what matters most is what happens between those ages — at exactly 32, does your salary jump or crawl? The derivative answers that.

Intuition. A speedometer shows your speed right now — not your average speed over the trip. A car that averages 50 km/h may have been doing 120 km/h when the camera flashed. The derivative is the speedometer of mathematics. It tells you the rate of change at one exact instant, not over a stretch.

The professor's own analogy: "You started your bike from A, reached B. Your average speed was 50 km/h. How can I get the challan? The traffic police did not care about your average speed. They only cared about your instantaneous speed when crossing that point."

A univariate function has one input. Think salary as a function of age: $y = f (x)$ , with $x$ as age and $y$ as salary. Fit a curve through the data. At any point $x = 32$ , you can ask two things. What is the salary there? And how fast does salary change as age nudges around 32?

The answer to "how fast" is the derivative. It is the rate of change of one variable with respect to another, measured exactly at a point.

8.8.2 The Formal Limit Definition

For $y = f (x)$ , nudge $x$ by a tiny amount $h$ . The function changes by $f (x + h) - f (x)$ . Divide by $h$ to get the average rate over that nudge. Then shrink $h$ to zero for the instantaneous rate:

\frac{d y}{d x} = h \to 0 lim \frac{f ( x + h ) - f ( x )}{h} .

That quotient is the derivative. The numerator is the change in the function. The denominator is the change in the input. The limit takes the change over an infinitely small nudge. It collapses the average rate into the instantaneous rate. The notation $d y / d x$ is read "the derivative of y with respect to x."

8.8.3 Worked Example — $f (x) = x^{2}$ at $x = 3$

Apply the limit definition:

f (x) \frac{d y}{d x} = x^{2}, = h \to 0 lim \frac{( 3 + h ) ^{2} - 3 ^{2}}{h} = h \to 0 lim \frac{9 + 6 h + h ^{2} - 9}{h} = h \to 0 lim \frac{6 h + h ^{2}}{h} = h \to 0 lim (6 + h) = 6.

At $x = 3$ , the derivative of $x^{2}$ is 6. That means a tiny nudge upward in $x$ (say from 3 to 3.001) increases $f (x)$ by about $6 \times 0.001 = 0.006$ . And indeed: $3.00 1^{2} = 9.006001 \approx 9.006$ . The derivative gave the right rate.

Sense check: The slope gets steeper as $x$ grows — $x^{2}$ is a parabola, and at $x = 3$ the tangent slope should be 6. The power rule ( $2 x$ ) confirms $2 \times 3 = 6$ .

8.8.4 Visual Intuition — The Tangent Line

Picture a hiker climbing a mountain whose height is $y$ and whose horizontal position is $x$ . The curve $y = f (x)$ traces the mountain's profile. The derivative at any point is the slope of the tangent line to the curve right there.

A steep uphill tangent means $d y / d x$ is large and positive — height increases fast as $x$ grows.
A flat tangent means $d y / d x = 0$ — the top (or bottom) of a hill.
A downhill tangent means $d y / d x$ is negative.

So $d y / d x$ is the slope of the curve at a single point. Slide $x$ along and watch the tangent line sweep along the curve — that moving slope is the derivative function.

8.8.5 Assumptions and Scope

Scope. The derivative exists at $x$ only if the limit exists — meaning the left-hand and right-hand limits agree. A sharp corner (like $∣ x ∣$ at $x = 0$ ) has no derivative.

Differentiability. If a function is differentiable at a point, it is also continuous there. But continuity does not guarantee differentiability — $∣ x ∣$ is continuous at 0 but not differentiable.

8.8.6 Pitfalls

Confusing average rate with instantaneous rate. The difference quotient $(f (x + h) - f (x)) / h$ without the limit gives average rate. You need $h \to 0$ for instantaneous rate.
Thinking $d y / d x$ is a fraction. It is a single symbol — the limit of a fraction. You can manipulate it like a fraction in the chain rule, but it is not literally a number divided by another number.
Forgetting the limit. Plugging in $h = 0$ directly gives $0/0$ — meaningless. The limit is essential.

8.8.7 Recap and Bridge

The derivative $d y / d x$ is the instantaneous rate of change — the speedometer reading. It is defined as the limit of the difference quotient as $h \to 0$ . Next, the power rule gives a shortcut for computing derivatives of $x^{n}$ without the limit.

8.8.8 Symbol Registry

$x$ — the input variable — scalar, real.
$y = f (x)$ — the output (function value) — scalar.
$h$ — the small nudge in $x$ — scalar, tends to 0.
$d y / d x$ — the derivative — instantaneous rate of change — scalar.

8.9 The Power Rule — Derivation

8.9.1 The Rule

Hook. Typing $lim_{h \to 0}$ every time you need a derivative is exhausting. The power rule is a one-step shortcut for the most common function family in mathematics. No limits needed — just multiply and drop.

Intuition. Think of $x^{n}$ as a stack of $n$ blocks. The derivative asks: when $x$ grows, how fast does the total volume grow? Each of the $n$ blocks contributes independently, and each block's contribution scales like $x^{n - 1}$ . So the derivative is $n \cdot x^{n - 1}$ — exactly $n$ copies of the smaller block.

The heuristic: "bring the exponent down as a multiplier, then reduce the exponent by one." In symbols: $x^{n}$ becomes $n x^{n - 1}$ .

For $y = x^{n}$ ,

\frac{d y}{d x} = n x^{n - 1} .

That is the power rule. Multiply by the exponent, then drop the exponent by one.

8.9.2 Proof via the Binomial Theorem

Start with the limit definition on $f (x) = x^{n}$ :

\frac{d y}{d x} = h \to 0 lim \frac{( x + h ) ^{n} - x ^{n}}{h} .

Expand $(x + h)^{n}$ by the binomial theorem:

(x + h)^{n} = (0 n) x^{n} h^{0} + (1 n) x^{n - 1} h^{1} + (2 n) x^{n - 2} h^{2} + \dots + (n n) x^{0} h^{n} .

The binomial coefficient:

(r n) = \frac{n !}{r ! ( n - r )!} .

Plug in for the first two coefficients:

(0 n) = \frac{n !}{0 ! ( n - 0 )!} = 1, (1 n) = \frac{n !}{1 ! ( n - 1 )!} = \frac{n \cdot ( n - 1 )!}{( n - 1 )!} = n .

Now substitute the expansion back into the limit:

\frac{( x + h ) ^{n} - x ^{n}}{h} = \frac{x ^{n} + n x ^{n - 1} h + ( 2 n ) x ^{n - 2} h ^{2} + \dots + h ^{n} - x ^{n}}{h} .

The $x^{n}$ and $- x^{n}$ cancel. Factor $h$ out of every remaining term:

= \frac{n x ^{n - 1} h + ( 2 n ) x ^{n - 2} h ^{2} + \dots + h ^{n}}{h} = n x^{n - 1} + (2 n) x^{n - 2} h + \dots + h^{n - 1} .

Now take $h \to 0$ . Every surviving term has at least one factor of $h$ except the first:

\frac{d y}{d x} = n x^{n - 1} .

That is the power rule, derived.

8.9.3 Worked Example — $f (x) = x^{5}$

Apply the power rule directly:

\frac{d}{d x} x^{5} = 5 x^{4} .

Check at $x = 2$ . The derivative says $5 \times 2^{4} = 5 \times 16 = 80$ . Using the limit definition for sanity: $(2 + 0.001)^{5} \approx 32.08008$ , so $Δ f /Δ x \approx (32.08008 - 32) /0.001 = 80.08 \approx 80$ . The power rule gives the right answer.

For $x^{5}$ at $x = 2$ , the power rule gives $5 \cdot 2^{4} = 80$ . A tiny nudge of $h = 0.001$ produces a function change of about $80 \times 0.001 = 0.080$ , which matches the direct calculation. The power rule works.

8.9.4 Assumptions and Scope

Scope. This derivation assumes $n$ is a positive integer. The power rule also holds for negative integers. It holds for fractions and real exponents too. But the proofs differ. The binomial theorem proof works only for positive integer $n$ .

When it fails. The function must actually be a pure power of $x$ . If $y = x^{n} + c$ , the derivative is still $n x^{n - 1}$ (constants vanish). But if $y = (x^{2} + x)^{3}$ , you need the chain rule, not the pure power rule.

8.9.5 Pitfalls

Forgetting to drop the exponent. $d / d x$ of $x^{3}$ is $3 x^{2}$ , not $3 x^{3}$ . The exponent decreases by exactly one.
Applying the power rule to the wrong base. The power rule works when $x$ is the base and $n$ is constant. For $a^{x}$ (exponential), the rule is different — do not use the power rule.
Mixing up power rule and chain rule. $(x^{3} + 1)^{5}$ is not $5 (x^{3} + 1)^{4}$ alone — you also need the derivative of the inside ( $3 x^{2}$ ). The pure power rule applies only to $x^{n}$ .

8.9.6 Recap and Bridge

The power rule — bring down the exponent, reduce by one — is the workhorse of differentiation. It was derived from the limit definition by expanding $(x + h)^{n}$ and letting $h \to 0$ . Next, the product rule handles the derivative of two functions multiplied together.

8.9.7 Symbol Registry

$x$ — the input variable — scalar, real.
$y = f (x) = x^{n}$ — the function — scalar.
$n$ — the exponent — scalar, a positive integer in this derivation.
$h$ — the small nudge in $x$ — scalar, tends to 0.
$(r n)$ — binomial coefficient — scalar.
$d y / d x$ — the derivative (instantaneous rate of change) — scalar.

8.10 Product Rule

8.10.1 The Rule

Hook. A single function is easy to differentiate. But what about $x sin x$ ? Two functions tangled together. The product rule untangles them.

Intuition. Two roommates share an apartment. When the rent goes up, both feel it — but in different ways. The product of two functions $u (x) \cdot v (x)$ is like that apartment. If $x$ changes, both $u$ and $v$ adjust. The total change is the sum of two pieces. First: ( $u$ 's change) × ( $v$ 's current value). Second: ( $v$ 's change) × ( $u$ 's current value). You decorate one at a time while the other stays put.

When you differentiate a product of two functions of $x$ , say $z = u (x) v (x)$ , differentiate each factor in turn and combine:

\frac{d z}{d x} = \frac{d u}{d x} v + \frac{d v}{d x} u .

Take the derivative of one factor, keep the other as is. Then swap the two. Add the results.

8.10.2 Worked Example — $z = x sin x$

Set $u = x$ and $v = sin x$ . Apply the product rule:

\frac{d z}{d x} = \frac{d u}{d x} v + \frac{d v}{d x} u .

Compute each piece:

\frac{d u}{d x} = \frac{d}{d x} (x) = 1, \frac{d v}{d x} = \frac{d}{d x} (sin x) = cos x .

Substitute:

\frac{d z}{d x} = 1 \cdot sin x + cos x \cdot x = sin x + x cos x .

The derivative of $z = x sin x$ is $sin x + x cos x$ .

Sense check at $x = 0$ : $sin 0 + 0 \cdot cos 0 = 0 + 0 = 0$ . The function $x sin x$ is flat at the origin (it touches the x-axis). The derivative should vanish there — and it does.

8.10.3 Extension — Product Rule with Chain Rule

A function like $x^{2} e^{s i n x}$ follows the same pattern. Set $u = x^{2}$ , $v = e^{s i n x}$ . The product rule gives:

\frac{d z}{d x} = 2 x \cdot e^{s i n x} + x^{2} \cdot \frac{d}{d x} e^{s i n x} .

But the derivative of $e^{s i n x}$ itself needs the chain rule (next section). Reach for the product rule first, then the chain rule on the inner piece. Rules compose — apply them in the right order.

8.10.4 Assumptions and Scope

Scope. The product rule holds wherever both $u$ and $v$ are differentiable. It extends to three or more factors: $(uv w)^{'} = u^{'} v w + u v^{'} w + uv w^{'}$ .

Not limited to scalars. The product rule generalizes to inner products of vector-valued functions. It also covers matrix products and the cross product. You must take care with non-commutativity in the last two cases.

8.10.5 Pitfalls

Adding instead of summing with the right pairing. The formula is $u^{'} v + v^{'} u$ , not $u^{'} v^{'}$ and not $u^{'} + v^{'}$ . Each derivative pairs with the other function, not its own.
Forgetting that factors can be composite. If $v (x) = sin (x^{2})$ , then $v^{'}$ requires the chain rule first. The product rule is only step one — decompose fully.
Swapping the order unnecessarily. For scalar functions, $u^{'} v + v^{'} u = v^{'} u + u^{'} v$ . Order does not matter. But for matrix products, order matters — so always write $u^{'} v + v^{'} u$ in the exact order stated.

8.10.6 Recap and Bridge

The product rule — $(uv)^{'} = u^{'} v + v^{'} u$ — handles derivatives of multiplied functions by treating one factor at a time. Next, the chain rule handles nested functions like $(x^{3} + 3)^{8}$ .

8.10.7 Symbol Registry

$z = u (x) v (x)$ — a product of two functions of $x$ — scalar.
$u (x), v (x)$ — the two factor functions — scalar.
$u^{'} = d u / d x$ , $v^{'} = d v / d x$ — their derivatives — scalars.
$d z / d x$ — the derivative of the product — scalar.

8.11 Chain Rule

8.11.1 The Rule

Hook. How does changing $x$ affect a function buried two layers deep — like $u = (x^{3} + 3)^{8}$ ? You cannot just use the power rule alone. You need to follow the chain of dependencies.

Intuition. You control a thermostat ( $x$ ). The thermostat sets the furnace temperature ( $z$ ). The furnace heats the room ( $u$ ). To know how much turning the dial warms the room, you first ask: "how much does a 1° furnace increase warm the room?" ( $d u / d z$ ). Then: "how much does turning the dial change the furnace?" ( $d z / d x$ ). Multiply these two effects to get the total: $d u / d x = d u / d z \cdot d z / d x$ .

Each link in the chain is a simple derivative. The chain rule says: multiply the links.

A function can be built by composing two others. Let $u = u (z)$ and $z = z (x)$ . The derivative of $u$ with respect to $x$ is the product of the derivatives along the chain:

\frac{d u}{d x} = \frac{d u}{d z} \cdot \frac{d z}{d x} .

$u$ depends on $z$ , and $z$ depends on $x$ . To know how $u$ moves with $x$ , first measure how $u$ moves with $z$ . Then multiply by how $z$ moves with $x$ . You follow the dependency chain link by link.

8.11.2 Worked Example

Take a composite power:

u = (x^{3} + 3)^{8} .

Identify the layers. The inner function is $z = x^{3} + 3$ . The outer function is $u = z^{8}$ .

Apply the chain rule:

\frac{d u}{d x} = \frac{d u}{d z} \cdot \frac{d z}{d x} .

Outer derivative (power rule on $z^{8}$ ):

\frac{d u}{d z} = 8 z^{7} .

Inner derivative (power rule + constant, on $x^{3} + 3$ ):

\frac{d z}{d x} = 3 x^{2} + 0 = 3 x^{2} .

Multiply and substitute back $z = x^{3} + 3$ :

\frac{d u}{d x} = 8 z^{7} \cdot 3 x^{2} = 24 x^{2} (x^{3} + 3)^{7} .

The derivative of $u = (x^{3} + 3)^{8}$ is $24 x^{2} (x^{3} + 3)^{7}$ .

Sense check at $x = 0$ : If $x = 0$ , then $u = 3^{8} = 6561$ . The derivative there is $24 \cdot 0 \cdot 3^{7} = 0$ . The function is flat at the origin because $x^{2}$ factor vanishes — the input $x = 0$ is a critical point. ✓

Sense check at $x = 1$ : $z = 4$ , $u = 4^{8} = 65536$ . The derivative is $d u / d z \cdot d z / d x = 8 \cdot 4^{7} \cdot 3 \approx 8 \cdot 16384 \cdot 3 = 393216$ . ✓

8.11.3 Extending — Three or More Layers

The chain rule extends to any depth. For $u = u (w)$ , $w = w (z)$ , $z = z (x)$ :

\frac{d u}{d x} = \frac{d u}{d w} \cdot \frac{d w}{d z} \cdot \frac{d z}{d x} .

Each link multiplies. Real neural networks chain hundreds of layers this way — that is backpropagation.

8.11.4 Assumptions and Scope

Scope. The chain rule applies wherever each inner function is differentiable. It holds for scalar, vector, and matrix inputs/outputs (generalizing to Jacobians in the multivariable case).

Notation danger. $d u / d z$ means "derivative of $u$ with respect to its immediate argument $z$ ," not with respect to $x$ . Always identify which variable goes in the denominator.

8.11.5 Pitfalls

Forgetting to multiply by the inner derivative. $(x^{3} + 3)^{8}$ differentiated as $8 (x^{3} + 3)^{7}$ alone is wrong. The $3 x^{2}$ factor is mandatory.
Applying the chain rule when the function is not composite. $x^{8}$ needs only the power rule. $(x^{3} + 3)^{8}$ needs the chain rule. Know the difference.
Stopping after one layer when there are more. $sin (e^{x^{2}})$ has three layers: $sin$ of $e^{z}$ of $x^{2}$ . Differentiate all three: $cos (e^{x^{2}}) \cdot e^{x^{2}} \cdot 2 x$ .

8.11.6 Recap and Bridge

The chain rule — $d u / d x = d u / d z \cdot d z / d x$ — lets you differentiate nested functions by peeling layers from the outside in. It is the foundation of backpropagation in neural networks. Next: partial derivatives extend differentiation to functions of multiple variables.

8.11.7 Real-World and Domain Connection

The chain rule powers every neural network on the planet. In a 100-layer deep network, the derivative of the loss with respect to the first layer's weights is a product of 99 local derivatives. That is exactly the chain rule applied 99 times. Backpropagation is the chain rule computed efficiently on a computation graph. The whole field of deep learning runs on this one rule.

8.11.8 Symbol Registry

$u$ — outer function output — scalar.
$z = z (x)$ — inner function of $x$ — scalar.
$d u / d z$ — derivative of outer w.r.t. $z$ — scalar.
$d z / d x$ — derivative of inner w.r.t. $x$ — scalar.
$d u / d x$ — the full chain-rule derivative w.r.t. $x$ — scalar.

8.12 Partial Derivatives

8.12.1 Definition

Hook. A function of two variables — like temperature depending on both latitude and longitude — changes differently along each axis. A single $d y / d x$ cannot capture both. You need partial derivatives.

Intuition. You are hiking in rolling hills. Your elevation $y$ depends on your east-west position $x_{1}$ and your north-south position $x_{2}$ . If you walk due east (fixing $x_{2}$ ), the slope under your feet is $\partial y / \partial x_{1}$ . If you walk due north (fixing $x_{1}$ ), the slope is $\partial y / \partial x_{2}$ . A partial derivative measures the slope along one direction while all other directions are frozen in place.

The analogy maps: hiking position $(x_{1}, x_{2})$ → function $y$ , eastward slope → $\partial y / \partial x_{1}$ , northward slope → $\partial y / \partial x_{2}$ . The analogy breaks because in reality you can walk diagonal — but the partial derivative only measures cardinal directions. The gradient (next section) handles arbitrary directions.

When $y$ depends on two variables, $y = f (x_{1}, x_{2})$ , you cannot write $d y / d x$ . You ask instead: how does $y$ change when only one input moves, holding the other fixed?

Write that as a partial derivative:

\frac{\partial y}{\partial x _{1}}, \frac{\partial y}{\partial x _{2}} .

The curly $\partial$ signals "differentiate with respect to one variable, treating every other variable as a constant." The computation is the same as ordinary differentiation. You just treat all other variables as fixed numbers.

8.12.2 Worked Example

Take $y = x_{1}^{2} + x_{2}^{2}$ .

Partial with respect to $x_{1}$ : hold $x_{2}$ constant. The derivative of any constant is 0, so the $x_{2}^{2}$ term vanishes. The power rule applies to $x_{1}^{2}$ :

\frac{\partial y}{\partial x _{1}} = 2 x_{1} + 0 = 2 x_{1} .

Partial with respect to $x_{2}$ : hold $x_{1}$ constant. Now the $x_{1}^{2}$ term vanishes:

\frac{\partial y}{\partial x _{2}} = 0 + 2 x_{2} = 2 x_{2} .

For $y = x_{1}^{2} + x_{2}^{2}$ :

$\partial y / \partial x_{1} = 2 x_{1}$ (treat $x_{2}$ as constant)
$\partial y / \partial x_{2} = 2 x_{2}$ (treat $x_{1}$ as constant)

At the point $(x_{1}, x_{2}) = (1, 2)$ : $\partial y / \partial x_{1} = 2$ , $\partial y / \partial x_{2} = 4$ . Along $x_{1}$ , the function rises at rate 2 per unit. Along $x_{2}$ , it rises twice as fast at rate 4 per unit. The bowl is steeper in the $x_{2}$ direction at this point.

8.12.3 Visual Intuition

A function $z = f (x_{1}, x_{2})$ lives in three dimensions — two inputs plus the output. Fixing $x_{2}$ slices the surface with a vertical plane parallel to the $x_{1}$ – $z$ axes. The intersection is a single 2D curve. The partial derivative $\partial z / \partial x_{1}$ is the slope of that curve — the slope you would measure walking along the $x_{1}$ direction only.

The classic "Pringles chip" saddle-shaped surface $z = x_{1}^{2} - x_{2}^{2}$ shows this well. Along $x_{1}$ (fix $x_{2}$ ), the curve is a U-shape opening upward — $\partial z / \partial x_{1} = 2 x_{1}$ (positive for positive $x_{1}$ ). Along $x_{2}$ (fix $x_{1}$ ), the curve is an upside-down U — $\partial z / \partial x_{2} = - 2 x_{2}$ (negative for positive $x_{2}$ ). Same point, two different slopes, because the surface tilts differently in each direction.

8.12.4 Assumptions and Scope

Scope. Partial derivatives exist at a point where the function is differentiable in each variable individually. They extend to any number of variables: for $f (x_{1}, \dots, x_{n})$ , there are $n$ partial derivatives $\partial f / \partial x_{i}$ .

Higher-order partials. You can take partial derivatives of partial derivatives — $\partial^{2} y / \partial x_{1} \partial x_{2}$ . For well-behaved functions ( $C^{2}$ ), the order does not matter: $\partial^{2} y / \partial x_{1} \partial x_{2} = \partial^{2} y / \partial x_{2} \partial x_{1}$ .

Notation warning. $\partial y / \partial x_{1}$ is a single symbol, not a fraction. You cannot "cancel" the $\partial$ 's.

8.12.5 Pitfalls

Forgetting to hold other variables constant. Differentiating $x_{1}^{2} + x_{1} x_{2}$ with respect to $x_{1}$ treats $x_{2}$ as a constant, so the derivative is $2 x_{1} + x_{2}$ — NOT $2 x_{1} + x_{1}$ . The chain/product rules only apply when the "constant" variable is actually constant.
Writing partial where ordinary is needed. If $y = f (x)$ depends on only one variable, write $d y / d x$ , not $\partial y / \partial x$ . Partial derivatives imply multiple inputs.
Mixing up $\partial$ and $d$ . The curly $\partial$ is a convention. On exams, using the wrong symbol is technically a notation error even if the computation is right.

8.12.6 Recap and Bridge

Partial derivatives measure how a function changes along one input direction while all others are frozen. They are the building blocks of the gradient, which packages them into a single vector pointing toward steepest increase.

8.12.7 Symbol Registry

$y = f (x_{1}, x_{2})$ — a function of two inputs — scalar.
$x_{1}, x_{2}$ — the two input variables — scalars, real.
$\partial y / \partial x_{1}$ , $\partial y / \partial x_{2}$ — partial derivatives — scalars.
$\partial$ — the partial-derivative operator — holds other variables constant.

8.13 Gradients

8.13.1 Definition

Hook. Partial derivatives give you slopes along each axis. But which direction should you walk to climb the hill fastest? You need all the slopes packaged together into one arrow. That arrow is the gradient.

Intuition. Picture a blindfolded hiker on a mountain. She cannot see the peak. But she can feel the ground slope under her feet. If she takes one step east, she goes up 2 meters. If she steps north, she goes up 3 meters. The gradient is the arrow pointing (2 east, 3 north) — the combined direction of steepest uphill. If she wants to go uphill fastest, she follows that arrow. If she wants to go downhill fastest, she walks exactly opposite.

The professor used this analogy: the gradient is the green arrow on a loss landscape, pointing toward increasing height. To descend (which is what we want in ML), you step against the arrow.

The gradient stacks every partial derivative of a function into a single vector. For $y = f (x_{1}, x_{2})$ :

\nabla y = [\partial y / \partial x_{1} \partial y / \partial x_{2}] .

It is a vector — one entry per input. Each entry says how fast the function changes along that input's direction. The symbol $\nabla$ is pronounced "nabla" or "del."

8.13.2 Geometric Meaning — Direction of Increase

Gradients are the multi-variable generalization of the single-variable derivative. With one input, $d y / d x$ is a slope — a single number. With two inputs, you need a direction, and a direction needs a vector. The gradient is that vector.

Take the running example $y = x_{1}^{2} + x_{2}^{2}$ . Its gradient is:

\nabla y = [2 x_{1} 2 x_{2}] .

At the point $(x_{1}, x_{2}) = (1, 1)$ , the gradient evaluates to:

\nabla y_{(1, 1)} = [22] .

The gradient of $y = x_{1}^{2} + x_{2}^{2}$ at $(1, 1)$ is $\nabla y = [22]$ . This vector points away from the origin toward where $y$ grows fastest. Since $y = x_{1}^{2} + x_{2}^{2}$ is a bowl opening upward, the steepest increase is radially outward from the center. The vector $(2, 2)$ indeed points radially outward from the origin through $(1, 1)$ .

Sense check: Walk one small step from $(1, 1)$ in the direction $(2, 2)$ , i.e., toward $(1 + ϵ, 1 + ϵ)$ . The function becomes $(1 + ϵ)^{2} + (1 + ϵ)^{2} = 2 (1 + 2 ϵ + ϵ^{2}) \approx 2 + 4 ϵ$ . The rate of change is $4 ϵ / ϵ = 4$ . Walking in any other direction gives a smaller increase — confirming $(2, 2)$ is indeed the direction of steepest ascent.

8.13.3 Gradient Descent Intuition

The gradient always points in the direction of steepest increase. To minimize a function — the whole point of training a machine learning model — you step in the opposite direction.

Gradient descent update rule (conceptual):

x_{new} = x_{old} - η \nabla y (x_{old}),

where $η$ (eta) is the step size, also called the learning rate. Bigger $η$ means faster descent but risks overshooting the minimum. Smaller $η$ is safer but slower. This trade-off drives all of optimization-based ML.

8.13.4 Visual Intuition

Draw a 3D bowl-shaped surface: $y = x_{1}^{2} + x_{2}^{2}$ . The bottom of the bowl is at $(0, 0)$ , where $y = 0$ . At any point on the bowl, the gradient points radially outward — directly away from the minimum. To reach the bottom, you always walk directly toward the origin, which is exactly opposite the gradient.

Now picture the gradient vector field: every point gets a little arrow pointing outward. These arrows get longer as you move away from the center (because $2 x_{1}$ and $2 x_{2}$ grow). The field tells you, everywhere, which way is "uphill." Gradient descent follows the negative of that field. It flows downhill to the bowl's bottom.

8.13.5 Assumptions and Scope

Scope. The gradient is defined for any scalar-valued differentiable function of multiple variables. It generalizes to $n$ inputs: $\nabla f \in R^{n}$ , with $\nabla f = [\partial f / \partial x_{1}, \dots, \partial f / \partial x_{n}]^{⊤}$ .

Direction of steepest ascent is proven. Among all unit-length directions $u$ , the directional derivative $\nabla f \cdot u$ is maximized when $u$ points along $\nabla f$ (by Cauchy-Schwarz).

When it fails. The gradient points toward steepest ascent, not necessarily toward the global maximum — only the local one. On a bumpy loss surface, the gradient takes you to the nearest peak, which might be a tiny hill, not the mountain.

8.13.6 Pitfalls

Thinking the gradient points toward the minimum. It points toward the maximum. Gradient descent steps in the opposite direction. The sign convention matters everywhere in ML.
Assuming the gradient is always positive. The gradient has a direction, not a sign. Its components can be negative. A component of $- 3$ means "the function decreases as you move in the $+ x$ direction." That in turn means the function increases as you move in the $- x$ direction.
Confusing $d y / d x$ (scalar) with $\nabla y$ (vector). With one input, the derivative is a number. With multiple inputs, the derivative is a gradient vector. Same concept, different dimension.

8.13.7 Recap and Bridge

The gradient $\nabla y$ is the vector of partial derivatives, pointing toward steepest increase. Gradient descent steps opposite the gradient: $x_{new} = x_{old} - η \nabla y$ . This single update rule trains every neural network.

8.13.8 Real-World and Domain Connection

Every training run of every neural network — from a 3-layer perceptron to GPT-5 — uses gradient descent. The algorithm is the same. Compute the gradient of the loss with respect to every parameter. Backpropagation does this: it is the chain rule applied to a computation graph. Then nudge each parameter slightly against its gradient. The learning rate $η$ is the most tuned hyperparameter in deep learning.

Gradients also drive reinforcement learning and computer vision. They power physics simulations and economics. In economics, marginal rates of substitution are literally partial derivatives in disguise.

8.13.9 Student Questions and Answers

Q: Is the gradient always positive?

A: No. You can stand anywhere on the surface. The gradient's direction points toward where the function increases. Its components can be negative. If the function rises as you move $+ x$ , that component is positive. If it rises as you move $- x$ , that component is negative. The gradient is about direction of increase, not a sign constraint.

Q: So even when the value is negative, the gradient points toward a positive direction of increase?

A: Yes — it is ascending toward larger function values. "Negative" describes the individual component value, not the goal. The gradient always points uphill — toward higher function values than the current point. If you are at $y = - 10$ , the gradient points toward $y = - 9$ , which is "more positive" (less negative).

8.13.10 Exam Notes

Exam note: Vector calculus (derivatives, partial derivatives, gradients, chain rule) is the most scoring block on the mid-semester. Students historically score better here than on linear algebra. The questions are more direct — compute the gradient, apply the chain rule, find partial derivatives.

Exam note: Principal component analysis (PCA) arrives in lecture ~11 and ties directly back to SVD. The "first singular value dominates when columns are correlated" rule from SVD becomes the "first principal component captures max variance" rule in PCA.

Exam note: Gradients of vector-valued functions (Jacobians) are deferred to the next session and will finish before the exam.

8.13.11 Symbol Registry

$y = f (x_{1}, x_{2})$ — scalar-valued function of two inputs — scalar, $x_{1}, x_{2} \in R$ .
$x_{1}, x_{2}$ — input variables — scalars, real.
$\partial y / \partial x_{1}$ , $\partial y / \partial x_{2}$ — partial derivatives — scalars.
$\nabla y$ — the gradient vector — $R^{2}$ .
$η$ — learning rate (step size) — scalar, $η > 0$ .
$u$ — a unit vector giving a direction — $R^{2}$ , $∥ u ∥ = 1$ .

Exam Guidance Summary

This is a consolidation of all exam-relevant guidance from Lecture 8. Use it as a study checklist for the mid-semester exam.

Mid-Semester Scope

Syllabus: Slides from lecture 1 through lecture 8. Lecture 8 is the final lecture in the mid-semester syllabus.
Gram-Schmidt process: Orthonormal basis computation ( $q_{1}, q_{2}, q_{3}$ ) is in scope. It will be reviewed in the final review lecture (~lecture 14) and the next extended session. Question 12 in the example solutions covers it — confirm it is in scope.
Slides: Uploaded to the course files section (lectures 1 through 8).

SVD — High Exam Priority

Exam note: A numerical SVD problem is highly likely. The professor explicitly told students to expect one.

What to practice:

Compute $A^{⊤} A$ for a given rectangular matrix $A$ .
Find eigenvalues $λ_{i}$ from $det (A^{⊤} A - λ I) = 0$ .
Find eigenvectors and normalize them to get columns of $V$ .
Take square roots of eigenvalues to get singular values: $σ_{i} = λ_{i}$ .
Build $Σ$ as a rectangular diagonal matrix matching $A$ 's shape.
Recover $U$ column by column: $u_{i} = A v_{i} / σ_{i}$ .
For zero singular values, use cross product (in $R^{3}$ ) or orthogonality to get the remaining columns of $U$ .

Key strategy rule:

Long-format ( $m > n$ ): compute $A^{⊤} A$ first (the smaller square).
Wide-format ( $n > m$ ): compute $A A^{⊤}$ first.
Always pick the smaller of the two. You can derive the larger from it.

Vector Calculus — Most Scoring Block

Exam note: Vector calculus is historically the best-scoring section. Students find it more approachable than linear algebra, and questions are more direct.

Key topics to master:

Limit definition of the derivative: $d y / d x = lim_{h \to 0} (f (x + h) - f (x)) / h$ .
Power rule: $d / d x (x^{n}) = n x^{n - 1}$ .
Product rule: $(uv)^{'} = u^{'} v + v^{'} u$ .
Chain rule: $d u / d x = d u / d z \cdot d z / d x$ .
Partial derivatives: treat all but one variable as constant.
Gradient: vector of partial derivatives, pointing toward steepest increase.
Gradient descent: $x_{new} = x_{old} - η \nabla f$ .

What Comes Next

Gradients of vector-valued functions (Jacobians) — deferred to next session, will finish before the exam.
Principal Component Analysis (PCA) — arrives in lecture ~11. Ties back to SVD: principal components = right singular vectors, variance captured = squared singular values.
Consolidated topic list — will be shared by the professor around the final review lecture (~14th).

The remaining calculus topics plus PCA will be covered with practice time before the exam. Focus your study on the SVD numerical and the five calculus rules (power, product, chain, partials, gradient).

Key Industry Applications

Large Language Model Fine-Tuning (LoRA / PEFT)

Low-Rank Adaptation (LoRA) decomposes giant language model weight matrices — think Gemini-class models with billions of parameters — into low-rank forms. Instead of updating the full weight matrix $W \in R^{d \times d}$ during fine-tuning, LoRA freezes $W$ . It inserts a trainable low-rank product $B A$ . Here $B \in R^{d \times r}$ and $A \in R^{r \times d}$ , with $r ≪ d$ (e.g., $r = 8$ for a $d = 4096$ matrix). The effective update is:

W_{new} = W + B A .

This reduces trainable parameters from $d^{2}$ to $2 d r$ . That is a reduction of orders of magnitude. It still matches full fine-tuning accuracy. Every article on "parameter-efficient fine-tuning" for LLMs applies the SVD principle. Most weight updates live in a low-dimensional subspace, so you can represent them with a small-rank product.

Image Compression (WhatsApp-style)

Sending a 1920 × 1080 photo as a raw matrix takes ~6 MB. WhatsApp-style compression decomposes the pixel matrix via SVD. It keeps the top $k$ singular values (typically the first few hundred out of 1080), and reconstructs from those. The image looks nearly identical because the small singular values correspond to imperceptible high-frequency noise. The storage drops from $1920 \times 1080$ numbers to $k \times (1920 + 1080 + 1)$ numbers — a 10× to 100× reduction.

Recommendation Systems (Netflix-style)

A user-movie rating matrix $A \in R^{m \times n}$ with millions of users and thousands of movies is typically rank-50 or lower. That means only ~50 underlying "taste factors" explain all ratings. SVD identifies these factors. Columns of $U$ are user-taste profiles, and columns of $V$ are movie-taste profiles. $Σ$ weights each factor's importance. The Netflix Prize was won using matrix factorization built on SVD.

NLP and Text Analysis

Tweet and document-term matrices are long-format (millions of documents, thousands of terms) — exactly the SVD regime. Latent Semantic Analysis (LSA) decomposes a term-document matrix via SVD. It projects both terms and documents into a lower-dimensional "semantic space." Terms that co-occur in similar contexts end up with similar vectors. For example, "car" and "automobile" land close together. That enables synonym-aware search without manual thesauri.

Computer Vision and Image Processing

Every pixel-level image manipulation — rotation, scaling, compression, denoising — decomposes the image matrix. SVD separates structure (large singular values) from noise (small singular values). Removing the noise terms and reconstructing gives denoised images. Game engines also use SVD's polar decomposition to separate rotation from deformation in physics simulations.

Principal Component Analysis (PCA)

PCA is SVD applied to centered data. Given a data matrix $X$ where each column is mean-centered, compute the SVD $X = U Σ V^{⊤}$ . The principal components are the columns of $V$ . The variance along each component is $σ_{i}^{2} / (n - 1)$ . The low-dimensional projection is $X V_{k}$ where $V_{k}$ keeps only the first $k$ columns. When features are highly correlated, $σ_{1}$ dominates and PCA captures nearly all variance in a single component. This is the same "age ≈ experience" effect you saw in section 8.2.

MFML Lecture 08 notes · Singular Value Decomposition and Vector Calculus

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-10

Sections Breakdown

18.1 Singular Value Decomposition — The Rectangular Reality

Why eigen decomposition fails on non-square matrices and what SVD is.

28.2 Rank and Maximum Possible Rank

Rank as independent directions, capped at min(m, n), and how correlation reduces it.

38.3 Low Rank Approximation

SVD as the sum of rank-1 pieces and the Eckart-Young optimality of truncation.

48.4 SVD Formal Definition and Dimensions

A = U Sigma V^T with the dimensions of each factor and the geometric roles.

58.5 Geometric Intuition — Rotation, Scaling, Rotation

The rotate-scale-rotate story and alignment to the direction of maximum variance.

68.6 Worked Example — SVD of a 3 x 2 Matrix

Full numerical SVD: A^T A, eigenvalues, eigenvectors, Sigma, and U via AV = U Sigma.

78.7 Low Rank Approximation by Truncating Singular Values

Rank-k truncation, the safe-truncation condition, and image compression.

88.8 Derivatives of Univariate Functions

The derivative as the limit of the difference quotient, with the x^2 example.

98.9 The Power Rule — Derivation

d/dx x^n = n x^{n-1} derived from the binomial theorem.

108.10 Product Rule

Differentiating a product: (uv)' = u'v + v'u, with the x sin x example.

118.11 Chain Rule

Differentiating composite functions by multiplying derivatives along the chain.

128.12 Partial Derivatives

Slopes along one input while holding others constant, for multi-variable functions.

138.13 Gradients

The gradient vector and gradient descent for minimizing a function.

14Exam Guidance Summary

Consolidated mid-semester exam strategy for SVD and vector calculus.

15Key Industry Applications

LoRA, image compression, recommender systems, NLP, PCA, and computer vision.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Singular Value Decomposition — Definition and Dimensions

Must-know: Any real matrix $A \in R^{m \times n}$ splits into $A = U Σ V^{⊤}$ , where $U$ (size $m \times m$ ) and $V$ (size $n \times n$ ) are orthogonal and $Σ$ (size $m \times n$ ) holds the singular values on its leading diagonal. SVD is the tool eigen decomposition cannot be: it works on rectangles.

$A = U Σ V^{⊤}, Σ \in R^{m \times n}, σ_{1} \geq σ_{2} \geq \dots \geq 0$

⚠️ Top pitfall: Forgetting that $Σ$ is rectangular — it has the exact shape of $A$ , and only the first $min (m, n)$ diagonal entries can be non-zero.

Self-check: For a 1000 × 2 matrix, what are the dimensions of $U$ , $Σ$ , and $V^{⊤}$ ?

Connects to: Rank, SVD worked computation, geometric intuition, low rank approximation.

Rank and Maximum Possible Rank

Must-know: Rank is the number of linearly independent rows or columns — the number of truly independent directions in your data. It is capped at $min (m, n)$ . Linear correlation among columns (like age ≈ experience) drops the actual rank below the maximum.

$rank (A) \leq min (m, n)$

⚠️ Top pitfall: Counting rows instead of independent directions — a 1000 × 2 matrix has rank at most 2, never 1000.

Self-check: If two columns are exact linear copies, what is the rank of a 1000 × 2 matrix?

Connects to: SVD, low rank approximation, truncation safety.

Low Rank Approximation (Eckart–Young)

Must-know: Rewriting $A = U Σ V^{⊤}$ as a sum of rank-1 outer products lets you keep the first $k$ terms (largest $σ_{i}$ ) and drop the rest. The Eckart–Young theorem says this is the best rank- $k$ approximation in the Frobenius norm — nothing else gets closer.

$A_{k} = i = 1 \sum k σ_{i} u_{i} v_{i}^{⊤}$

⚠️ Top pitfall: Truncating when singular values are comparable — if the spectrum is flat, dropping terms destroys signal. Truncation is only safe when the tail $σ_{i}$ are tiny.

Self-check: When is it safe to keep only the top singular value, and when is it destructive?

Connects to: Rank, SVD definition, image compression, LoRA.

Geometric Intuition — Rotate, Scale, Rotate

Must-know: Every linear map is rotate → scale → rotate. $V^{⊤}$ aligns the data to its axes of greatest variance, $Σ$ stretches/squishes along those axes, and $U$ rotates back to the output frame. A unit circle becomes an ellipse whose semi-axes are the singular values.

$A x = U Σ V^{⊤} x$

⚠️ Top pitfall: Thinking "scaling" means only stretching — a singular value of 0.01 squishes an axis to 1% of its size. Scaling includes shrinking.

Self-check: Why must you rotate before scaling, rather than scaling the original features directly?

Connects to: SVD definition, PCA, truncation.

SVD Worked Computation (the Recipe)

Must-know: For a tall matrix start from the smaller square $A^{⊤} A$ . Eigen-decompose it for $λ_{i}$ and $v_{i}$ , set $σ_{i} = λ_{i}$ , then recover $u_{i} = A v_{i} / σ_{i}$ . A numerical SVD problem is highly likely on the exam — practice the full walkthrough.

$A V = U Σ ⟹ u_{i} = \frac{A v _{i}}{σ _{i}}, σ_{i} = λ_{i} (A^{⊤} A)$

⚠️ Top pitfall: Dividing by $σ_{i} = 0$ for the leftover $U$ column. Use the cross product (in $R^{3}$ ) or orthogonality to fill it; also forgetting to normalize eigenvectors breaks $U^{⊤} U = I$ .

Self-check: For the matrix $[110011]$ , what are $σ_{1}$ and $σ_{2}$ ?

Connects to: SVD definition, eigen decomposition (L6/L7), rank.

Truncated SVD and Image Compression

Must-know: Keeping only the top $k$ singular values stores a 1920 × 1080 image with roughly $k \times (1920 + 1080 + 1)$ numbers instead of ~2 million — a 10× to 100× shrink with no visible quality loss when the spectrum drops steeply.

$storage \approx k (m + n + 1) vs. m \times n$

⚠️ Top pitfall: Assuming truncation is always safe. In our worked 3 × 2 example $σ_{1} = 3 \approx 1.73$ and $σ_{2} = 1$ are comparable, so a rank-1 cut visibly smears the zeros in the corners.

Self-check: Why does a steep singular-value spectrum make compression nearly lossless?

Connects to: Low rank approximation, Eckart–Young, LoRA.

Derivative — Limit Definition

Must-know: The derivative is the instantaneous rate of change — the speedometer reading. It is the limit of the average rate $(f (x + h) - f (x)) / h$ as the nudge $h$ collapses to zero. Never plug in $h = 0$ directly (that gives 0/0).

$\frac{d y}{d x} = h \to 0 lim \frac{f ( x + h ) - f ( x )}{h}$

⚠️ Top pitfall: Treating $d y / d x$ as an ordinary fraction, or confusing the average rate (without the limit) with the instantaneous rate (with $h \to 0$ ).

Self-check: Using the limit, what is the derivative of $x^{2}$ at $x = 3$ ?

Connects to: Power rule, partial derivatives, gradient.

Power Rule

Must-know: For $y = x^{n}$ , bring the exponent down as a multiplier and reduce it by one. This was derived from the limit definition using the binomial theorem (valid for positive integer $n$ ).

$\frac{d}{d x} x^{n} = n x^{n - 1}$

⚠️ Top pitfall: Forgetting to drop the exponent — $d / d x (x^{3}) = 3 x^{2}$ , not $3 x^{3}$ . Also, the power rule needs the base to be $x$ and the exponent constant; for $a^{x}$ it does not apply.

Self-check: What is $d / d x (x^{5})$ at $x = 2$ ?

Connects to: Limit definition, chain rule, product rule.

Product Rule

Must-know: Differentiate one factor while holding the other, then swap and add. Each derivative pairs with the other function, not its own.

$\frac{d}{d x} (u (x) v (x)) = \frac{d u}{d x} v + \frac{d v}{d x} u$

⚠️ Top pitfall: Writing $u^{'} v^{'}$ or $u^{'} + v^{'}$ . Also, if a factor is composite (like $sin (x^{2})$ ), you still need the chain rule on that factor first.

Self-check: Differentiate $z = x sin x$ and check the result at $x = 0$ .

Connects to: Power rule, chain rule.

Chain Rule

Must-know: For nested functions, multiply the derivative of the outer layer by the derivative of the inner layer. Each link is a simple derivative; you follow the dependency chain link by link. This is the backbone of backpropagation.

$\frac{d u}{d x} = \frac{d u}{d z} \cdot \frac{d z}{d x}$

⚠️ Top pitfall: Dropping the inner derivative — $(x^{3} + 3)^{8}$ is $8 (x^{3} + 3)^{7} \cdot 3 x^{2}$ , never just $8 (x^{3} + 3)^{7}$ . Stop only after differentiating every layer.

Self-check: Differentiate $u = (x^{3} + 3)^{8}$ and verify the value at $x = 0$ .

Connects to: Power rule, product rule, gradient descent in neural nets.

Partial Derivatives

Must-know: A partial derivative measures the slope along one input direction while freezing all others as constants. The computation is ordinary differentiation with the other variables treated as fixed numbers.

$\frac{\partial y}{\partial x _{1}}, \frac{\partial y}{\partial x _{2}} for y = f (x_{1}, x_{2})$

⚠️ Top pitfall: Forgetting to hold other variables constant — $\partial / \partial x_{1} (x_{1}^{2} + x_{1} x_{2}) = 2 x_{1} + x_{2}$ , not $2 x_{1} + x_{1}$ . Use $\partial$ for multi-input functions, $d$ for single-input.

Self-check: For $y = x_{1}^{2} + x_{2}^{2}$ , what are the two partials at $(1, 2)$ ?

Connects to: Gradient, derivative, SVD variance intuition.

Gradients and Gradient Descent

Must-know: The gradient stacks all partial derivatives into one vector pointing toward steepest increase. To minimize a function you step opposite the gradient, scaled by the learning rate $η$ . This single rule trains every neural network.

$\nabla y = [\partial y / \partial x_{1} \partial y / \partial x_{2}], x_{new} = x_{old} - η \nabla y$

⚠️ Top pitfall: Thinking the gradient points to the minimum — it points to the maximum. Gradient descent uses the opposite sign. Also, the gradient has a direction, not a fixed sign; its components can be negative.

Self-check: For $y = x_{1}^{2} + x_{2}^{2}$ , what is $\nabla y$ at $(1, 1)$ and which way does descent step?

Connects to: Partial derivatives, chain rule, PCA via SVD.

Previous LectureLecture 07 · Matrix Decompositions, Vector Spaces, and Determinants

Next LectureLecture 09 · Gradients, the Jacobian, and Taylor Series

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Singular Value Decomposition and Vector Calculus

8.1 Singular Value Decomposition — The Rectangular Reality

8.1.1 Definition and Motivation

8.1.2 Why Eigen Decomposition Falls Short

8.1.3 Long Format vs Wide Format Data

8.1.4 Symbol Registry

8.1.5 Assumptions and Scope

8.1.6 Visual Intuition

8.1.7 Pitfalls

8.1.8 Recap and Bridge

8.1.9 Real-World and Domain Connection

8.1.10 Student Questions and Answers

8.2 Rank and Maximum Possible Rank

8.2.1 Definition

8.2.2 Worked Intuition — Age vs Experience

8.2.3 When Correlation Reduces Rank

8.2.4 Assumptions and Scope

8.2.5 Visual Intuition

8.2.6 Pitfalls

8.2.7 Recap and Bridge

8.2.8 Real-World and Domain Connection

8.2.9 Student Questions and Answers

8.3 Low Rank Approximation

8.3.1 Intuition

8.3.2 Reducing Rows Loses Nothing When They Overlap

8.3.3 Formalizing: The SVD Connection

8.3.4 Assumptions and Scope

8.3.5 Visual Intuition

8.3.6 Pitfalls

8.3.7 Recap and Bridge

8.3.8 Real-World: LoRA and Parameter-Efficient Fine-Tuning

8.3.9 Student Questions and Answers

8.4 SVD Formal Definition and Dimensions

8.4.1 The Decomposition A=UΣV⊤

8.4.2 Geometric Roles of the Three Matrices

8.4.3 Worked Dimension Walk — A 1000 × 2 Matrix

8.4.4 Assumptions and Scope

8.4.5 Visual Intuition

8.4.6 Pitfalls

8.4.7 Recap and Bridge

8.4.8 Real-World and Domain Connection

8.4.9 Symbol Registry

8.5 Geometric Intuition — Rotation, Scaling, Rotation

8.5.1 Align to the Direction of Most Variance

8.5.2 Scaling Is Stretching — and Squishing

8.5.3 Rotate Back, Land on Principal Axes

8.5.4 Assumptions and Scope

8.5.5 Pitfalls

8.5.6 Recap and Bridge

8.5.7 Real-World and Domain Connection

8.5.8 Student Questions and Answers

8.6 Worked Example — SVD of a 3 × 2 Matrix

8.6.1 Setup and Strategy

8.6.2 Compute A⊤A (the Smaller Square)

8.6.3 Eigenvalues of A⊤A

8.6.4 Eigenvectors of A⊤A — the V Matrix

8.6.5 The Singular Value Matrix Σ

8.6.6 Computing U — the AV=UΣ Identity

8.6.7 Symbol Registry

8.6.8 Assumptions and Scope

8.6.9 Visual Intuition

8.6.10 Pitfalls

8.6.11 Recap and Bridge

8.6.12 Student Questions and Answers

8.6.13 Exam Notes

8.7 Low Rank Approximation by Truncating Singular Values

8.7.1 The Truncated Sum

8.7.2 Worked Example — Rank-1 Approximation

8.7.3 When Truncation Is Safe

8.7.4 Assumptions and Scope

8.7.5 Visual Intuition

8.7.6 Pitfalls

8.7.7 Recap and Bridge

8.7.8 Real-World: Image Compression

8.7.9 Student Questions and Answers

8.8 Derivatives of Univariate Functions

8.8.1 Setup — A Single Variable Predicts Another

8.8.2 The Formal Limit Definition

8.4.1 The Decomposition $A = U Σ V^{⊤}$

8.6.2 Compute $A^{⊤} A$ (the Smaller Square)

8.6.3 Eigenvalues of $A^{⊤} A$

8.6.4 Eigenvectors of $A^{⊤} A$ — the $V$ Matrix

8.6.5 The Singular Value Matrix $Σ$

8.6.6 Computing $U$ — the $A V = U Σ$ Identity

8.8.3 Worked Example — $f (x) = x^{2}$ at $x = 3$

8.9.3 Worked Example — $f (x) = x^{5}$

8.10.2 Worked Example — $z = x sin x$