Mathematical Foundations for Machine Learning

Eigenvalues, Eigenvectors, and Determinants

📅 Published: 2026-07-09

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning

Eigenvalues, Eigenvectors, and Determinants

This lecture covers three cornerstone ideas in linear algebra. They are the Cauchy-Schwarz inequality, the determinant, and eigenvalues with eigenvectors.

The Cauchy-Schwarz inequality is a bound on inner products. The determinant is a scalar that captures how much a matrix scales area and volume. Eigenvalues and eigenvectors are the special directions along which a matrix acts like pure scaling.

Together, these concepts form the mathematical foundation for principal component analysis (PCA) and dimensionality reduction. These techniques appear in every modern machine learning pipeline.

6.1 Cauchy-Schwarz Inequality

Hook: How do you prove that two vectors cannot be "more similar" than the product of their lengths? The answer is the Cauchy-Schwarz inequality — a simple bound that puts a ceiling on the inner product. It sits in every linear algebra toolbox, not because you compute it daily, but because proofs in ML lean on it silently.

6.1.1 Definition and Explanation

The Cauchy-Schwarz inequality is a simple bound from linear algebra. It relates the inner product of two vectors to the product of their lengths. Think of it as one of those fundamental inequalities — like the triangle inequality. That one says the sum of two sides of a triangle is always at least the third side. Cauchy-Schwarz is the same kind of thing. Someone discovered it, and now it sits in the linear algebra toolkit. The inequality says: the absolute inner product of two vectors is never larger than the product of their lengths.

∣ ⟨ x, y ⟩ ∣ \leq ∥ x ∥ ∥ y ∥

You do not need to know the proof. You just need the result. The proof is in the slides if you want to see it, but it is not relevant to this course.

Symbol Registry

Symbol	Meaning	Type	Shape
$x, y$	Vectors in an inner product space	Vector	$d \times 1$
$⟨ x, y ⟩$	Inner product of x and y	Scalar	Scalar
$∥ x ∥$	Euclidean norm (length) of x	Scalar	Scalar

6.1.2 Geometric Origin via the Dot Product

Intuition + Analogy: Think of two arrows

x

and

y

drawn from a common starting point. Their inner product

⟨ x, y ⟩

tells you how much one arrow "projects" onto the other — how aligned they are. If the arrows point in exactly the same direction, the inner product equals the product of their lengths. If they are perpendicular, the inner product is zero. Cauchy-Schwarz says: no matter what, the inner product cannot exceed the product of their lengths. The normalized inner product —

\frac{⟨ x , y ⟩}{∥ x ∥∥ y ∥}

— is always clamped between

- 1

and

1

. This is exactly the cosine of the angle between them. The analogy: cosine can never be larger than 1, no matter what angle you pick.

You can see where Cauchy-Schwarz comes from by looking at the dot product. The dot product between two vectors can be expressed using the cosine of the angle between them.

x \cdot y = ∥ x ∥ ∥ y ∥ cos θ

Now divide both sides by the product of the lengths.

\frac{x \cdot y}{∥ x ∥ ∥ y ∥} = cos θ

The maximum value of cosine is 1 and the minimum is -1. So the ratio on the left is always bounded between -1 and 1. That is the geometric origin of Cauchy-Schwarz. Take the inner product of two vectors. Divide it by the product of their lengths. The result can never exceed 1 in absolute value. You have an upper bound and a lower bound.

Assumptions & Scope: The Cauchy-Schwarz inequality holds for any inner product space — not just the standard dot product on

R^{n}

. It applies to function spaces with integral inner products. It also applies to complex vector spaces (with the appropriate complex-conjugate inner product) and to weighted inner products.

The key requirement: the inner product must be a valid inner product. It must be bilinear (or sesquilinear for complex), symmetric (or conjugate-symmetric), and positive-definite. When these axioms hold, Cauchy-Schwarz follows. Equality holds if and only if $x$ and $y$ are linearly dependent — one is a scalar multiple of the other.

6.1.3 Practical Relevance

This inequality is rarely used directly in machine learning practice. In eleven years of working in ML, you may never call on it explicitly. It is a foundational tool — something used inside proofs and derivations, not something you compute day to day. It tells you the lower bound and upper bound for normalized inner products. The ratio is always clamped between -1 and 1.

Visual Intuition: Picture two arrows on a 2D plane, both starting at the origin. The first arrow has length 3, the second has length 4. Their inner product is 12 only if they point in exactly the same direction ( $cos θ = 1$ ). It is 0 if they are perpendicular. It is $- 12$ if they point in exactly opposite directions ( $cos θ = - 1$ ). The Cauchy-Schwarz inequality says: the inner product lives in $[- 12, 12]$ , period. The takeaway: the inner product divided by the length product gives $cos θ$ , a number between $- 1$ and $1$ .

Pitfalls:

Confusing inner product with dot product. The dot product $x \cdot y$ is one specific inner product on $R^{n}$ . But Cauchy-Schwarz holds for any valid inner product $⟨ x, y ⟩$ , not just the dot product. The notation $⟨ \cdot, \cdot ⟩$ is the general inner product.
Forgetting the absolute value. The inequality is about $∣ ⟨ x, y ⟩ ∣$ , not about $⟨ x, y ⟩$ itself. Without the absolute value, the lower bound is $- ∣ x ∣∣ y ∣$ , not 0.
Assuming equality is common. Equality requires that $x$ and $y$ be scalar multiples of each other. Most pairs of random vectors do not achieve equality.

Recap + Bridge: Cauchy-Schwarz is a universal bound: the absolute inner product of two vectors cannot exceed the product of their lengths. Geometrically, it says the cosine of the angle between vectors is always between -1 and 1. It is a proof tool — rarely used directly in ML practice, but essential for the derivations that underpin the field. Next: we move from bounds on vectors to a number that characterizes an entire matrix — the determinant.

Real-World & Domain Connection: In statistics, the correlation coefficient between two random variables is exactly the Cauchy-Schwarz inequality applied to centered random variables: $ρ = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}$ and $∣ ρ ∣ \leq 1$ .

In signal processing, the matched filter is the optimal linear filter for detecting a known signal in noise. It achieves its maximum signal-to-noise ratio when the filter is proportional to the signal. This is a direct consequence of Cauchy-Schwarz. In quantum mechanics, the Heisenberg uncertainty principle — $Δ x Δ p \geq ℏ/2$ — is derived from Cauchy-Schwarz applied to wave functions.

6.2 Determinants — Geometric Interpretation

6.2.1 The Transformation View

Hook: What does a determinant actually mean? It is not just a number you crank out from a formula — it is the scaling factor of area, volume, or hyper-volume under a linear transformation. A single scalar captures whether a matrix stretches, squishes, or destroys the space it acts on.

When a matrix $A$ multiplies a vector $x$ to produce a new vector $y$ , the matrix does more than move that one vector. It transforms the entire space.

A x = y

Think of the input space as a rubber sheet of paper with a grid drawn on it. The matrix $A$ applies some force to this sheet — stretching it, squishing it, or shearing it. Every vector in the space moves. Every grid cell deforms. The determinant measures how much the area of any grid cell scales under that transformation.

Symbol Registry

Symbol	Meaning	Type	Shape
$A$	Transformation matrix	Matrix	$n \times n$
$det (A)$	Determinant: area/volume scaling factor	Scalar	Scalar
$x$	Input vector	Vector	$n \times 1$
$y$	Transformed output vector	Vector	$n \times 1$

Intuition + Analogy: Imagine a rubber stamp that prints a square of area 1. Press it onto an elastic sheet. Now stretch the sheet horizontally by a factor of 3. The stamp's imprint now covers area 3. The determinant of the stretch matrix is 3 — the same factor. More generally: take any shape on the grid — a circle, a triangle, an irregular blob — and apply the same matrix. Every shape's area changes by the same factor

det (A)

. The determinant is the universal area multiplier. The mapping is:

Area (A (region)) = ∣ det (A) ∣ \cdot Area (region)

. This analogy breaks for non-square matrices. Determinants are only defined for square matrices. It also breaks for transformations that involve reflection. The absolute value handles area, but the sign carries separate information about orientation.

6.2.2 Area in Two Dimensions

Formalize: A square matrix

A \in R^{2 \times 2}

maps the standard basis vectors

e_{1} = (1, 0)^{T}

and

e_{2} = (0, 1)^{T}

to its own columns. The columns of

A

are the images of the basis vectors:

a_{1} = A e_{1} = (a_{11} a_{21}), a_{2} = A e_{2} = (a_{12} a_{22})

The area spanned by

a_{1}

and

a_{2}

— the area of the parallelogram formed from the two column vectors — is the absolute value of the determinant. For

A = (a_{11} a_{21} a_{12} a_{22})

det (A) = a_{11} a_{22} - a_{12} a_{21}

This formula arises from the parallelogram area: base times height, where height is the perpendicular component of one vector relative to the other. A parallelogram is a four-sided shape whose opposite sides are parallel and equal — here it is the region swept out by the two column vectors placed tail-to-tail. Derivation from parallelogram area: Let

a_{1} = (a_{11}, a_{21})^{T}

and

a_{2} = (a_{12}, a_{22})^{T}

. The area of the parallelogram they span equals the product of their lengths and the sine of the angle

θ

between them:

Area (a_{1}, a_{2}) = ∣ a_{1} ∣ ∣ a_{2} ∣ sin θ = ∣ a_{1} \times a_{2} ∣ = ∣ a_{11} a_{22} - a_{21} a_{12} ∣ = ∣ a_{11} a_{22} - a_{12} a_{21} ∣ = ∣ det (A) ∣ (parallelogram area: base \times height) (cross product magnitude in 2D) (computing the 2D cross product) (rearranging — standard form) (by definition of 2 \times 2 determinant)

∣ det (A) ∣

is precisely the area scaling factor. The sign of

det (A)

tells you whether the transformation preserves the orientation (handedness) of the space. When

det (A) > 0

, the parallelogram has the same handedness as the original square. When

det (A) < 0

, the space was flipped.

In a 2D grid, you have two basis vectors — say the standard basis $(1, 0)$ and $(0, 1)$ . The area enclosed between them starts as 1. Every parallelogram cell in the grid has that same area. The determinant is that uniform area scaling factor.

Now apply a horizontal shear. The Y basis vector moves sideways. The grid slants. The area of each cell may or may not change — shear actually preserves area. But apply a stretch instead: pull the sheet horizontally. The grid cells widen. If the original area was 1 unit and the new area is 8 units, the determinant is 8. The determinant is the multiplicative factor by which area scales, not the difference.

Under a pure rotation, area is preserved. Rotate a square piece of paper by 45 degrees, and the paper's area is exactly the same. The grid lines stay parallel and equidistant. The determinant before rotation equals the determinant after rotation. Rotation does not stretch or squish.

Worked Example: Let

A = (2013)

Compute

det (A)

and interpret the result geometrically.

Step 1 — Identify the columns: The columns are $a_{1} = (2, 0)^{T}$ and $a_{2} = (1, 3)^{T}$ . These are the images of the standard basis vectors under $A$ .

Step 2 — Compute the determinant:

det (A) = (2) (3) - (1) (0) = 6 - 0 = 6

Step 3 — Interpret geometrically: The matrix $A$ stretches the plane. The unit square (area 1) maps to a parallelogram of area 6. Every grid cell scales by a factor of 6. Since $det (A) = 6 > 0$ , orientation is preserved — the transformed parallelogram has the same handedness as the original square.

Sense-check: The matrix is upper triangular (all entries below the diagonal are zero). For triangular matrices, the determinant equals the product of the diagonal entries: $2 \times 3 = 6$ . The result is consistent.

Assumptions & Scope: The geometric interpretation of the determinant as an area/volume scaling factor rests on these conditions:

The matrix must be square ( $n \times n$ ). Non-square matrices have no determinant — the concept of scaling a space onto itself requires equal input and output dimensions.
The absolute value $∣ det (A) ∣$ gives the magnitude of scaling. The sign carries separate information about orientation (preserved or reversed). The two pieces of information are independent.
When $det (A) = 0$ , the geometric object degenerates — a parallelogram collapses to a line, a parallelepiped flattens to a plane. The "volume" is zero, but the object still exists as a set of lower dimension. The formula $∣ det (A) ∣$ still yields the correct zero measure.
The interpretation holds for any square matrix with real entries. Over $C$ , the determinant is complex-valued, and the geometric picture requires a different notion of volume.
Viewing columns as images of basis vectors is equivalent to viewing rows that way, because $det (A^{T}) = det (A)$ — transposing does not change the determinant.

Visual Intuition: Picture the 2D coordinate plane with horizontal $x$ -axis and vertical $y$ -axis. The original unit square sits at $(0, 0)$ , $(1, 0)$ , $(1, 1)$ , $(0, 1)$ — a perfect square of area 1. Apply the matrix $A = (2013)$ from the worked example. Each vertex transforms: $(0, 0)$ stays at the origin, $(1, 0)$ maps to $(2, 0)$ , $(1, 1)$ maps to $(3, 3)$ , and $(0, 1)$ maps to $(1, 3)$ . The four transformed points form a parallelogram. Its base runs from $(0, 0)$ to $(2, 0)$ — length 2 along the $x$ -axis. Its left edge rises vertically from $(2, 0)$ to $(3, 3)$ — a vertical climb of 3. The horizontal base times the vertical height gives $2 \times 3 = 6$ , matching $det (A)$ . The takeaway: the four matrix entries dictate the shape of the transformed grid, but the determinant alone — one number — tells you the area of any transformed shape.

6.2.3 Volume in Three Dimensions and Beyond

In three dimensions, the determinant becomes the scaling factor of volume — the volume of a parallelepiped defined by three basis vectors. A parallelepiped is the 3D analog of a parallelogram: a box-like shape whose opposite faces are parallel parallelograms. Multiply a $3 \times 3$ matrix by a 3D vector. The transformation changes a unit cube of volume 1 into a parallelepiped. The determinant tells you by how much the volume scales.

For $A \in R^{3 \times 3}$ , the three columns give three vectors $a_{1}, a_{2}, a_{3}$ . The volume of the parallelepiped they span equals $∣ det (A) ∣$ . Algebraically, the determinant recovers the scalar triple product — $a_{1} \cdot (a_{2} \times a_{3})$ — which is exactly the signed volume of the parallelepiped.

In $n$ dimensions, the determinant generalizes to hyper-volume — the $n$ -dimensional analog of area and volume. The concept stays the same: the determinant is always a single scalar. Whether you think in 2D (area), 3D (volume), or $n$ D (hyper-volume), the determinant is one number that captures how much the transformation scales the measure of space.

6.2.4 Determinant of Zero — Dimensional Collapse

When both basis vectors become collinear — lying on the same line — the entire 2D space collapses to a single line. There is no area on a line. The determinant is 0. In 3D, if all three basis vectors lie in the same plane, the volume collapses and the determinant is 0.

A zero determinant signals that the transformation has destroyed a dimension. The matrix is singular — it is non-invertible. You cannot undo the transformation because information was lost when the space collapsed. Multiple different input vectors map to the same output vector, so you cannot recover the original input from the output alone.

Pitfalls:

A zero determinant does not mean the output is always the zero vector. A singular matrix can produce non-zero outputs — it simply maps the entire space onto a lower-dimensional subspace (a line, a plane). Example: $A = (1224)$ has $det (A) = 0$ because row 2 is twice row 1. Yet $A (10) = (12) \neq = 0$ . The space collapses, not every output.
Do not confuse $∣ det (A) ∣$ with $det (∣ A ∣)$ . The notation $∣ det (A) ∣$ means the absolute value of the determinant (always non-negative). Writing absolute values inside the matrix first — $det (∣ A ∣)$ — produces an entirely different quantity with no standard geometric meaning.
Large entries do not imply a large determinant. A matrix with all entries in $[0, 1]$ can have a large determinant if the columns are long and nearly orthogonal. A matrix with entries in the thousands can have a determinant near zero if the columns are nearly collinear. The determinant depends on geometry, not magnitude of entries.
The professor flags this repeatedly: Students often compute $det (A) \neq = 0$ , correctly conclude the matrix is invertible, but fail to apply the contrapositive — $det (A) = 0$ implies singular and non-invertible. Both directions matter equally on exams.

Exam note: If

det (A) = 0

, the matrix is singular and the transformation squishes the space into a lower dimension. Equivalent statements:

det (A) = 0

⟺

the columns are linearly dependent

⟺

the matrix is not invertible

⟺

the homogeneous system

A x = 0

has a non-trivial solution. This chain of equivalences is one of the most heavily tested facts in the course.

6.2.5 Student Questions and Answers

Q: Is the determinant the actual volume of the newly transformed area, or is it the change between the earlier and new area?

A: The determinant is the multiplicative scaling factor — the ratio of new area to old area. It is not a difference. If the original unit square has area 1 and the transformed parallelogram has area 32, then $det (A) = 32$ . Mathematically, for any measurable region $R$ in the domain, the image $A (R)$ has area equal to $∣ det (A) ∣ \cdot Area (R)$ . The determinant is the constant of proportionality.

Contrast this with subtraction: the area grew by 31, but the determinant is the factor 32. If the original area were not 1, the "change" would vary by region, but the multiplicative factor stays the same everywhere — that is why the determinant is the meaningful quantity.

Q: When we rotate a rubber sheet, the grid lines stay parallel and equidistant. Does the determinant change?

A: No. Every pure rotation matrix has determinant exactly $+ 1$ . The standard rotation matrix for angle $θ$ is

R_{θ} = (cos θ sin θ - sin θ cos θ)

and its determinant is $cos^{2} θ + sin^{2} θ = 1$ for every angle $θ$ . A rotation preserves area (no stretching), preserves shape (no squishing), and preserves orientation (no flipping). Rotate a square piece of paper by 45 degrees: same area, same shape — just pointed in a different direction. The determinant is unchanged.

Recap + Bridge: The determinant is a single scalar. It measures how much a linear transformation scales area (2D), volume (3D), or hyper-volume (

n

D). A zero determinant means the space collapses — the matrix is singular and non-invertible. A positive determinant means orientation is preserved. A negative determinant means orientation is reversed.

Next: we move from the geometric meaning of determinants to another perspective on matrices. We look at the special directions (eigenvectors) along which the transformation acts like pure scaling. The scaling factors (eigenvalues) define those directions.

Real-World & Domain Connection: In computer graphics, GPU pipelines use the determinant for back-face culling. When rendering a 3D triangle mesh, the sign of the determinant of the $3 \times 3$ matrix formed by the triangle's vertex positions tells the renderer something useful. It tells the renderer whether the triangle faces toward or away from the viewer. Triangles facing away are culled — skipped — saving substantial rendering time.

In continuum mechanics, the deformation gradient tensor's determinant $J = det (F)$ gives the local volume change ratio. It governs how materials compress or expand under stress. In multivariable calculus, the Jacobian determinant converts between coordinate systems during integration — $d x d y = ∣ det (J) ∣ d u d v$ . The determinant captures exactly how area elements stretch. In probability theory, the same principle rescales probability density functions when you transform random variables. The determinant sits at the center of any discipline that involves changing coordinates or deforming space.

6.3 Eigenvalues and Eigenvectors — Geometric Intuition

6.3.1 Hook — What Survives the Transformation?

Apply a matrix to a handful of vectors. Most of them move — their direction shifts, their length changes. But pause and ask: do any vectors keep their direction? Are there special directions that survive the transformation intact, merely stretching or shrinking along the same line they started on?

Those directions exist. They are the eigenvectors. The matrix acts on them the way a zoom lens acts on a photograph: everything along that axis scales, nothing rotates away. Finding these invariant directions is the central geometric problem of this section.

6.3.2 Intuition and Analogy — Three Physical Metaphors

Spinning Globe. Place your finger on a globe and spin it. Every point on the surface traces a circle — except the two points at the North and South Poles. Those points stay fixed. The axis connecting them does not move. In the language of eigenvectors: the rotation axis is an eigenvector. Its eigenvalue is 1. Points on that axis do not get scaled at all.

Stretching Dough. Roll out a sheet of dough with a rolling pin. The dough stretches in one direction and compresses in the perpendicular direction. Vectors aligned with the stretch direction get longer — they are eigenvectors with $λ > 1$ . Vectors aligned with the compression direction get shorter — they are eigenvectors with $0 < λ < 1$ . Both directions are eigenvector axes. The dough does not rotate vectors; it only scales them along those two principal directions.

Shearing a Deck of Cards. Push the top half of a deck sideways. The cards slide, but the bottom card stays put. Every card slides parallel to the bottom card. Horizontal vectors — vectors parallel to the cards — do not change direction. They slide but stay horizontal. Those horizontal vectors are eigenvectors. Their eigenvalue is $λ = 1$ . They neither stretch nor shrink; they translate within their own axis.

A matrix transformation can also be thought of as applying a force to the input space. A force has a direction and a magnitude. The direction of the force is the eigenvector. The magnitude of the force is the eigenvalue. Vectors that align with the force direction only get stretched or squished along that same line — they stay on their axis. Any vector not aligned with the force direction gets displaced to a new direction entirely.

6.3.3 The Core Equation — Formal Definition

A nonzero vector $x$ is an eigenvector of a square matrix $A$ if multiplying $A$ by $x$ returns a scalar multiple of $x$ :

A x = λ x

The scalar $λ$ is the eigenvalue associated with $x$ . This equation says: the matrix $A$ acts on $x$ the same way scalar multiplication by $λ$ would. The output is parallel to the input.

The zero vector is excluded from the definition. If $x = 0$ , the equation $A 0 = λ 0$ holds for every $λ$ , which makes the eigenvalue meaningless.

Symbol Registry

Symbol	Meaning	Type	Shape
$A$	Square transformation matrix	Matrix	$n \times n$
$x$ (or $u$ )	Eigenvector — must be nonzero	Vector	$n \times 1$
$λ$ (Lambda)	Eigenvalue: scaling factor along that direction	Scalar	Scalar

Eigenspace. For a given eigenvalue $λ$ , the set of all eigenvectors associated with $λ$ , together with the zero vector, forms a subspace called the eigenspace. Every vector in that eigenspace satisfies $A x = λ x$ .

6.3.4 Worked Example

Take the matrix

A = [1562]

and the vector

u = [6 - 5]

A u = [1562] [6 - 5] = [1 (6) + 6 (- 5) 5 (6) + 2 (- 5)] = [6 - 30 30 - 10] = [- 24 20]

Factor the output: $[- 24 20] = - 4 [6 - 5] = - 4 u$ .

The output is $- 4 \times u$ . So $λ = - 4$ is the eigenvalue. The vector $u$ got scaled by a factor of $- 4$ . Its direction stayed the same. The negative sign means it flipped to the opposite side of the same axis. It is still on that axis. The eigen relationship holds: $A u = λ u$ .

Non-Example. Consider $v = [3 - 2]$ with the same matrix $A$ .

A v = [1562] [3 - 2] = [1 (3) + 6 (- 2) 5 (3) + 2 (- 2)] = [3 - 12 15 - 4] = [- 9 11]

For $v$ to be an eigenvector, we need $A v = λ v$ for a single $λ$ . Checking component-wise: $- 9/3 = - 3$ would give $λ = - 3$ , but $11/ (- 2) = - 5.5$ would give $λ = - 5.5$ . The components demand inconsistent scaling factors. No single $λ$ satisfies the equation. So $v$ is not an eigenvector.

6.3.5 Assumptions and Scope

Square matrices only. The definition $A x = λ x$ requires $A$ to be square. If $A$ is $n \times n$ , then $x$ must be $n \times 1$ and the output $A x$ is also $n \times 1$ — matching the shape of $λ x$ .

Up to $n$ eigenvalues. An $n \times n$ matrix has up to $n$ eigenvalues, counting multiplicity. Some matrices have fewer than $n$ distinct eigenvalues. The characteristic polynomial $det (A - λ I) = 0$ has degree $n$ , so it yields $n$ roots in the complex plane, though some may repeat.

Real and complex eigenvalues. A real matrix can produce complex eigenvalues. They always appear in conjugate pairs. Rotation matrices in 2D, for instance, have eigenvalues $cos θ \pm i sin θ$ — purely real only when $θ = 0$ or $θ = π$ .

Symmetric matrices. If $A$ is symmetric ( $A = A^{T}$ ), all eigenvalues are real. This property makes symmetric matrices central in applications like principal component analysis and spectral clustering.

Triangular matrices. If $A$ is upper or lower triangular, the eigenvalues sit on the main diagonal. No computation needed — read them off directly.

Negative eigenvalues. A negative $λ$ means the eigenvector flips direction — a reflection across the origin — and scales by $∣ λ ∣$ . The vector stays on the same axis; it just points the opposite way.

6.3.6 Visual Intuition — Axes, Scaling, and Rotation

Eigenvectors are entire axes. An eigenvector is a direction, not a single arrow. If $[6 - 5]$ is an eigenvector, then $[12 - 10]$ , $[- 6 5]$ , and $[3 - 2.5]$ — every scalar multiple along that line — are also eigenvectors. The eigenvector is the axis itself.

Stretching and shrinking are both eigen behavior. A vector that elongates ( $λ > 1$ ) is an eigenvector. A vector that shrinks ( $0 < λ < 1$ ) is an eigenvector. The test is always: does the vector remain on the same axis after the transformation? The amount of scaling — stretch or squish — does not matter for eigenvector status.

Flipping. When $λ < 0$ , the vector flips 180 degrees to the opposite side of the axis. It stays collinear with the original direction. A flip is still an eigenvector: the axis is the same, only the sign of the coordinate along that axis changes.

Rotation matrices. A pure 2D rotation matrix sweeps every vector along a circular arc. No vector stays on its original axis. So a 2D rotation matrix has no real eigenvectors. Its eigenvalues are complex: $e^{\pm i θ} = cos θ \pm i sin θ$ . In 3D, however, every rotation happens around some axis. Points lying on that rotation axis do not move. So in 3D, the axis of rotation is an eigenvector with $λ = 1$ .

6.3.7 Comparison — Determinant vs Eigenvalue

Both the determinant and the eigenvalue describe scaling, but at different scopes.

Aspect	Determinant	Eigenvalue
What it measures	Scaling of area or volume across the entire space	Scaling along one specific direction
Count	One number for the whole matrix	One number per eigenvector direction (up to $n$ )
Relationship	$det (A) = λ_{1} λ_{2} \dots λ_{n}$	Each $λ_{i}$ is a factor of $det (A)$
Zero value means	Space collapses — at least one dimension squashes to zero; at least one eigenvalue is zero	Vectors along that direction get annihilated; the matrix is singular along that axis
Sign	Positive means orientation preserved; negative means orientation reversed	Positive means same direction; negative means flipped direction
Geometric picture	The volume of the unit cube after transformation	The stretch factor along one principal axis

The determinant is the product. The eigenvalues are the factors. Knowing the determinant tells you the net volume change. Knowing the eigenvalues tells you how that volume change distributes across each principal direction.

6.3.8 Common Pitfalls — Student Questions

Q1: What is the difference between determinant and eigenvalue? They both seem to be about scaling.

A: The determinant measures the scaling of area or volume across the entire space — it is one number for the whole transformation. The eigenvalue measures the scaling along a specific direction. They are related: $det (A) = λ_{1} λ_{2} \dots λ_{n}$ . But they answer different questions. "How much did the total volume change?" — that is the determinant. "Along which directions does the transformation act like pure scaling, and by how much?" — those are the eigenvalues. See Section 6.3.7 for a side-by-side table.

Q2: When the determinant is zero, are the eigenvalues also zero? Can eigenvectors even exist when the space collapses?

(Asked by several students — 3 variants of this question appeared.)

A: When $det (A) = 0$ , the space collapses — area or volume vanishes. Since the determinant equals the product of all eigenvalues, a zero determinant means at least one eigenvalue must be zero. But not all eigenvalues need to be zero.

Eigenvectors still exist when the space collapses. The transformation still has a direction along which it acts. Vectors lying along the nullspace direction — the direction that gets squashed to the origin — are eigenvectors with $λ = 0$ . Vectors along other principal directions remain eigenvectors with their respective eigenvalues. The only case where no real eigenvectors exist is a pure rotation in 2D, where every vector changes direction and no axis remains invariant.

Q3: What qualifies a vector as an eigenvector? Can a matrix have multiple eigenvectors? Do shrinking vectors count?

(Asked by two students — covering multiple eigenvectors and the stretching vs. shrinking distinction.)

A: An $n \times n$ matrix has up to $n$ distinct eigenvector directions — the principal axes along which the transformation acts independently. In a 2D transformation, you typically get two eigenvector directions. Each direction is an entire axis, not a single vector. Any scalar multiple of an eigenvector is also an eigenvector.

The test for eigenvector status is simple: does the vector remain on the same axis before and after the transformation? If yes, it is an eigenvector — whether it got longer ( $λ > 1$ ), shorter ( $0 < λ < 1$ ), flipped ( $λ < 0$ ), or stayed the same length ( $λ = 1$ ). The amount and sign of scaling do not affect eigenvector status. Only a change in direction disqualifies a vector.

Q4: Do eigenvectors depend on the matrix? Do rotation matrices have eigenvectors?

(Asked by two students — covering matrix dependence and rotation.)

A: Eigenvectors depend entirely on the matrix. Change the matrix, and the eigenvectors change. For a given matrix $A$ , the eigenvectors are fixed — they are the principal axes of that specific transformation. Any vector could be an eigenvector for some matrix, but for a given $A$ , only vectors lying along its principal axes qualify.

In a 2D rotation, every vector changes direction. There is no axis along which vectors merely stretch or shrink. So no real eigenvectors exist — the eigenvalues are complex conjugates. In a 3D rotation, the axis of rotation is an eigenvector with $λ = 1$ . Points on that axis do not move. Higher-dimensional rotations always have at least one real eigenvector when the dimension is odd.

6.3.9 Recap and Bridge

A matrix transformation reshapes space. Most vectors change direction. But every square matrix (except pure 2D rotations) hides special directions — eigenvectors — along which the matrix acts like a scalar:

A x = λ x

. The scalar

λ

is the eigenvalue. Eigenvectors are entire axes, not single vectors. A negative eigenvalue flips direction. A zero eigenvalue collapses that axis to a point. The determinant is the product of all eigenvalues — it captures the net volume scaling, not the directional breakdown. These geometric facts set up the algebraic question: given a matrix

A

, how do you actually find its eigenvalues and eigenvectors? The answer — the characteristic equation

det (A - λ I) = 0

— is the topic of section 6.5.

6.3.10 Real-World Connections

Principal Component Analysis (PCA). Given a cloud of data points, PCA finds the directions of maximum variance. Those directions are the eigenvectors of the covariance matrix. The eigenvalues tell how much variance each direction captures. This underpins dimensionality reduction, facial recognition, and data compression.

Google PageRank. The web is a giant graph. PageRank models a random surfer clicking links. The steady-state probability of being on each page is the eigenvector of the web's link matrix corresponding to $λ = 1$ . That dominant eigenvector ranks the pages.

Vibration Analysis. When a bridge or a building shakes, its natural vibration modes are eigenvectors of the stiffness matrix. The eigenvalues give the squares of the natural frequencies. Engineers check that these frequencies do not match driving forces — avoiding resonance disasters.

Quantum Mechanics. Observable quantities — position, momentum, energy — are represented by matrices. The possible measurement outcomes are the eigenvalues. The state of the system after measurement collapses to the corresponding eigenvector.

Markov Chains. The long-run steady-state distribution of a Markov chain is the eigenvector of the transition matrix with eigenvalue $λ = 1$ .

6.4 Applications in Machine Learning — Dimensionality Reduction

6.4.1 Hook: The Curse of 100 Columns

You sit at your desk at a music streaming company. The dataset has 100 columns per user — age, location, device type, time spent listening, time spent skipping, favorite genre, daily active minutes. The label is binary: "subscribed" (yes) or "not subscribed" (no).

Your first impulse is to throw all 100 columns into a model and let it figure things out. But 100 columns hides a problem. Some columns are redundant. Time spent listening and time spent active? Nearly identical. Some columns are noise. Device firmware version number? Purely random. Some columns have mathematical relationships you cannot see by scanning the raw data.

If you fit a model on 100 columns, your model must learn coefficients for all 100 — even the useless ones. This wastes compute, inflates variance in predictions, and makes the model hard to inspect.

If you could squeeze those 100 columns into just 3 numbers, the model would train faster. Those 3 numbers would capture all the meaningful variation. The model would also generalize better. You could plot the data on a single graph.

Eigenvectors deliver exactly that: they compress high-dimensional data into a tiny handful of truly informative coordinates.

6.4.2 Intuition + Analogy: Finding the Grain of the Wood

Imagine a long, slender wooden plank floating in a room. You want to describe its position with as few coordinates as possible. You could describe it by projecting it onto the floor, the left wall, and the back wall — the standard

x

y

z

axes of the room.

But those axes do not align with the plank. A better approach: find the direction the plank naturally points in. Describe every point on the plank by how far along that grain direction it lies. One coordinate — not three — captures the plank's extent. The cross-section, perpendicular to the grain, is tiny and boring.

High-dimensional data behaves the same way. Points do not fill the space evenly. They cluster along a few dominant directions — the "grain" of the data. Eigenvectors locate that grain. The data spreads most along the first eigenvector, less along the second, and negligibly along the rest.

Analogy Key: Standard axes (age, experience, location) are the room's $x$ , $y$ , $z$ . The plank is your data cloud. Eigenvectors are the plank's natural axes. Eigenvalues measure how far the plank extends along each natural axis. The analogy breaks when data curves (non-linear structure) — PCA only finds straight axes, not curved ones.

6.4.3 Why Eigenvectors Matter in ML

Eigenvalues and eigenvectors are the most important concept in linear algebra for machine learning. Along with singular value decomposition — covered in the next lecture — they form the foundation of dimensionality reduction.

The central idea: high-dimensional data often has only a few directions of meaningful variation. You might have 100 features, but the target variable likely depends on only a handful of them. Eigenvectors help you find those important directions.

6.4.4 The Variance Intuition

Suppose you work at a music streaming company and want to predict whether a user will buy a subscription. You have hundreds of features: age, gender, location, listening habits, device type, and so on. The historical data tags some users as subscribers (yes) and others as non-subscribers (no).

Your hypothesis is that subscription depends on some of those features. But if a feature has zero variance, it cannot explain any variation in the output. Say every user on the platform is exactly 30 years old. That feature contributes nothing. Only features that vary can explain variation in the output.

So you want to find which directions in your data have the most variance. You want the axes along which the data spreads out the most. Those axes are combinations of your original features — not just "age" or "experience" alone, but linear combinations like $z = w_{1} \cdot age + w_{2} \cdot experience$ .

6.4.5 Worked Intuition: Age vs Experience

Imagine you have two features: age and years of experience. These are correlated — older people tend to have more experience. Plot all your data points: age on the X axis, experience on the Y axis. The points form an elongated cloud.

The direction of elongation is not purely age. It is not purely experience. It is a new diagonal axis — a linear combination of both. If you project all data points onto that diagonal axis, you capture most of the spread. The perpendicular direction captures very little spread.

Now instead of storing two numbers per person (age, experience), you store one number: their coordinate on that principal axis. You went from 2D to 1D with minimal information loss.

Worked calculation. Suppose after mean-centering, your two features have covariance matrix:

Σ = [σ_{age}^{2} σ_{exp,age} σ_{age,exp} σ_{exp}^{2}]

Let the eigenvalues be $λ_{1} = 0.8$ and $λ_{2} = 0.2$ , with eigenvectors $v_{1} = (0.71, 0.71)^{T}$ and $v_{2} = (- 0.71, 0.71)^{T}$ .

The first eigenvector points at 45° — an equal mix of age and experience. This axis captures $\frac{0.8}{0.8 + 0.2} = 80%$ of the total variance. The second eigenvector, orthogonal, captures the remaining 20%.

Projecting a user with $x = (age_{i}, exp_{i})$ onto the first principal component:

z_{i} = v_{1}^{T} x_{i} = 0.71 \cdot age_{i} + 0.71 \cdot exp_{i}

Each user is now represented by a single scalar $z_{i}$ . You compressed 2D to 1D while retaining 80% of the information.

6.4.6 Eigenvectors as Principal Axes of Variance

The eigenvectors of the data covariance matrix point along the directions of maximum variance. The first eigenvector — the one with the largest eigenvalue — captures the most variance. The second eigenvector, orthogonal to the first, captures the next most. And so on.

Symbol Registry

Symbol	Meaning	Type	Shape
$X$	Data matrix (observations × features), mean-centered	Matrix	$m \times d$
$Σ$	Covariance matrix $\frac{1}{m} X^{T} X$	Matrix	$d \times d$
$λ_{i}$	i-th eigenvalue: variance captured along that direction	Scalar	Scalar
$v_{i}$	i-th eigenvector: principal direction, $∥ v_{i} ∥ = 1$	Vector	$d \times 1$
$z$	Projected coordinate on a principal axis	Scalar	Scalar
$Z$	Data in the reduced $k$ -dimensional subspace	Matrix	$m \times k$

The covariance matrix is symmetric, so the Spectral Theorem guarantees real eigenvalues and an orthonormal eigenbasis. The eigendecomposition:

Σ = V Λ V^{T}, V = [v_{1} v_{2} \dots v_{d}], Λ = diag (λ_{1}, λ_{2}, \dots, λ_{d})

This factors the transformation into three operations: rotate into the eigenbasis ( $V^{T}$ ), scale each axis independently by $λ_{i}$ ( $Λ$ ), rotate back ( $V$ ).

6.4.7 Formalize: PCA Step by Step

Given $m$ observations and $d$ features, stored as matrix $X \in R^{m \times d}$ :

Mean-center the data. Subtract the column mean from every entry. The centered matrix is $X ~$ .
Compute the covariance matrix. $Σ = \frac{1}{m} X ~^{T} X ~$ . This $d \times d$ matrix encodes pairwise feature relations.
Eigendecompose. Solve $Σ v_{i} = λ_{i} v_{i}$ . Sort eigenpairs by $λ_{i}$ descending.
Select top $k$ . Choose the first $k$ eigenvectors forming $V_{k} \in R^{d \times k}$ .
Project. Compute the reduced representation: $Z = X ~ V_{k} \in R^{m \times k}$ .

The reconstruction error equals the sum of dropped eigenvalues:

Error (k) = i = k + 1 \sum d λ_{i}

The fraction of variance retained is:

R^{2} (k) = \frac{\sum _{i = 1}^{k} λ _{i}}{\sum _{i = 1}^{d} λ _{i}}

A common heuristic: pick the smallest $k$ such that $R^{2} (k) \geq 0.95$ , retaining 95% of variance. Plotting $λ_{i}$ against $i$ produces the scree plot. The "elbow" — where eigenvalues flatten — suggests a natural cutoff.

6.4.8 Dimensionality Reduction by Truncation

Suppose you compute all 100 eigenvalues from your 100-feature dataset. The first eigenvalue might be 20, the second 10, the third 2, and the remaining 97 might all be tiny — below 0.01.

The first eigenvector alone captures a huge amount of the total variance. The first two together capture even more. The first three might capture 99% of the total variance. You can discard the other 97 dimensions and lose only 1% of the information. Your data compresses from 100 columns to 3 columns.

This is dimensionality reduction. You find the eigenvectors of your data matrix. The eigenvalues tell you how much variance each direction captures. You keep only the top few directions and project your data onto them. You store fewer numbers. You build simpler models. And you lose almost nothing.

Assumptions & Scope:

Assumption	Why It Matters
Linearity. Data lives on or near a linear subspace.	PCA projects onto flat hyperplanes. A spiral or swiss roll cannot be flattened usefully. Consider kernel PCA or autoencoders for non-linear data.
Mean-centered data.	Without centering, the first component points at the data centroid, not the direction of maximum variance. Always center.
Feature scaling.	Features on different scales (age vs. salary) distort the covariance matrix. Standardize to zero mean and unit variance before PCA.
Large variances imply signal.	PCA assumes high-variance = signal. This fails if noise has high variance (sensor drift, artifacts). Inspect loadings.

Visual Intuition: Picture a 3D ellipsoid floating in space. The ellipsoid's longest axis is the first principal component, with length $λ_{1}$ . The second axis, perpendicular to the first, has length $λ_{2}$ . The shortest axis completes the ellipsoid. PCA finds this ellipsoid, reorients coordinates to align with it, then chops off the shortest axes. For the age–experience scatterplot: an elongated ellipse tilted at 45° — the major axis is the first principal component, the minor axis the second.

Pitfalls:

Forgetting to standardize. A feature in [0,1000] dominates one in [0,1] purely by numeric scale, not relevance. Standardize all features to unit variance.
Misinterpreting loadings. A near-zero loading does not mean the feature is unimportant — it may contribute via a correlated proxy loaded on another component.
Dropping low-variance = dropping signal. In anomaly detection, anomalies live in the low-variance subspace. Always ask: is my signal in the head or tail of the eigenvalue spectrum?
Over-relying on the elbow rule. The scree plot may not show a clear elbow. Use the 95%-variance rule or cross-validation when eigenvalues decay smoothly.

Student Questions and Answers

Q: Is there ever a case where we need to keep the low-variance directions?

A: Generally no for dimensionality reduction. If a direction captures only 2% of the variance, discarding it loses very little information. But one exception: anomaly detection. Anomalies often break the dominant correlation pattern, so their signal lives in the low-variance subspace. If you discard those components, you discard the anomaly signal. The rule: if you seek the dominant pattern, keep top eigenvalues. If you seek the exception, inspect the tail.

Q: Does the sign of the eigenvalue — positive or negative — matter in machine learning?

A: Eigenvalues of a covariance matrix are always non-negative because covariance matrices are positive semi-definite. The Gram matrix $X ~^{T} X ~ = M^{T} M$ gives $z^{T} M^{T} M z = ∥ M z ∥^{2} \geq 0$ for all $z$ , so all eigenvalues satisfy $λ_{i} \geq 0$ . A negative eigenvalue would mean "negative variance" — impossible since variance is a sum of squares. In other contexts — non-symmetric matrices, Hessians at saddle points — negative eigenvalues signal a reflection or a saddle point. For PCA, all eigenvalues are zero or positive.

Q: When we reduce dimensions, is the data actually getting scaled smaller or is something else happening?

A: You are projecting the data onto new axes — the eigenvector directions. It is not about shrinking data values. It is about choosing which axes to keep and which to discard. Your original point $x = (3.2, 4.1, 0.02)$ becomes $z = (5.1, 0.8)$ after projecting onto the top two eigenvectors and dropping the third. The numbers changed — they are coordinates in a new basis — but they are not "scaled down" uniformly. The third coordinate vanished because its eigenvector captured negligible variance. Think of it as taking a 3D object and capturing its most informative 2D shadow.

Q: What is the significance of eigenvectors in machine learning beyond PCA?

A: Eigen decomposition appears everywhere. In spectral clustering, eigenvectors of the graph Laplacian embed data points so clusters separate nicely. In NLP, latent semantic analysis (LSA) uses SVD — the generalized eigen decomposition — on term-document matrices to capture latent topic structure. In optimization, the condition number of the Hessian — the ratio of largest to smallest eigenvalue — determines how quickly gradient descent converges. In computer vision, eigenfaces compress images by keeping only the top pixel combinations. In PageRank, the dominant eigenvector of the web's link matrix ranks every webpage. The next lecture covers SVD, which generalizes eigen decomposition to rectangular matrices.

Recap + Bridge: Principal Component Analysis finds the directions of maximum variance in your data. It does this by eigendecomposing the covariance matrix. The eigenvectors are new axes ordered by importance. The eigenvalues measure how much variance each axis captures.

Keeping only the top $k$ eigenvectors compresses $d$ -dimensional data into $k$ dimensions with minimal information loss. PCA is a linear method — it works on flat subspaces. This limitation motivates the next lecture. Singular value decomposition handles rectangular matrices, low-rank approximations, and generalizes eigen decomposition to non-square matrices.

Real-World & Domain Connection: In image compression, eigenfaces represent faces with ~50 numbers instead of tens of thousands of pixels — the top eigenvectors correspond to lighting direction, head orientation, and expression. In genomics, PCA on 20,000+ genes measured on a few hundred patients reduces dimensions before clustering disease subtypes — the top components often match known biological pathways. In finance, PCA on asset return covariance matrices separates systematic risk (market factor, first component) from diversifiable risk (tail components). In signal processing, reconstructing from top components yields a denoised signal — used in EEG artifact removal and audio separation.

6.5 Computing Eigenvalues and Eigenvectors

Hook: You know what eigenvalues and eigenvectors mean geometrically. Now you need to find them from a given matrix. The answer is the characteristic equation. One equation in one unknown — solve a polynomial, then solve a linear system. This is the procedure you will use on every exam and in every application from PCA to PageRank.

6.5.1 The Characteristic Equation

The Procedure: An eigenvalue decomposition turns a matrix-vector problem into two simpler problems: first solve a polynomial, then solve a linear system. Start with

A x = λ x

and rearrange:

A x A x - λ x (A - λ I) x = λ x = 0 = 0 (the defining equation) (move all terms to one side) (factor, but note: λ must multiply I)

Now factor out

x

. But

λ

is a scalar, and you cannot subtract a scalar from a matrix. You need to multiply

λ

by the identity matrix

I

so the subtraction is matrix-minus-matrix:

(A - λ I) x = 0

For this equation to hold with a non-zero

x

, the matrix

(A - λ I)

must be singular. If

(A - λ I)

were invertible, you could multiply both sides by its inverse and get

x = 0

— the trivial solution. You need the non-trivial one. A singular matrix has zero determinant. This gives the characteristic equation:

det (A - λ I) = 0

Solve this for

λ

, and you get the eigenvalues. Once you have each eigenvalue, plug it back into

(A - λ I) x = 0

and solve for

x

to get the eigenvectors. Key idea — decoupling: You have one equation

(A - λ I) x = 0

with two unknowns (

λ

and

x

). The determinant condition gives you an equation in

λ

alone — the characteristic polynomial. Solve the polynomial first for

λ

, then solve the linear system for

x

. The determinant condition is what decouples the problem. Symbol Registry

Symbol	Meaning	Type	Shape
$A - λ I$	Characteristic matrix	Matrix	$n \times n$
$det (A - λ I)$	Characteristic polynomial in $λ$	Scalar	Scalar
$I$	Identity matrix	Matrix	$n \times n$
$0$	Zero vector	Vector	$n \times 1$

Shortcut for triangular matrices. If

A

is upper or lower triangular, the eigenvalues are simply the diagonal entries. The determinant of a triangular matrix is the product of the diagonal entries, so

det (A - λ I) = (a_{11} - λ) (a_{22} - λ) \dots (a_{nn} - λ) = 0

. Each diagonal entry is an eigenvalue. Use this shortcut when you can — it saves time on exams.

6.5.2 Why the Identity Matrix?

A scalar $λ$ cannot be directly subtracted from a matrix. Think of it in simple terms. You have a matrix $[2546]$ . You want to subtract $λ$ . You need to subtract $λ$ from each diagonal entry — not from every entry. The identity matrix $I$ places $λ$ only on the diagonal and leaves the off-diagonal entries untouched.

A - λ I = [a_{11} - λ a_{21} a_{12} a_{22} - λ]

If you used a matrix of all ones instead, you would subtract $λ$ from every entry, which would destroy the transformation's structure. The identity matrix preserves the matrix while only adjusting the diagonal. It is the same idea as multiplying a scalar by 1 in ordinary algebra. The identity matrix is the "1" of matrix multiplication. It does not change anything it multiplies.

Q: Why do we use the identity matrix when multiplying by lambda? A: Lambda is a scalar — just a number. You cannot subtract a number from a matrix term-by-term. The identity matrix places lambda only on the diagonal entries. This subtracts lambda from the diagonal elements of

A

and leaves the off-diagonal elements unchanged. That is the only way to make the subtraction dimensionally valid while preserving the structure of

A

. Algebraically,

A - λ

has no meaning. But

A - λ I

is a valid matrix difference. If you used any other matrix, you would change values that should not change.

6.5.4 Full Worked Example

Let

A = [- 4 3 - 6 5]

. Find the eigenvalues and eigenvectors. Step 1: Form $A - λ I$ .

A - λ I = [- 4 - λ 3 - 6 5 - λ]

Step 2: Set the determinant to zero and solve for $λ$ . The determinant of a 2×2 matrix is

a d - b c

det (A - λ I) = (- 4 - λ) (5 - λ) - (- 6) (3) = 0

(- 4 - λ) (5 - λ) + 18 = 0

Expand the product term by term:

(- 4) (5) (- 4) (- λ) (- λ) (5) (- λ) (- λ) = - 20 = + 4 λ = - 5 λ = + λ^{2}

Sum all terms and add 18:

- 20 + 4 λ - 5 λ + λ^{2} + 18 = 0

Combine like terms:

λ^{2} - λ - 2 = 0

Factor:

(λ - 2) (λ + 1) = 0

The eigenvalues are $λ_{1} = 2$ and $λ_{2} = - 1$ . Pause — check your polynomial before factoring. A common mistake is a sign error in the expansion. Verify by plugging one eigenvalue back: for

λ = 2

, does

det (A - 2 I) = 0

A - 2 I = [- 6 3 - 6 3]

, determinant =

(- 6) (3) - (- 6) (3) = - 18 + 18 = 0

. Correct. Step 3: Find the eigenvector for $λ_{1} = 2$ . Substitute

λ = 2

into

(A - λ I) x = 0

[- 4 - 2 3 - 6 5 - 2] [x_{1} x_{2}] = [- 6 3 - 6 3] [x_{1} x_{2}] = [00]

This gives two equations:

- 6 x_{1} - 6 x_{2} = 0 \Rightarrow x_{1} + x_{2} = 0

3 x_{1} + 3 x_{2} = 0 \Rightarrow x_{1} + x_{2} = 0

Both equations are the same — the system is dependent. This always happens when you plug in an eigenvalue: the rows become multiples of each other. That is the whole point. The matrix is singular, so the rows are linearly dependent. Set

x_{2} = 1

. Then

x_{1} = - 1

. One eigenvector is

[- 1 1]

. Any scalar multiple is also an eigenvector:

[- 6 6]

[8 - 8]

[- 9 9]

. Plug in any value of

r

. You get another eigenvector on the same axis. The eigenvector is the direction, not a single point. Step 4: Find the eigenvector for $λ_{2} = - 1$ . Substitute

λ = - 1

[- 4 - (- 1) 3 - 6 5 - (- 1)] = [- 3 3 - 6 6]

This gives:

- 3 x_{1} - 6 x_{2} = 0 \Rightarrow x_{1} + 2 x_{2} = 0

3 x_{1} + 6 x_{2} = 0 \Rightarrow x_{1} + 2 x_{2} = 0

Set

x_{2} = 1

. Then

x_{1} = - 2

. The eigenvector is

[- 2 1]

. Result: Eigenvalues: 2 and -1. Eigenvector directions: along

[- 1 1]

and

[- 2 1]

. Sense-check: Multiply

A

by each eigenvector to verify:

A [- 1 1] = [- 4 3 - 6 5] [- 1 1] = [4 - 6 - 3 + 5] = [- 2 2] = 2 [- 1 1] ✓ A [- 2 1] = [- 4 3 - 6 5] [- 2 1] = [8 - 6 - 6 + 5] = [2 - 1] = - 1 [- 2 1] ✓

Both satisfy

A x = λ x

6.5.5 Why Set the Determinant to Zero?

The reasoning is geometric. From $(A - λ I) x = 0$ , there are two possibilities: either $x = 0$ (trivial, not useful) or $(A - λ I)$ is singular. A singular matrix collapses space onto a lower dimension.

When you are on the eigenvector axis, the concept of area vanishes — there is only a line. A line has no area. So the determinant must be zero. Setting $det (A - λ I) = 0$ says: find the $λ$ values that collapse the space onto a line. That line is the eigenvector direction.

The characteristic polynomial $det (A - λ I)$ is always degree $n$ for an $n \times n$ matrix. For $2 \times 2$ you get a quadratic — find its roots. For $3 \times 3$ you get a cubic — factor or use the rational root theorem. Some roots may be repeated (repeated eigenvalues) or complex. Complex eigenvalues come in conjugate pairs and correspond to rotations in the transformation.

Q: Why do we set determinant equal to zero? Why not just solve

(A - λ I) x = 0

directly? A: Because

x

is unknown and

λ

is unknown. You have one equation with two unknowns. Setting the determinant to zero gives you an equation in

λ

alone — the characteristic polynomial. Solve that for

λ

first. Then, for each

λ

, the system

(A - λ I) x = 0

becomes a linear system you can solve for

x

. The determinant condition is what decouples the problem. Geometrically, you are looking for the

λ

values that collapse the space onto the eigenvector line — and collapse means zero determinant. Algebraically, if

(A - λ I)

were invertible, you could left-multiply by its inverse and get

x = 0

. That is the trivial solution you do not want.

Common Mistakes:

Forgetting the identity matrix. Writing $A - λ$ instead of $A - λ I$ . A scalar cannot be subtracted from a matrix — the dimensions do not match.
Sign errors in determinant expansion. Especially with the minus signs from $(- λ)$ . Write out each term of the product explicitly before combining.
Forgetting that eigenvectors are not unique. Any non-zero scalar multiple of an eigenvector is also an eigenvector. Graders will accept any correct scalar multiple unless a specific form is requested.
Dividing both sides wrongly when solving for eigenvectors. The system $(A - λ I) x = 0$ is dependent by design. Do not try to invert the matrix — it is singular. Express one variable in terms of the other, pick a convenient value, and read off the eigenvector.

6.5.7 The Complete Procedure (Summary)

Step	Action	What you get
1	Form $A - λ I$	The characteristic matrix
2	Compute $det (A - λ I)$ and set to zero	The characteristic polynomial
3	Solve the polynomial for $λ$	The eigenvalues
4	For each $λ$ , substitute into $(A - λ I) x = 0$	A dependent linear system
5	Solve the linear system to express $x$	The eigenvector direction

Verification step. Always multiply $A$ by your eigenvector. The result must equal $λ$ times the eigenvector. This quick check catches sign mistakes and mis-factored quadratics.

Real-World & Domain Connection: The characteristic equation method is not just a classroom exercise. In structural engineering, eigenvalue problems determine the natural frequencies of buildings and bridges — the $λ$ values tell engineers which frequencies to avoid to prevent resonance. In Google's PageRank algorithm, the dominant eigenvector of the web's link matrix (found by solving a giant characteristic equation numerically) ranks every webpage. In quantum mechanics, the Schrödinger equation is an eigenvalue problem where $λ$ gives the possible energy levels of a system. In machine learning, PCA solves the eigenvalue problem of the covariance matrix to find the principal components — the directions that capture the most variance in your data.

6.6 Computing Determinants via Minors and Cofactors

Hook: The 2×2 determinant formula

a d - b c

is simple. But what about 3×3? 10×10? There is no one-line shortcut. Instead, you break a large determinant into smaller ones — recursively — until you hit a 2×2 at the bottom. This is cofactor expansion, and it is the universal computational method for determinants of any size.

6.6.1 Why Minors and Cofactors

For matrices larger than $2 \times 2$ , the determinant is computed recursively using minors and cofactors. The formula $det (A) = a d - b c$ only works for $2 \times 2$ . For $3 \times 3$ and larger, you break the matrix down into smaller pieces.

6.6.2 Minor of an Element

The minor

M_{ij}

is the determinant of the submatrix you get by deleting row

i

and column

j

from the original matrix. For a

3 \times 3

matrix

A = 26947105811

, the minor

M_{11}

comes from deleting row 1 and column 1.

M_{11} = det [710811] = 7 (11) - 8 (10) = 77 - 80 = - 3

If you stand at position

(1, 1)

— value

2

— you eliminate that row and column. What remains is the

2 \times 2

submatrix

[710811]

. Its determinant is the minor.

6.6.3 Cofactor

The cofactor

C_{ij}

adds a sign based on the position. The sign pattern follows

(- 1)^{i + j}

C_{ij} = (- 1)^{i + j} M_{ij}

For position

(1, 1)

: the exponent is

1 + 1 = 2

, so

(- 1)^{2} = + 1

. The cofactor equals the minor. For position

(1, 2)

: the exponent is

1 + 2 = 3

, so

(- 1)^{3} = - 1

. The cofactor is the negative of the minor. Think of a checkerboard pattern starting with

+

at the top-left:

+, -, +, -

alternating across rows and columns. This sign pattern is the most common source of errors — get it right and the rest is arithmetic. Symbol Registry

Symbol	Meaning	Type	Shape
$M_{ij}$	Minor: determinant after deleting row i, column j	Scalar	Scalar
$C_{ij}$	Cofactor: signed minor, $(- 1)^{i + j} M_{ij}$	Scalar	Scalar
$a_{ij}$	Entry of A at row i, column j	Scalar	Scalar

6.6.4 Determinant by Cofactor Expansion

The determinant is the sum, along any row or column, of each entry multiplied by its cofactor.

det (A) = j = 1 \sum n a_{ij} C_{ij} (expansion along row i)

The column version works identically:

det (A) = \sum_{i = 1}^{n} a_{ij} C_{ij}

. This is a recursive formula. For a

3 \times 3

matrix, each cofactor requires computing a

2 \times 2

determinant. For a

4 \times 4

matrix, each cofactor requires a

3 \times 3

determinant, which itself needs

2 \times 2

determinants. The recursion bottoms out at

2 \times 2

, where

det [a c b d] = a d - b c

. Strategic shortcut: Always choose a row or column with the most zeros. Each zero entry wipes out a term — no need to compute the cofactor. Fewer nonzero entries means fewer cofactors to compute. Scan all rows and columns before expanding to find the path with fewest computations.

6.6.5 Worked Example — 3×3 Determinant

Compute the determinant of

A = 3 - 2 5 1 - 4 4 03 - 2

. Expand along row 1 (which has a zero at

a_{13}

). Entry $a_{11} = 3$ : Delete row 1, column 1. The submatrix is

[- 4 4 3 - 2]

M_{11} Sign: Contribution: = (- 4) (- 2) - (3) (4) = 8 - 12 = - 4 1 + 1 = 2 (even) \to C_{11} = + M_{11} = - 4 3 \times (- 4) = - 12

Entry $a_{12} = 1$ : Delete row 1, column 2. The submatrix is

[- 2 5 3 - 2]

M_{12} Sign: Contribution: = (- 2) (- 2) - (3) (5) = 4 - 15 = - 11 1 + 2 = 3 (odd) \to C_{12} = - M_{12} = - (- 11) = 11 1 \times 11 = 11

Entry $a_{13} = 0$ : Zero times anything is zero. Contribution: 0.

det (A) = 3 (- 4) + 1 (11) + 0 = - 12 + 11 = - 1

The determinant is $- 1$ . Sense-check: A nonzero determinant (

det (A) = - 1 \neq = 0

) confirms the matrix is invertible. The negative sign means the transformation reverses orientation. If

det (A)

were

0

, the matrix would be singular — the rows or columns would be linearly dependent.

Common Pitfalls:

Sign errors on $(- 1)^{i + j}$ . The checkerboard sign is the single most frequent mistake. Many students compute the minor perfectly and then forget the sign. Pause after each cofactor and verify: is $i + j$ even or odd?
Wrong submatrix. Deleting an incorrect row or column produces a different $2 \times 2$ determinant. Double-check your deletion — the submatrix must exclude all entries that share a row or column with the current entry.
Mixing rows mid-expansion. Once you pick a row, use only that row. The formula sums $a_{ij} C_{ij}$ for a fixed $i$ . Do not pull entries from different rows.
Forgetting the product rule: $det (A B) = det (A) det (B)$ and $det (A^{- 1}) = 1/ det (A)$ . Swapping two rows flips the sign. Scaling a row by $k$ scales $det$ by $k$ .

Visual Intuition: Picture a $3 \times 3$ matrix as a cube of numbers. The cofactor expansion along row 1 is like slicing the cube into three $2 \times 2$ faces, computing each face's area, and summing them with alternating signs. The checkerboard pattern tells you which faces flip orientation. The final sum — one number — is the signed volume of the parallelepiped spanned by the three column vectors of the matrix.

Real-World & Domain Connection: Cofactor expansion is the algorithmic backbone of determinant computation in every linear algebra library. In computer graphics, the $3 \times 3$ determinant appears in the cross product formula. That formula computes surface normals for lighting and collision detection.

In statistics, the determinant of a covariance matrix appears in the multivariate Gaussian density function. It is computed via cofactor expansion or LU decomposition. A near-zero determinant warns of multicollinearity — features that are nearly linearly dependent. In robotics, the Jacobian determinant determines whether a robot arm can move freely. It also shows when the arm has hit a singularity where one direction of motion is lost.

6.7 Exam Guidance Summary

Exam note: At least one 5-mark question will come from eigen decomposition (this lecture) or singular value decomposition (next lecture). This subject typically yields lower marks than other subjects because it is one of the toughest in the semester.

The characteristic equation method — $det (A - λ I) = 0$ — is the standard approach for computing eigenvalues and eigenvectors of $2 \times 2$ and $3 \times 3$ matrices. Know it cold. You solve a quadratic ( $2 \times 2$ ) or cubic ( $3 \times 3$ ) for eigenvalues, then solve linear systems for eigenvectors.
Determinant computation via minors and cofactors is tested on $3 \times 3$ matrices. Know the checkerboard sign pattern. For larger matrices, computational tools are used in practice — understand the recursive process conceptually.
Eigenvalues and eigenvectors are the most important concept in linear algebra for machine learning. They are foundational for PCA and dimensionality reduction. The next lecture on singular value decomposition generalizes this idea to non-square matrices — expect exam crossover between these two topics.

6.7.1 Common Exam Mistakes to Avoid

Top exam mistakes flagged by the professor:

Forgetting to multiply $λ$ by the identity matrix. Writing $A - λ$ directly is dimensionally wrong — you are trying to subtract a scalar from a matrix. Always write $A - λ I$ .
Sign errors in the cofactor $(- 1)^{i + j}$ . Use the checkerboard pattern: positive at $(1, 1)$ , negative at $(1, 2)$ , positive at $(1, 3)$ , alternating. A wrong sign cascades — the whole determinant is wrong.
Forgetting to verify your answer. Always plug your eigenvalues and eigenvectors back into $A x = λ x$ . If it does not check, you made an arithmetic error somewhere. This quick check catches sign mistakes and mis-factored quadratics.
Not recognizing triangular matrices. If the matrix is upper or lower triangular, the eigenvalues are simply the diagonal entries. Do not waste time computing the characteristic polynomial when you can read eigenvalues directly off the diagonal.

6.7.2 Key Facts for Exams

$det (A) = 0$ $⟺$ matrix is singular $⟺$ columns are linearly dependent $⟺$ matrix is not invertible $⟺$ space collapses to lower dimension
$det (A B) = det (A) det (B)$ , $det (A^{T}) = det (A)$ , $det (A^{- 1}) = 1/ det (A)$
For a $2 \times 2$ rotation matrix, the determinant is always 1 — rotation preserves area
Eigenvalues of a covariance matrix are always non-negative (positive semi-definite)
The determinant equals the product of all eigenvalues: $det (A) = λ_{1} λ_{2} \dots λ_{n}$
An $n \times n$ matrix has up to $n$ eigenvectors; eigenvectors are entire directions (axes), not single vectors
A negative eigenvalue means the vector flips direction along the same axis
2D rotation matrices have no real eigenvectors; their eigenvalues are complex conjugates

6.8 Key Industry Applications

6.8.1 Dimensionality Reduction (PCA)

Eigen decomposition of the data covariance matrix finds the directions of maximum variance. The eigenvectors (principal components) are new orthogonal axes ordered by importance. The eigenvalues tell you how much variance each axis captures. You project high-dimensional data onto the top few principal components and discard the rest. This compresses data with minimal information loss — going from hundreds of features to a handful while retaining 95%+ of the variance. PCA is used everywhere in ML pipelines: as a preprocessing step before clustering or classification, as a visualization tool (projecting to 2D/3D for plotting), and as a noise filter (discarding low-variance components).

6.8.2 Subscription Prediction

Streaming platforms like Spotify use user features to predict subscription likelihood. These features include age, gender, location, listening habits, device type, and session length. With hundreds of features, many are correlated. For example, time spent listening correlates with time spent active.

PCA reduces hundreds of features to a handful of principal components before modeling. This makes the prediction model faster to train. It is also less prone to overfitting and easier to interpret. You can inspect which original features dominate each principal component.

6.8.3 Image Compression

Eigenvectors help identify the most important pixel combinations in an image. The eigenfaces technique represents faces with ~50 numbers instead of tens of thousands of pixels — keeping only the top eigenvector projections. The dropped components correspond to fine texture detail that carries little visual information. The same principle applies to general image compression: SVD (generalized eigen decomposition) decomposes an image matrix into rank-1 layers, and keeping only the top few layers gives an excellent low-rank approximation.

6.8.4 Feature Engineering

Correlated features (like age and experience) get replaced by uncorrelated axes from eigenvectors — the principal components. These new axes capture the same information more compactly. The new features are linear combinations of the originals. The eigenvalues tell you how important each new feature is.

This decorrelation helps linear models, which assume independent features. It also speeds up gradient-based optimization, which slows down when features are correlated.

6.8.5 More Applications

Finance: PCA on asset return matrices separates systematic market risk (first component) from diversifiable risk.
Genomics: PCA on gene expression data (20,000+ genes) finds biological pathway signals before disease clustering.
PageRank: Google's original ranking algorithm computes the dominant eigenvector of the web's link matrix.
Vibration analysis: Natural frequencies of structures are eigenvalues of the stiffness matrix — engineers avoid resonance frequencies to prevent collapse.
Recommender systems: SVD (next lecture) factors user-item rating matrices into low-rank embeddings for collaborative filtering.

MFML Lecture 06 notes · Eigenvalues, Eigenvectors, and Determinants

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-09

Sections Breakdown

1Cauchy-Schwarz Inequality

A bound on the inner product of two vectors, with geometric origin via the dot product and cosine.

2Determinants — Geometric Interpretation

The determinant as the area/volume scaling factor of a linear transformation, including zero-determinant collapse.

3Eigenvalues and Eigenvectors — Geometric Intuition

Special directions along which a matrix acts like pure scaling, with physical metaphors and real-world connections.

4Applications in Machine Learning — Dimensionality Reduction

How eigenvectors of the covariance matrix power PCA and compress high-dimensional data.

5Computing Eigenvalues and Eigenvectors

The characteristic equation det(A - lambda I) = 0 and the full worked procedure.

6Computing Determinants via Minors and Cofactors

Recursive cofactor expansion for 3x3 and larger determinants, with a worked example.

7Exam Guidance Summary

Common exam mistakes and key facts for eigenvalues, eigenvectors, and determinants.

8Key Industry Applications

PCA, subscription prediction, image compression, feature engineering, and additional applications.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Cauchy-Schwarz Inequality

Must-know: The absolute inner product of two vectors can never exceed the product of their lengths. It bounds the normalized inner product between $- 1$ and $1$ — exactly the cosine of the angle between them. It is a proof tool, rarely computed directly in ML.

$∣ ⟨ x, y ⟩ ∣ \leq ∥ x ∥ ∥ y ∥$

⚠️ Top pitfall: Forgetting the absolute value, or confusing the general inner product $⟨ x, y ⟩$ with the dot product $x \cdot y$ . Cauchy-Schwarz holds for any valid inner product, not just the dot product.

Self-check: If two vectors have lengths 3 and 4, what is the largest possible value of their inner product?

Connects to: Determinant (both are bounds on geometric quantities), Eigenvalues (inner products appear in covariance matrices for PCA).

Determinant — Geometric Scaling Factor

Must-know: The determinant is the single scalar that measures how much a linear transformation scales area (2D), volume (3D), or hyper-volume ( $n$ D). A zero determinant means the space collapses — the matrix is singular and non-invertible.

$det (A) = a_{11} a_{22} - a_{12} a_{21} (2 \times 2), Area (A (R)) = ∣ det (A) ∣ \cdot Area (R)$

⚠️ Top pitfall: Thinking $det (A) = 0$ means every output is the zero vector. It does not — the matrix maps the space onto a lower-dimensional subspace. Also, $det (A) = 0 ⟺$ columns linearly dependent $⟺$ not invertible $⟺$ $A x = 0$ has a non-trivial solution.

Self-check: A pure rotation matrix has determinant $+ 1$ . Why does its determinant not change even though the shape is "turned"?

Connects to: Eigenvalues (the determinant equals the product of all eigenvalues), Minors and Cofactors (how to compute it for larger matrices).

Eigenvalues and Eigenvectors

Must-know: An eigenvector is a direction that survives a transformation unchanged except for scaling. The eigenvalue $λ$ is that scale factor. The defining equation is $A x = λ x$ , with $x \neq = 0$ . Eigenvectors are entire axes, not single arrows.

$A x = λ x, det (A) = λ_{1} λ_{2} \dots λ_{n}$

⚠️ Top pitfall: Assuming a vector must stretch to be an eigenvector. Shrinking ( $0 < λ < 1$ ), staying the same ( $λ = 1$ ), or flipping ( $λ < 0$ ) all count. Only a change in direction disqualifies a vector. A 2D rotation has no real eigenvectors.

Self-check: If $det (A) = 0$ , must all eigenvalues be zero? Explain why at least one is.

Connects to: Characteristic Equation (how to find them), PCA (eigenvectors of the covariance matrix), Determinant (product of eigenvalues).

Principal Component Analysis (PCA)

Must-know: PCA finds the directions of maximum variance in data by eigendecomposing the covariance matrix. The eigenvectors are new axes ordered by importance; the eigenvalues say how much variance each captures. Keep the top $k$ to compress $d$ dimensions into $k$ .

$Σ = \frac{1}{m} X ~^{T} X ~, Σ v_{i} = λ_{i} v_{i}, Z = X ~ V_{k}, R^{2} (k) = \frac{\sum _{i = 1}^{k} λ _{i}}{\sum _{i = 1}^{d} λ _{i}}$

⚠️ Top pitfall: Forgetting to standardize features before PCA. A feature on a large numeric scale dominates one on a small scale purely by magnitude, distorting the covariance matrix. Also, eigenvalues of a covariance matrix are always non-negative.

Self-check: You keep the top 3 of 100 principal components and retain 99% of variance. Roughly what fraction of information did you discard?

Connects to: Eigenvalues and Eigenvectors (the engine of PCA), Singular Value Decomposition (next lecture generalizes this to rectangular matrices).

Characteristic Equation

Must-know: To find eigenvalues, solve $det (A - λ I) = 0$ . This decouples the problem: first solve the characteristic polynomial for $λ$ , then plug each $λ$ into $(A - λ I) x = 0$ to get the eigenvector direction.

$det (A - λ I) = 0$

⚠️ Top pitfall: Writing $A - λ$ instead of $A - λ I$ . You cannot subtract a scalar from a matrix — the identity matrix places $λ$ only on the diagonal. Also, for triangular matrices, read eigenvalues straight off the diagonal and skip the polynomial.

Self-check: After finding an eigenvalue $λ = 2$ and eigenvector $x$ , how do you verify your answer in one line?

Connects to: Eigenvalues and Eigenvectors (the geometric meaning), Determinant (the determinant is set to zero), Minors and Cofactors (used inside the characteristic polynomial for 3 $\times$ 3).

Minors and Cofactors

Must-know: For matrices larger than $2 \times 2$ , compute the determinant recursively. The minor $M_{ij}$ is the determinant after deleting row $i$ , column $j$ . The cofactor adds the checkerboard sign $(- 1)^{i + j}$ . Expand along any row or column.

$C_{ij} = (- 1)^{i + j} M_{ij}, det (A) = j = 1 \sum n a_{ij} C_{ij}$

⚠️ Top pitfall: Sign errors on the checkerboard $(- 1)^{i + j}$ . The most common mistake is computing the minor perfectly then forgetting the sign. Always choose a row or column with the most zeros to minimize work.

Self-check: In a $3 \times 3$ matrix, what is the sign of the cofactor $C_{23}$ ?

Connects to: Determinant (this is how you compute it), Characteristic Equation (cofactor expansion builds the characteristic polynomial).

Previous LectureLecture 05 · Dot Products and Inner Products

Next LectureLecture 07 · Matrix Decompositions, Vector Spaces, and Determinants

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Eigenvalues, Eigenvectors, and Determinants

6.1 Cauchy-Schwarz Inequality

6.1.1 Definition and Explanation

6.1.2 Geometric Origin via the Dot Product

6.1.3 Practical Relevance

6.2 Determinants — Geometric Interpretation

6.2.1 The Transformation View

6.2.2 Area in Two Dimensions

6.2.3 Volume in Three Dimensions and Beyond

6.2.4 Determinant of Zero — Dimensional Collapse

6.2.5 Student Questions and Answers

6.3 Eigenvalues and Eigenvectors — Geometric Intuition

6.3.1 Hook — What Survives the Transformation?

6.3.2 Intuition and Analogy — Three Physical Metaphors

6.3.3 The Core Equation — Formal Definition

6.3.4 Worked Example

6.3.5 Assumptions and Scope

6.3.6 Visual Intuition — Axes, Scaling, and Rotation

6.3.7 Comparison — Determinant vs Eigenvalue

6.3.8 Common Pitfalls — Student Questions

6.3.9 Recap and Bridge

6.3.10 Real-World Connections

6.4 Applications in Machine Learning — Dimensionality Reduction

6.4.1 Hook: The Curse of 100 Columns

6.4.2 Intuition + Analogy: Finding the Grain of the Wood

6.4.3 Why Eigenvectors Matter in ML

6.4.4 The Variance Intuition

6.4.5 Worked Intuition: Age vs Experience

6.4.6 Eigenvectors as Principal Axes of Variance

6.4.7 Formalize: PCA Step by Step

6.4.8 Dimensionality Reduction by Truncation

Student Questions and Answers

6.5 Computing Eigenvalues and Eigenvectors

6.5.1 The Characteristic Equation

6.5.2 Why the Identity Matrix?

6.5.4 Full Worked Example

6.5.5 Why Set the Determinant to Zero?

6.5.7 The Complete Procedure (Summary)

6.6 Computing Determinants via Minors and Cofactors

6.6.1 Why Minors and Cofactors

6.6.2 Minor of an Element

6.6.3 Cofactor

6.6.4 Determinant by Cofactor Expansion

6.6.5 Worked Example — 3×3 Determinant

6.7 Exam Guidance Summary

6.7.1 Common Exam Mistakes to Avoid

6.7.2 Key Facts for Exams

6.8 Key Industry Applications

6.8.1 Dimensionality Reduction (PCA)

6.8.2 Subscription Prediction

6.8.3 Image Compression

6.8.4 Feature Engineering

6.8.5 More Applications

MFML Lecture 06 notes · Eigenvalues, Eigenvectors, and Determinants

Sections Breakdown

Exam Revision Notes

Cauchy-Schwarz Inequality

Determinant — Geometric Scaling Factor

Eigenvalues and Eigenvectors

Principal Component Analysis (PCA)

Characteristic Equation

Minors and Cofactors

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat