Mathematical Foundations for Machine Learning

Lecture 1: Introduction to Linear Algebra

📅 Published: 2026-06-24

🎓 Level: postgraduate

👥 Audience: Postgraduate students in machine learning, data science, and AI

The subject is "slightly difficult" compared to the other courses in this programme. But the instructor promises: if you stay with the visual, interactive approach and the companion documents, you will do well. Quizzes and assignments are where you control your marks. Submit them on time. In a previous offering, two exam questions appeared directly from the practice set.

1.1 What Is Linear Algebra

Hook

You live in a 3D world. You can point to a spot on a map. You can picture a corner of a room. But what if your data has 500 dimensions — 500 features describing every house in a city? Your brain cannot draw that. So how do you work with it?

Intuition + Analogy

Linear algebra is a tool for visualizing, manipulating, and doing mathematics with data that has more than two or three dimensions. Think of it as a "spreadsheet language." A spreadsheet has rows (items) and columns (features). Linear algebra takes that same idea and turns it into a system of geometry and algebra that works for any number of columns — 2, 15, 500, or millions. Everyday analogy — the architect's blueprint: An architect looks at a 2D blueprint and "sees" the 3D building. Linear algebra is your blueprint for N-dimensional data. You cannot see 500-dimensional space, but you can draw conclusions about it — distances, patterns, clusters — using the same rules that work in 2D and 3D. Where the analogy breaks: a blueprint is a fixed 2D-to-3D mapping. Linear algebra generalises the rules themselves to any dimension.

Formalize

Linear algebra is the study of vectors, matrices, and the linear transformations between vector spaces. It is built from exactly two operations: 1. Scalar multiplication — multiplying a vector by a number (scaling it). 2. Vector addition — adding two vectors together by adding their corresponding components. These two operations combine to form linear combinations, which embed every other concept in linear algebra. A transformation

T

is linear if it respects both operations:

T (u + v) T (c \cdot v) = T (u) + T (v) = c \cdot T (v) (additivity) (homogeneity)

A nonlinear transformation breaks at least one of these. Linear algebra studies only the linear ones. Notation note: Throughout this course, scalars are lowercase (

c, k

), vectors are bold lowercase (

v, x

), and matrices are bold uppercase (

A, M

). This matches the instructor's notation.

Worked Example

Is $f (x) = x^{2}$ linear? Test the two conditions with

u = 2, v = 3

: - Additivity:

f (2 + 3) = 5^{2} = 25

. But

f (2) + f (3) = 4 + 9 = 13

. Not equal. - Homogeneity:

f (3 \cdot 2) = 6^{2} = 36

. But

3 \cdot f (2) = 3 \cdot 4 = 12

. Not equal. Answer:

x^{2}

is nonlinear. A linear function must be of the form

f (x) = k x

— straight line through the origin. Sense-check: graphs of linear functions are straight lines;

x^{2}

is a parabola.

Real-World Picture

The transformer architecture inside every large language model (GPT, Gemini, Claude) is built entirely from matrix multiplications. Every attention head, every feed-forward layer — it is all $A X = B$ at scale. The math in this course is the same math that runs the world's most powerful AI systems.

Visual Intuition

Imagine a 2D coordinate grid printed on a rubber sheet. The grid lines form equal-sized unit squares. A linear transformation is any stretch, shear, rotation, or combination that keeps grid lines parallel and equidistant after the deformation. A nonlinear transformation would make grid lines curve, converge, or spread unevenly — like pressing your thumb into the rubber.

Pitfalls

- "Linear" means more than "straight line." In everyday English,

y = 2 x + 3

is a "linear equation" because its graph is a line. In linear algebra, that is an affine function, not a linear one — it fails homogeneity because

f (0) = 3 \neq = 0

. True linear functions must pass through the origin. - Confusing dimension with physical space. Dimension in linear algebra just means "number of features." A vector with 100 components lives in 100-dimensional space. It has nothing to do with physical length, width, and height. - Thinking you need to "see" it. You do not need to picture 500 dimensions. You trust the algebra. The 2D and 3D geometric pictures are training wheels — they teach the rules that generalise to any dimension.

Recap + Bridge

Linear algebra is the language of N-dimensional data — built from just scalar multiplication and vector addition. Next: the fundamental data object in this language — the vector.

Domain Connection

Every real-world entity a machine learning system encounters — a customer, an image, a sentence, a genome — gets converted into a vector before any algorithm touches it. Linear algebra is not a prerequisite for ML; it is the operating system of ML.

1.2 Vectors

Hook

How does a bank decide whether to approve your loan? It does not look at you. It looks at your vector — hundreds of numbers: salary, years at current job, existing debt, credit score, age, address stability. You are a point in N-dimensional space. The bank's algorithm checks which cluster of points you land near.

Intuition + Analogy

A vector has exactly two properties: magnitude (how long it is) and direction (which way it points). This is the same definition used in physics — a force, a velocity, a displacement. In data science, the "force" is replaced by a "data point," but the geometry is identical. Everyday analogy — the shopping list: A shopping list is a vector. Each row is a dimension (apples, bread, milk). The number next to each item is the coordinate. Your list: (3 apples, 1 bread, 2 milk). Your neighbour's list: (0 apples, 2 bread, 5 milk). The difference between these lists tells you who is baking and who is making smoothies. Comparing vectors is comparing shopping lists — dimension by dimension. Where the analogy breaks: a shopping list has no natural notion of "magnitude" in a geometric sense, though we can compute one mathematically using all the quantities.

Formalize

A vector

v \in R^{n}

is an ordered list of

n

real numbers:

v = v_{1} v_{2} ⋮ v_{n}

Each component

v_{i}

is the coordinate along the

i

-th dimension (the

i

-th feature). The vector's two defining properties: - Magnitude (length, norm):

∥ v ∥ = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2}

— the straight-line distance from the origin to the point. - Direction: the angle the vector makes with the positive X-axis (in 2D:

θ = arctan (v_{2} / v_{1})

). In higher dimensions, direction is captured by the unit vector

v^= v /∥ v ∥

. A position vector is a vector drawn from the origin

(0, 0, \dots, 0)

to the point

(v_{1}, v_{2}, \dots, v_{n})

. It "dictates the position" of that point in the coordinate system.

Worked Example

House A as a 2D vector: square footage = 1,800 sq ft, price = 3 crores (30 million INR).

v_{A} = [18003]

Magnitude:

∥ v_{A} ∥ = 180 0^{2} + 3^{2} = 3, 240, 000 + 9 = 3, 240, 009 \approx 1800.0025

. The magnitude is dominated by the square-footage dimension because the units are not on the same scale — a critical observation that motivates normalization (covered later in the course). Direction (angle from X-axis):

θ = arctan (3/1800) \approx 0.09 5^{\circ}

— almost flat along the X-axis, because square footage has a much larger numerical value than price in crores. House B: 2,500 sq ft, 10 crores →

v_{B} = [250010]

. Difference:

v_{B} - v_{A} = [2500 - 1800 10 - 3] = [7007]

. Sense-check: House B has 700 more sq ft and is 7 crores more expensive. The difference vector points in the direction of "more expensive, larger houses."

Real-World Picture

Loan approval systems in every Indian bank (HDFC, ICICI, SBI) use exactly this framework. Each applicant is an N-dimensional vector. Historical data — vectors labelled "approved" or "rejected" — train a classifier. A new applicant's vector is compared against the clusters. Near the "rejected" cluster → higher risk. This is the geometric intuition behind logistic regression, SVMs, and neural network classifiers.

Visual Intuition

Plot a 2D graph with "Bank Balance" on the X-axis and "Years of Experience" on the Y-axis. Each applicant is a dot. Colour approved applicants green and rejected applicants red. You will see two rough clusters. The decision boundary — the line (or curve) separating green from red — is what the ML algorithm learns. A new applicant appears as a gray dot. If it falls on the red side of the boundary, the model predicts rejection.

Pitfalls

- Ignoring scale differences across dimensions. If square footage is in thousands and price is in single-digit crores, the magnitude is dominated by square footage. Always ask: are my features on comparable scales? Normalization fixes this. - Thinking vectors are only about numbers. A vector is a geometric object with length and direction. The list of numbers is just one representation (in the standard basis). The vector itself is independent of the coordinate system. - Confusing a point with a vector. A point

(3, 4)

is a location. The position vector

[34]

is the arrow from the origin to that point. They contain the same information, but the vector carries the idea of magnitude and direction — which points alone do not.

Recap + Bridge

A vector is an arrow in N-dimensional space — defined by magnitude and direction. Every data point in ML is a vector. When you have many data points, you need a way to store them all together — that is a matrix.

Domain Connection

Word embeddings (Word2Vec, GloVe, BERT embeddings) represent every word in a vocabulary as a 300-dimensional (or larger) vector. The famous relationship $v_{king} - v_{man} + v_{woman} \approx v_{queen}$ is vector arithmetic in an embedding space. Every modern NLP system starts here.

1.3 Matrices

Hook

A single house vector tells you about one house. But what if you have 10,000 houses, each with 50 features (sq ft, price, bedrooms, bathrooms, age, location, school district, crime rate...)? You need a way to store all of them and operate on all of them at once.

Intuition + Analogy

A matrix is a 2D grid of numbers — a stack of vectors. Think of it as a spreadsheet. Each row is one entity (one house, one customer). Each column is one feature (price, age, income). The whole spreadsheet is the matrix. When you apply a formula to an entire column in Excel, you are doing the kind of operation that linear algebra generalises to any number of rows and columns simultaneously. Everyday analogy — the class photograph: A class photograph arranges students in rows and columns. Each row is a student (a vector of facial features). Each column position captures the same feature across all students. The photograph itself is the matrix. When you squint, you see the whole class — that is what a matrix lets you do: see all your data at once.

Formalize

A matrix

A \in R^{m \times n}

has

m

rows and

n

columns. The entry in row

i

, column

j

A_{ij}

(or

a_{ij}

A = a_{11} a_{21} ⋮ a_{m 1} a_{12} a_{22} ⋮ a_{m 2} \dots \dots ⋱ \dots a_{1 n} a_{2 n} ⋮ a_{mn}

- Each row is a row vector of length

n

(one data point with

n

features). - Each column is a column vector of length

m

(one feature across all

m

data points). - A matrix with

m = n

is a square matrix. - The main diagonal of a square matrix consists of entries

a_{11}, a_{22}, \dots, a_{nn}

. Matrix as data storage: If you have

m

houses and each house is described by

n

features, stacking them as rows gives an

m \times n

matrix. This is the standard data-representation convention in machine learning: rows = samples, columns = features.

Worked Example

Two houses, two features: square footage (sq ft) and price (crores INR).

A = [18002500310]

- Row 1 = House A: 1,800 sq ft, 3 crores. - Row 2 = House B: 2,500 sq ft, 10 crores. - Column 1 = square footage for all houses:

(1800, 2500)

. - Column 2 = price for all houses:

(3, 10)

. Add a third house (1,200 sq ft, 1.5 crores) and a third feature (bedrooms):

A = 180025001200 310 1.5 241

Now

A \in R^{3 \times 3}

. Each row is still one house. Columns: sq ft, price, bedrooms. Sense-check: the shape tells you exactly how many entities and how many features you have.

Real-World Picture

In computer vision, a grayscale image is a matrix of pixel intensities. A 1080p image is a $1920 \times 1080$ matrix of numbers between 0 (black) and 255 (white). Every convolution in a CNN is a matrix operation sliding across this pixel matrix. A colour image is three such matrices stacked (RGB channels) — a 3D tensor, which is the generalisation of a matrix to more than two dimensions.

Visual Intuition

Picture a spreadsheet with rows labelled "House 1, House 2, House 3" and columns labelled "Sq Ft, Price, Bedrooms." The matrix is that grid of numbers. Now imagine that instead of just 3 houses and 3 features, you have 10,000 rows and 500 columns. The matrix is too large to display, but the same rules apply — you can still extract any row (one house vector), any column (one feature across all houses), or compute operations across the entire grid.

Pitfalls

- Mixing up rows and columns. Always check: is my data stacked as rows (one row per sample) or columns (one column per sample)? The instructor uses rows-as-samples. Most ML libraries (scikit-learn, PyTorch) use rows-as-samples. But some textbooks stack as columns. Read the shape:

m \times n

means

m

rows,

n

columns. - Thinking a matrix is "just a table." A matrix is also a transformation — it can stretch, shear, rotate, or project the space. The next section explains this. If you think of a matrix only as a storage container, you miss half of linear algebra.

Recap + Bridge

A matrix is a 2D grid where rows are data points and columns are features — a "spreadsheet" for N-dimensional data. But a matrix is more than storage: it is also a transformation machine.

Domain Connection

In collaborative filtering (the algorithm behind Netflix and Amazon recommendations), the user-item rating matrix has rows = users, columns = movies, and entries = ratings. Matrix factorisation techniques (SVD, the topic of a later lecture) decompose this matrix to discover latent features — genres, styles, preferences — that drive recommendations.

1.4 Matrices as Transforming Agents: $A X = B$

Hook

Take the number 3. Multiply it by 2. You get 6. The scalar 2 scaled 3. Now take a vector. Multiply it by a matrix. The vector moves — its length changes, its direction changes. The matrix is not just scaling it. The matrix is transforming the entire space it lives in.

Intuition + Analogy

A X = B

is the single most important equation in linear algebra. When

A

is a scalar and

X

is a scalar,

A

simply scales

X

. When

A

is a matrix and

X

is a vector,

A

transforms

X

into a new vector

B

. The matrix applies a "force" that displaces the vector from its original position to a new position. Everyday analogy — the rubber sheet: Draw a dot on a rubber sheet. Now pull the sheet diagonally from two opposite corners. The dot moves. Its distance from the origin changes. Its direction changes. The matrix

A

is the description of exactly how you pulled the sheet.

X

is where the dot started.

B

is where it ended up after the pull. Different matrices = different ways of deforming the rubber sheet (stretching, shearing, rotating). Where the analogy breaks: a physical rubber sheet cannot "project" — you cannot flatten a real rubber sheet into a line. But a projection matrix mathematically collapses dimensions.

Formalize

Given a matrix

A \in R^{m \times n}

and a vector

X \in R^{n}

(written as a column), the product

B = A X

produces a vector

B \in R^{m}

B_{i} = j = 1 \sum n A_{ij} X_{j} for each row i = 1, \dots, m

In full:

a_{11} a_{21} ⋮ a_{m 1} a_{12} a_{22} ⋮ a_{m 2} \dots \dots ⋱ \dots a_{1 n} a_{2 n} ⋮ a_{mn} x_{1} x_{2} ⋮ x_{n} = a_{11} x_{1} + a_{12} x_{2} + \dots + a_{1 n} x_{n} a_{21} x_{1} + a_{22} x_{2} + \dots + a_{2 n} x_{n} ⋮ a_{m 1} x_{1} + a_{m 2} x_{2} + \dots + a_{mn} x_{n}

Shape rule:

A

m \times n

X

n \times 1

, so

B

m \times 1

. The number of columns of

A

must match the number of rows of

X

. This is the fundamental compatibility condition for matrix-vector multiplication. Two ways to interpret the product $A X$ : 1. Row picture: Each row of

A

computes one component of

B

via a weighted sum (dot product with

X

). 2. Column picture:

B

is a linear combination of the columns of

A

, where the coefficients are the entries of

X

. That is,

B = x_{1} \cdot (column 1) + x_{2} \cdot (column 2) + \dots + x_{n} \cdot (column n)

Worked Example

Simple $2 \times 2$ case:

A = [1224], X = [12]

Row computation: - Row 1:

1 \cdot 1 + 2 \cdot 2 = 1 + 4 = 5

- Row 2:

2 \cdot 1 + 4 \cdot 2 = 2 + 8 = 10

B = [510]

The original vector

(1, 2)

has been displaced to

(5, 10)

. Both magnitude and direction changed. Column picture verification:

B = 1 \cdot [12] + 2 \cdot [24] = [12] + [48] = [510]

. ✓ House matrix example:

A = [18002500310], X = [510]

- Row 1:

1800 \cdot 5 + 3 \cdot 10 = 9000 + 30 = 9030

- Row 2:

2500 \cdot 5 + 10 \cdot 10 = 12500 + 100 = 12600

B = [903012600]

Sense-check:

X = (5, 10)

is a vector where the second component is twice the first. The output

(9030, 12600)

does not preserve this ratio — the matrix changed the direction.

Real-World Picture

Every layer of a neural network is exactly $A X = B$ (followed by a nonlinear activation). The weight matrix $W$ transforms the input vector $x$ (the activations from the previous layer) into the pre-activation vector $z = W x$ . The "learning" in deep learning is finding the right entries of $W$ so that the transformations cascade into the correct prediction. A GPT model with 175 billion parameters is, at its core, an enormous stack of matrix multiplications.

Visual Intuition

Picture a 2D grid with a vector drawn from the origin to $(1, 2)$ . Now apply the matrix $A = [1224]$ . The entire grid deforms. The vector $(1, 2)$ slides to $(5, 10)$ . The unit square of area 1 in the original grid becomes a parallelogram — its area is the determinant (covered next). The grid lines that were parallel remain parallel — confirming the transformation is linear. The takeaway: the matrix moves every point in the space, not just one vector.

Pitfalls

- Dimension mismatch. If

A

3 \times 2

and

X

3 \times 1

, the product

A X

is undefined — the inner dimensions (2 and 3) do not match. Always check: columns of

A

= rows of

X

. - Forgetting that $A X \neq = X A$ . Matrix multiplication is not commutative. Multiplying a vector on the left by a matrix (

A X

) is a column operation. Multiplying on the right (

X A

) is a row operation — and usually undefined unless dimensions align. - Treating the matrix as "just numbers." Every matrix

A

is a specific geometric transformation. The numbers encode a physical deformation of space. Always ask: what is this matrix doing to the space?

Recap + Bridge

A X = B

is the heart of linear algebra: the matrix

A

transforms the input vector

X

into the output vector

B

by deforming the entire coordinate space. But not all deformations are created equal — some are linear and some are not. Next: what makes a transformation linear.

Domain Connection

In computer graphics, every time you rotate a 3D model, zoom in, or pan across a scene, you are applying a sequence of $4 \times 4$ transformation matrices to homogeneous coordinates. The entire GPU rendering pipeline is built on $A X = B$ . The same math that moves a vector in your textbook moves the characters in a Pixar film.

1.5 Linear Transformations

Hook

You have a rubber sheet with a perfect square grid printed on it. You can stretch it, shear it, rotate it. But if you poke it with your finger — making a dimple — the grid lines near the dimple stop being parallel. Which of these deformations are "linear" and which are not? And why does it matter?

Intuition + Analogy

A linear transformation is any deformation of space that keeps grid lines parallel and equidistant after the transformation. Think of a deck of playing cards. Push the top card sideways — the deck becomes a slanted rectangular prism. The cards remain parallel. The spacing between them stays the same. That is a shear — a linear transformation. Now try to bend the deck into a C-shape. The cards are no longer parallel — that is nonlinear. Everyday analogy — the photocopier: A photocopier can enlarge (uniform scaling), stretch only horizontally (non-uniform scaling), or slightly skew an image (shear). It cannot make one half of the image bigger than the other half — that would break equidistance. The photocopier's built-in transformations are all linear. Photoshop's "liquify" tool, where you can push pixels around arbitrarily, is nonlinear.

Formalize

A transformation

T : R^{n} \to R^{m}

is linear if and only if, for all vectors

u, v

and all scalars

c

T (u + v) T (c \cdot v) = T (u) + T (v) = c \cdot T (v) (additivity) (homogeneity)

These two algebraic conditions are equivalent to the geometric property of preserving parallel and equidistant grid lines. Here is why: - Additivity ensures that the vector sum

u + v

(the diagonal of the parallelogram formed by

u

and

v

) maps to the diagonal of the parallelogram formed by

T (u)

and

T (v)

. A grid of parallelograms remains a grid of parallelograms — parallel lines stay parallel. - Homogeneity ensures that scaling a vector by

c

scales its image by the same factor

c

. Grid spacings that were equal remain equal — equidistance is preserved. Every linear transformation from

R^{n}

R^{m}

can be represented as a matrix multiplication

T (x) = A x

for some

m \times n

matrix

A

. Conversely, every matrix defines a linear transformation. This is the fundamental connection between matrices and transformations.

Worked Example

Is $T ([x y]) = [x + 2 y]$ linear? Test additivity with

u = [10], v = [01]

T (u + v) = T ([11]) = [31]

T (u) + T (v) = [30] + [21] = [51]

. Not equal. The transformation is not linear. The "+2" in the first component shifts everything right — it is a translation, which is an affine transformation, not a linear one. Sense-check: a linear transformation must send the origin to the origin.

T (0) = [20] \neq = 0

. Is $T ([x y]) = [2 x + y x - y]$ linear? This is

T (x) = A x

with

A = [21 1 - 1]

. Since it can be written as a matrix multiplication, it is linear. Answer: yes.

Real-World Picture

Principal Component Analysis (PCA), a core dimensionality reduction technique covered in Module 7, is a linear transformation. It rotates the data axes to align with the directions of maximum variance. Because the transformation is linear, distances and angles between data points are preserved — the structure of the data is maintained even as the coordinate system changes.

Visual Intuition

Draw a 2D grid of unit squares. Now apply a horizontal shear: $A = [1011]$ . Every vertical line tilts right by exactly the same angle. All unit squares become equal-area parallelograms. The X-axis labels on the plot: original grid (dashed gray), transformed grid (solid blue). Note a single vector in red: it starts at $(1, 1)$ and ends at $(2, 1)$ . The takeaway: linear = uniform deformation across the entire space. No localised stretching or bending.

Pitfalls

- Assuming all matrix transformations are linear. They are — by definition,

T (x) = A x

is always linear. The confusion is the reverse: not all functions that "transform" vectors are linear. Translation (

T (x) = x + b

with

b \neq = 0

) is not. - Forgetting the origin condition. A quick test: does

T (0) = 0

? If not, the transformation is not linear. This catches translation errors immediately. - Thinking "linear transformation" and "linear function" are the same. In high school,

y = m x + b

is called a "linear function." In linear algebra, only

y = m x

(with

b = 0

) qualifies. The high school version is affine.

Recap + Bridge

A linear transformation keeps grid lines parallel and equidistant — it deforms space uniformly. Every linear transformation is a matrix. Next: the number that tells you how much area (or volume) that transformation changes — the determinant.

Domain Connection

In reinforcement learning, the state transition from one timestep to the next in a linear dynamical system is a linear transformation: $s_{t + 1} = A s_{t} + B a_{t}$ . Understanding when this system is stable (the state does not blow up) depends on the eigenvalues of $A$ — concepts introduced later in this course. Robotics, control theory, and game AI all depend on linear dynamical systems.

1.6 Determinant

Hook

You have been computing determinants since class 11. You know the formula:

a d - b c

for a

2 \times 2

matrix. But if someone asks you "What is a determinant? What does it actually measure?" — could you answer without reaching for a formula?

Intuition + Analogy

The determinant of a matrix tells you by how much the area (or volume) between the grid lines changes when the matrix transforms the space. Everyday analogy — the pizza dough: Roll out a square of pizza dough. Its area is 1 unit. Now stretch it to twice its width. The dough is now a rectangle of area 2. The determinant of the stretching transformation is 2 — the factor by which every unit of area scaled. If you roll it back to the original square, the determinant is 1/2 — the inverse scaling. If you squash the dough flat into a thin line, its area goes to 0. Determinant = 0 — the dough lost a dimension. The determinant is not about the area of any specific shape. It is the uniform area scaling factor for the entire space. Every unit square in the input space multiplies its area by exactly

∣ det (A) ∣

in the output space.

Formalize

For a square matrix

A \in R^{n \times n}

, the determinant

det (A)

(also written

∣ A ∣

) is a scalar with the following geometric meaning: - In 2D:

det (A)

is the signed area of the parallelogram formed by the images of the standard basis vectors

e_{1} = (1, 0)

and

e_{2} = (0, 1)

under

A

. Equivalently, it is the factor by which any area in the input space is scaled. - In 3D:

det (A)

is the signed volume of the parallelepiped. - In

n

det (A)

is the signed

n

-dimensional hypervolume scaling factor. For a $2 \times 2$ matrix:

det [a c b d] = a d - b c

This is the area of the parallelogram spanned by the column vectors

[a c]

and

[b d]

. For a diagonal matrix:

det d_{1} 0 ⋮ 0 0 d_{2} ⋮ 0 \dots \dots ⋱ \dots 00 ⋮ d_{n} = d_{1} \cdot d_{2} \dots d_{n}

The determinant is the product of the stretch factors along each axis. Key properties: 1.

det (I) = 1

— the identity does nothing, area scales by 1. 2.

det (A B) = det (A) \cdot det (B)

— determinants multiply for composed transformations. 3.

det (A^{- 1}) = 1/ det (A)

— the inverse undoes the scaling. 4.

det (A) = 0

if and only if

A

is singular (non-invertible) — the transformation collapses a dimension. 5. Swapping two rows flips the sign of the determinant. 6. The determinant is n-linear as a function of rows (or columns): linear in each row when others are held fixed.

Worked Example

Example 1 — Identity:

A = [1001]

det (A) = 1 \cdot 1 - 0 \cdot 0 = 1

. The unit square stays a unit square. Area scaling factor = 1. Example 2 — Horizontal stretch:

A = [2001]

det (A) = 2 \cdot 1 - 0 \cdot 0 = 2

. A

1 \times 1

square (area 1) becomes a

2 \times 1

rectangle (area 2). The space is stretched horizontally by 2×. Example 3 — Rotation by 90°:

A = [01 - 1 0]

det (A) = 0 \cdot 0 - (- 1) \cdot 1 = 0 - (- 1) = 1

. A unit square rotates but its area stays 1. Example 4 — Shear:

A = [1011]

det (A) = 1 \cdot 1 - 1 \cdot 0 = 1

. A shear turns a square into a parallelogram of equal area (base × height unchanged). Example 5 — Projection (collapse):

A = [1224]

det (A) = 1 \cdot 4 - 2 \cdot 2 = 4 - 4 = 0

. The two columns are multiples of each other — the transformation collapses the entire 2D plane onto a single line. Area becomes 0. Sense-check: The second row is

2 \times

the first row, confirming linear dependence and zero determinant.

Real-World Picture

In 3D computer graphics, the determinant of a $3 \times 3$ transformation matrix tells you whether the transformation preserves or flips orientation. If $det (A) > 0$ , the transformation is orientation-preserving (a right hand stays a right hand). If $det (A) < 0$ , it is orientation-reversing (mirror reflection). Game engines check this to ensure objects do not get rendered inside-out.

Visual Intuition

Plot a 2D grid with one unit square highlighted in yellow (area = 1). Now apply $A = [20 0 1.5]$ . The yellow square becomes a $2 \times 1.5$ yellow rectangle (area = 3). The determinant display reads $det = 3$ . Now change $A$ to a projection matrix $[1224]$ . The entire grid collapses to a single diagonal line. The yellow square becomes a line segment on that line — area = 0. The determinant display reads $det = 0$ . The takeaway: determinant = area scaling factor. Zero = collapse.

Pitfalls

- Thinking the determinant is "the value of the matrix." A matrix does not have a "value." The determinant is a number derived from the matrix that measures area/volume scaling. This is the exact misconception the instructor called out. - Confusing $det (A + B)$ with $det (A) + det (B)$ . Determinants do not distribute over addition.

det (A + B) \neq = det (A) + det (B)

in general. - Assuming $det (A) = 0$ means "no solution." It means the matrix is singular — the system

A X = B

either has no solution or infinitely many solutions, depending on

B

. - Ignoring the sign. A negative determinant means the transformation includes a reflection — orientation is flipped. The absolute value

∣ det (A) ∣

is the area scaling factor. The sign tells you about orientation.

Recap + Bridge

The determinant is the factor by which a transformation scales area (2D), volume (3D), or hypervolume (nD).

det = 0

means the space collapsed — a dimension was lost. That loss of dimension is why singular matrices have no inverse.

Domain Connection

In probability and Bayesian inference, the determinant of the Jacobian matrix appears in change-of-variables formulas. When you transform a random variable $Y = g (X)$ , the probability density scales by $∣ det (J) ∣$ where $J$ is the Jacobian of $g$ . Normalising flows — a modern generative modelling technique — are built on composing invertible transformations whose Jacobian determinants are easy to compute.

1.7 Inverse

Hook

You apply a transformation to a vector. It moves. Now you want it back where it started. Can you always undo a transformation? What if the transformation squashed the entire 2D plane into a single line — can you "unsquash" a line back into a plane?

Intuition + Analogy

The inverse of a matrix, written

A^{- 1}

, undoes the transformation that

A

does. If

A

stretches the space horizontally by 2×, then

A^{- 1}

compresses it back by 1/2×. If

A

rotates the space by 45°, then

A^{- 1}

rotates it back by −45°. Everyday analogy — the Ctrl+Z (undo) button: Whatever you just did — typed, deleted, moved a file — Ctrl+Z reverses it. The inverse is the Ctrl+Z for matrix transformations. But there is a catch: if you delete a file and then empty the recycle bin, Ctrl+Z cannot bring it back. The information is destroyed. Similarly, if a matrix collapses a dimension (

det = 0

), the information in that dimension is destroyed — and no inverse can recover it. Where the analogy breaks: real-world undo has a limited history stack; a matrix inverse is a single perfect reversal (when it exists).

Formalize

Let

A

be an

n \times n

square matrix. The inverse

A^{- 1}

is the unique matrix satisfying:

A \cdot A^{- 1} = A^{- 1} \cdot A = I

where

I

is the

n \times n

identity matrix (1s on the diagonal, 0s elsewhere). Existence condition:

A^{- 1}

exists if and only if

det (A) \neq = 0

. Such a matrix is called invertible or non-singular. Why $det = 0$ kills the inverse: If

det (A) = 0

, the transformation collapses the

n

-dimensional input space into an output space of dimension

< n

. For example, a

2 \times 2

matrix with

det = 0

sends the entire 2D plane onto a 1D line (or a 0D point). From a point on that line, you cannot uniquely determine which original 2D point it came from — infinitely many input points map to the same output point. The transformation is many-to-one. No function can reverse a many-to-one mapping uniquely. For a $2 \times 2$ matrix:

A^{- 1} = \frac{1}{a d - b c} [d - c - b a] provided a d - b c \neq = 0

This formula makes the role of the determinant explicit: you divide by

det (A) = a d - b c

. If it is zero, division is impossible.

Worked Example

Invertible case:

A = [2003], det (A) = 6 \neq = 0

A^{- 1} = [1/2 0 0 1/3]

Check:

A A^{- 1} = [2 \cdot 1/2 0 0 3 \cdot 1/3] = [1001] = I

. ✓ Geometric meaning:

A

stretches X by 2× and Y by 3×.

A^{- 1}

compresses X by 1/2× and Y by 1/3×. The two transformations cancel perfectly. Non-invertible case:

A = [1224], det (A) = 0

There is no

A^{- 1}

. The transformation sends every vector

(x, y)

to a point on the line

y = 2 x

. From the output

(x + 2 y, 2 x + 4 y)

, you cannot recover the unique

(x, y)

that produced it — both

(1, 0)

and

(- 1, 1)

map to

(1, 2)

. Sense-check: two different inputs produce the same output — information was lost.

Real-World Picture

In GPS navigation, your position is solved from a system of linear equations involving satellite distances. The coefficient matrix (geometry of satellite positions) must be invertible to get a unique position fix. If satellites are poorly arranged (nearly collinear), the matrix becomes nearly singular — small measurement errors get massively amplified. This is the geometric dilution of precision (GDOP) problem, and it is a direct consequence of a matrix having a very small (but non-zero) determinant.

Visual Intuition

Imagine a 2D space with a gray vector $X$ at $(1, 1)$ . Apply $A$ (horizontal stretch 2×): the vector moves to $B = (2, 1)$ . Now apply $A^{- 1}$ (horizontal compression 1/2×): the vector slides back to $(1, 1)$ . The determinant changed from 1 to 2 (during stretch) and back from 2 to 1 (during compression). The space was deformed and then perfectly restored. You can do this interactively — move the vector, watch it deform, then hit "undo" and watch it return.

Pitfalls

- Thinking $(A + B)^{- 1} = A^{- 1} + B^{- 1}$ . This is false. The inverse of a sum is not the sum of inverses. Counterexample:

A = I, B = I

(I + I)^{- 1} = (2 I)^{- 1} = \frac{1}{2} I

, but

I^{- 1} + I^{- 1} = 2 I

. - Forgetting the order reversal for products.

(A B)^{- 1} = B^{- 1} A^{- 1}

, not

A^{- 1} B^{- 1}

. The undo operations must be applied in reverse order — last transformation undone first. - Confusing "singular" with "zero matrix." A singular matrix is not necessarily all zeros. The matrix

[1224]

is singular because its rows are linearly dependent (

det = 0

), not because its entries are zero.

Recap + Bridge

A^{- 1}

is the undo button — it perfectly reverses the transformation

A

applied. But undo only works if no information was destroyed:

det (A) \neq = 0

. Next: a way to measure how many dimensions survive a transformation — rank.

Domain Connection

In linear regression, the normal equation is $β^= (X^{T} X)^{- 1} X^{T} y$ . The matrix $X^{T} X$ must be invertible for a unique solution to exist. If features are collinear (one feature is a linear combination of others), $X^{T} X$ becomes singular — the regression coefficients are not uniquely determined. This is the multicollinearity problem, diagnosed by a determinant near zero.

1.8 Rank

Hook

2 \times 2

matrix can produce an output that is 2-dimensional (a plane), 1-dimensional (a line), or 0-dimensional (a single point — the origin). The same input space. Three possible fates. How do you know which one a given matrix will deliver?

Intuition + Analogy

The rank of a matrix is the dimension of the output space after the matrix transforms the input. It answers: "How many independent directions actually survive the transformation?" Everyday analogy — the document scanner: A scanner captures a 3D object (a page of a book) and produces a 2D image. The transformation from 3D to 2D has rank 2 — two dimensions survive. Now put the book flat against the scanner glass. The scanner still captures it as 2D — rank is still 2. But if you only scan a single line of text… you get a 1D output. Rank = 1. If you scan a completely blank page, you get a single colour value everywhere — rank = 0. Rank is the number of meaningful dimensions in the output. It cannot exceed the number of rows or columns of the matrix.

Formalize

The rank of a matrix

A \in R^{m \times n}

, denoted

rank (A)

, is the dimension of the column space (or equivalently, the row space) of

A

— the space of all possible outputs

A X

X

ranges over all vectors in

R^{n}

. Equivalently: -

rank (A)

= the number of pivot columns in the RREF of

A

. -

rank (A)

= the number of linearly independent rows (or columns) of

A

. -

rank (A) \leq min (m, n)

. When

rank (A) = min (m, n)

, the matrix has full rank. - For a square

n \times n

matrix: full rank (

n

) means

det \neq = 0

and invertible; rank

< n

means

det = 0

and singular. Geometric interpretation: - Rank =

n

(full rank for a square matrix): the transformation preserves all

n

dimensions. The output is

n

-dimensional. - Rank =

r < n

: the transformation collapses

n - r

dimensions. The output lies in an

r

-dimensional subspace (a line for

r = 1

, a plane for

r = 2

, etc.). - Rank = 0: the zero matrix — everything maps to the origin.

Worked Example

Rank 2 (full rank):

A = [1221], det (A) = 1 - 4 = - 3 \neq = 0

Take the house vector

v = [18003]

B = A v = [1 \cdot 1800 + 2 \cdot 3 2 \cdot 1800 + 1 \cdot 3] = [18063603]

The output is a 2-dimensional vector — two meaningful numbers. The input space was 2D. The output space is 2D.

rank (A) = 2

. Rank 1:

A = [1224], det (A) = 0

Every output vector lies on the line

y = 2 x

. For any input

(x, y)

, the output is

(x + 2 y, 2 x + 4 y) = (x + 2 y) \cdot [12]

— a scalar multiple of one direction vector. The output space is a 1D line.

rank (A) = 1

. Rank 0:

A = [0000]

Every input maps to

[00]

. The output space is a single point.

rank (A) = 0

. Sense-check for Rank 1: The second row is exactly

2 \times

the first — one row is redundant. The matrix only carries one independent direction of information, so rank = 1.

Real-World Picture

In dimensionality reduction (PCA, Module 7), you compute the rank of the data matrix to understand its intrinsic dimensionality. A dataset with 100 features might have an effective rank of 5 — meaning only 5 independent patterns exist; the other 95 features are linear combinations of those 5. PCA finds those 5 directions and projects the data onto them, reducing 100 dimensions to 5 without losing information.

Visual Intuition

Take a 2D input space (a plane of points). Apply a rank-2 matrix: the points stay spread across a 2D region — a distorted plane but still a plane. Apply a rank-1 matrix: every point collapses onto a single line through the origin. You can see it — the 2D scatter plot becomes a 1D line of dots. Apply a rank-0 matrix: every point goes to the origin — the entire scatter plot becomes one dot. The rank tells you the dimension of that final shape.

Pitfalls

- Confusing rank of a vector with rank of a matrix. A vector has no "rank" in this sense. Rank is a property of a matrix (or a linear transformation). Rank is about the space of all possible outputs, not a single output vector. - Thinking rank = number of non-zero rows. Not true in general. The matrix

[1224]

has two non-zero rows but rank 1. Rank is the number of linearly independent rows, which equals the number of pivot columns in RREF. - Assuming a $2 \times 2$ matrix always has rank 2. It can have rank 0, 1, or 2. Check the determinant:

det \neq = 0 ⟹

rank 2;

det = 0

but matrix is not all zeros

⟹

rank 1.

Recap + Bridge

Rank = the number of dimensions that survive the transformation. Full rank means nothing was lost. Reduced rank means the space collapsed. Zero rank means everything went to the origin. Next: the special directions that survive any transformation — eigenvectors.

Domain Connection

In collaborative filtering and recommender systems, the user-item rating matrix is typically low-rank — a few latent factors (genres, preferences) explain most of the variation in ratings. Matrix completion algorithms (like those behind the Netflix Prize winner) exploit low-rank structure to predict missing ratings from a sparse matrix.

1.9 Eigenvectors and Eigenvalues (Preview)

Hook

Apply a matrix to a vector. Usually, the vector changes both its length and its direction. But for every matrix, there are certain special vectors whose direction stays exactly the same — they only get stretched or squashed. What are they, and why are they the most important vectors in linear algebra?

Intuition + Analogy

Eigenvectors are the directions that survive a transformation unchanged in direction — they only scale. Eigenvalues are how much they scale. Think of a transformation as a force. Every force has a direction. The eigenvectors are those directions — the "principal axes" of the force. The eigenvalues tell you how strong the force is along each axis. Everyday analogy — the wind on a flag: Wind blows across a flag. The flag flutters in all directions. But there is one direction — straight downwind — where a thread on the flag only stretches (gets longer or shorter) without flapping sideways. That direction is an eigenvector. The wind speed in that direction is the eigenvalue. Crosswinds change direction continuously — those are not eigenvectors. This is a preview. The formal treatment with characteristic polynomials, diagonalization, and spectral theorems comes in Lectures 4 and 5.

Formalize

For a square matrix

A \in R^{n \times n}

, a non-zero vector

v \in R^{n}

is an eigenvector of

A

if:

A v = λ v

for some scalar

λ \in R

(or

C

), called the eigenvalue corresponding to

v

. Geometric meaning: The transformation

A

sends

v

to a scalar multiple of itself. The direction of

v

is unchanged (though it may flip if

λ < 0

). Only its length changes — by the factor

∣ λ ∣

. Standard form: The eigenvalues are the roots of the characteristic polynomial

det (A - λ I) = 0

. For each eigenvalue

λ

, the eigenvectors are the non-zero solutions to

(A - λ I) v = 0

. The instructor's description matches the standard form: "the matrix applies force, eigenvectors are the directions of force, eigenvalues are the amount of force" — this is a physical interpretation of

A v = λ v

. The "force" is the transformation; the eigenvectors are the directions along which the force acts without changing direction; the eigenvalues quantify the intensity of the force in each direction.

Worked Example

Preview example:

A = [2003]

This is a diagonal scaling matrix. The eigenvectors are the standard basis vectors: -

v_{1} = [10]

A v_{1} = [20] = 2 \cdot v_{1}

. Eigenvalue

λ_{1} = 2

. -

v_{2} = [01]

A v_{2} = [03] = 3 \cdot v_{2}

. Eigenvalue

λ_{2} = 3

. Sense-check: The matrix stretches X by 2× and Y by 3×. Vectors pointing purely along the X or Y axes only stretch — they do not rotate. That is exactly what eigenvectors are: the directions that do not rotate under the transformation.

Real-World Picture

Google's PageRank algorithm (the original Google search ranking) computes the principal eigenvector of the web's link matrix. The eigenvector's components are the PageRank scores — the importance of each page. The corresponding eigenvalue (always 1 for the stochastic matrix used) ensures convergence. Every time you search, you benefit from an eigenvector computation at massive scale.

Visual Intuition

Plot a 2D transformation. Draw many vectors from the origin in different directions. Apply the matrix. Most vectors rotate — their direction changes. But one or two vectors (the eigenvectors) only stretch or shrink along their original line. In the interactive application, these are displayed in a distinct colour (often red). Slide a parameter to change the matrix; watch the eigenvectors rotate to stay aligned with the principal axes of the transformation. The takeaway: eigenvectors are the "natural coordinate system" of the matrix.

Pitfalls

- Thinking every vector is an eigenvector. Only special directions are eigenvectors. For a generic

2 \times 2

matrix, there are at most two independent eigenvector directions. - Assuming eigenvectors must be unit length. The defining equation

A v = λ v

does not require

∥ v ∥ = 1

. If

v

is an eigenvector, so is

c v

for any

c \neq = 0

. We often normalise them to unit length for convenience. - Expecting real eigenvalues always. A rotation matrix

[01 - 1 0]

(90° rotation) has no real eigenvectors — every vector changes direction by 90°. Its eigenvalues are complex:

\pm i

. This is normal and covered in the full treatment (Lecture 4/5).

Recap + Bridge

Eigenvectors are the special directions that a transformation leaves unchanged in direction — it only scales them. Eigenvalues are the scaling factors. Full treatment in Lectures 4 and 5.

Domain Connection

In principal component analysis (PCA) (Module 7), the principal components are the eigenvectors of the data covariance matrix. The corresponding eigenvalues are the variances along each principal component. The eigenvector with the largest eigenvalue is the direction of maximum variance — the most "informative" direction in the data.

1.10 Types of Matrix Transformations

Hook

You now know that a matrix is a transformation and that the determinant measures area change. But not all transformations are created equal. Some stretch. Some shear. Some rotate. Some destroy dimensions entirely. Can you recognise each type just by looking at the matrix?

Intuition + Analogy

There are five fundamental types of linear transformations, each with a distinct geometric fingerprint. Think of them as the five basic "moves" you can do to a rubber sheet with a grid printed on it. Everyday analogy — photo editing tools: - Identity = opening the photo and doing nothing. - Stretch/Scale = dragging the corner handles to resize (uniform) or dragging one side only (non-uniform). - Shear = the "skew" tool — slanting the image sideways. - Rotation = the rotate tool — spinning the image around its centre. - Projection = flattening the image onto a single line — like "collapse all layers." Each tool corresponds to a specific matrix structure. Learn the matrix, and you can predict what the tool will do.

Formalize

1. Identity Matrix

I = [1001], det (I) = 1

Geometric effect: No transformation.

I X = X

for all

X

. Grid lines unchanged. Vector unchanged.

2. Stretching (Scaling) Matrix

S = [s_{x} 0 0 s_{y}], det (S) = s_{x} \cdot s_{y}

Geometric effect: Stretches the X-axis by factor

s_{x}

and the Y-axis by factor

s_{y}

. Grid lines remain parallel but spacing changes. A circle becomes an ellipse (if

s_{x} \neq = s_{y}

). The determinant is the product of stretch factors — the area of the stretched unit square. - Uniform scaling (

s_{x} = s_{y} = k

S = k I

. Shapes are preserved, just bigger. - Non-uniform scaling (

s_{x} \neq = s_{y}

): Shapes distort. A square becomes a rectangle.

3. Shear Matrix

H_{x} = [10 k 1] (horizontal shear), H_{y} = [1 k 01] (vertical shear)

det (H) = 1

Geometric effect: Pushes the top of the space sideways (

H_{x}

) or upwards (

H_{y}

) in proportion to the perpendicular coordinate. Grid lines become slanted but remain parallel. A rectangle becomes a parallelogram of equal area — that is why the determinant is 1.

4. Rotation Matrix

R_{θ} = [cos θ sin θ - sin θ cos θ], det (R_{θ}) = cos^{2} θ + sin^{2} θ = 1

Geometric effect: Rotates the entire space counter-clockwise by angle

θ

around the origin. Lengths preserved. Areas preserved. The determinant is always 1 for any pure rotation.

5. Projection Matrix

P = [1000] (projection onto X-axis), det (P) = 0

Geometric effect: Collapses the space onto a lower-dimensional subspace. Every point

(x, y)

maps to

(x, 0)

— its "shadow" on the X-axis. The Y-dimension is destroyed. Determinant = 0 because area collapses to zero.

Worked Example

Transformation gallery — apply each to $X = [11]$ :

Type	Matrix $A$	$B = A X$	$det (A)$	What happened?
Identity	$[1001]$	$[11]$	1	Nothing.
Stretch	$[2003]$	$[23]$	6	X stretched 2×, Y stretched 3×.
Shear (H)	$[1011]$	$[21]$	1	Shifted right; Y unchanged.
Rotate 90°	$[01 - 1 0]$	$[- 1 1]$	1	Rotated 90° CCW.
Project	$[1000]$	$[10]$	0	Collapsed to X-axis.

Sense-check for shear: The X-component changed from 1 to 2 (displaced by

k \cdot y = 1 \cdot 1

), but the Y-component stayed at 1. The area of the unit square before (1) equals the area of the resulting parallelogram after (base 1 × height 1 = 1). Determinant = 1. ✓

Real-World Picture

In computer vision, the geometric transformations between two images of the same scene (taken from different angles) are combinations of rotation, scaling, shear, and translation. The homography matrix (a $3 \times 3$ matrix in homogeneous coordinates) encodes all of these. Image stitching (panoramas), augmented reality, and camera calibration all rely on recognising and inverting these transformation matrices.

Visual Intuition

Open the interactive web application. Select "Identity" — the grid is a perfect checkerboard. Select "Horizontal Shear" — vertical lines tilt right; the more positive the Y-coordinate, the more they tilt. Watch one red vector: its tip slides right. The highlighted unit square becomes a parallelogram — same area. Now select "Projection onto X-axis" — the entire grid collapses to the X-axis line. All vectors become horizontal. The determinant display reads 0. The takeaway: each matrix type has an unmistakable visual signature.

Pitfalls

- Mixing up shear and rotation. A shear changes shape but preserves area (

det = 1

). A rotation preserves both shape and area (

det = 1

). The difference: a shear distorts angles; a rotation preserves them. - Thinking a projection is just "removing a column." A projection is a full

n \times n

matrix with rank

< n

. It is not the same as simply deleting a dimension from the vector — it actively maps every point onto the subspace. - Assuming all diagonal matrices are uniform scaling.

[2001]

is diagonal but non-uniform — it stretches only one axis. Uniform scaling requires all diagonal entries to be equal.

Recap + Bridge

There are five fundamental transformation types: identity (do nothing), stretch (scale axes), shear (slant), rotation (spin), and projection (collapse). Every matrix is a combination of these. Next: the two elementary operations that build all of this — scalar multiplication and vector addition.

Domain Connection

In deep learning, batch normalisation applies a learnable scaling (stretch) and shifting to each layer's activations. The scaling is a diagonal matrix multiplication. The shifting is an affine translation. Together they stabilise training by controlling the scale and mean of activations — a direct application of transformation types at industrial scale.

1.11 Fundamental Operations: Scalar Multiplication, Vector Addition, Linear Combinations

Hook

Everything you have learned so far — vectors, matrices, transformations, determinants, inverses, eigenvectors — is built from exactly two operations. Two. If you deeply understand these two, everything else follows.

Intuition + Analogy

The two fundamental operations of linear algebra are: 1. Scalar multiplication — take a vector and stretch or shrink it by a number. If the number is negative, flip its direction. Think of it as turning a volume knob: one number scales the whole thing up or down. 2. Vector addition — take two vectors and combine them tip-to-tail. Think of walking: you walk one block east (first vector), then one block north (second vector). Your final position is the sum. A linear combination is just doing both at once: scale the first vector, scale the second, add them. That is it. Matrix multiplication, determinants, solving equations — all are elaborate linear combinations. Everyday analogy — the recipe: A recipe is a linear combination. 2 cups flour + 1 cup sugar + 3 eggs = a cake batter. Each ingredient is a "dimension." The amounts are the scalar coefficients. The sum is the final product. Changing one coefficient changes the batter. A linear combination asks: what can I make by varying the amounts?

Formalize

Scalar Multiplication

For a scalar

c \in R

and a vector

v \in R^{n}

c \cdot v = c \cdot v_{1} v_{2} ⋮ v_{n} = c \cdot v_{1} c \cdot v_{2} ⋮ c \cdot v_{n}

Geometric meaning: The vector's length multiplies by

∣ c ∣

. If

c > 0

, direction is unchanged. If

c < 0

, direction reverses. If

c = 0

, the vector collapses to the origin.

Vector Addition

For vectors

u, v \in R^{n}

u + v = u_{1} u_{2} ⋮ u_{n} + v_{1} v_{2} ⋮ v_{n} = u_{1} + v_{1} u_{2} + v_{2} ⋮ u_{n} + v_{n}

Geometric meaning: The parallelogram rule: place the tail of

v

at the tip of

u

. The sum is the diagonal of the parallelogram formed by

u

and

v

Linear Combination

For scalars

c_{1}, c_{2}, \dots, c_{k}

and vectors

v_{1}, v_{2}, \dots, v_{k}

c_{1} v_{1} + c_{2} v_{2} + \dots + c_{k} v_{k} = i = 1 \sum k c_{i} v_{i}

This is the most general operation in linear algebra. Everything — matrix-vector multiplication, solving systems, finding inverses — reduces to computing linear combinations.

Worked Example

Scalar multiplication: House A:

v = [18003]

. Triple it:

3 \cdot v = 3 \cdot [18003] = [54009]

The vector now points to (5,400 sq ft, 9 crores). Every component tripled. The direction is the same (the angle

arctan (9/5400) = arctan (3/1800)

). Sense-check: multiplying a vector by a positive scalar only changes its length, not its direction. ✓ Negative scalar:

- 1 \cdot v = [- 1800 - 3]

. The vector now points in the exact opposite direction. Its magnitude is the same:

∥ - v ∥ = ∥ v ∥

. Vector addition:

v_{A} + v_{B} = [18003] + [250010] = [430013]

Linear combination with $c_{1} = 2, c_{2} = - 1$ :

2 v_{A} - v_{B} = 2 [18003] - [250010] = [3600 - 2500 6 - 10] = [1100 - 4]

Sense-check: the result has positive square footage but negative price — not a real house, but a valid vector. Linear combinations can produce vectors that do not correspond to physically meaningful data points, but they are mathematically essential for solving equations.

Real-World Picture

In portfolio theory (finance), your investment portfolio is a linear combination of asset vectors. Each asset is a vector of returns across different market scenarios. Your allocation percentages are the scalar coefficients. The resulting portfolio return vector is the linear combination. Markowitz's modern portfolio theory — finding the optimal coefficients to maximise return for a given risk — is a linear algebra optimisation problem.

Visual Intuition

Open the interactive linear combination tool. Two vectors $v_{1}$ (blue) and $v_{2}$ (green) are displayed. Two sliders control $c_{1}$ and $c_{2}$ . The resultant vector $c_{1} v_{1} + c_{2} v_{2}$ is shown in orange. Slide $c_{1}$ from 0 to 3: the orange vector stretches along the $v_{1}$ direction. Slide $c_{2}$ : it displaces from the tip of $c_{1} v_{1}$ . As you vary both sliders, the orange tip sweeps across the entire plane — every reachable point is the span. The takeaway: linear combinations let you reach anywhere in the span of your vectors.

Pitfalls

- Adding vectors component-wise without thinking geometrically. The component-wise rule is the how. The parallelogram rule is the what. Always visualise tip-to-tail addition alongside the algebraic computation. - Forgetting that vector addition requires the same dimension. You cannot add a 2D vector to a 3D vector. The dimensions must match — every component needs a partner. - Thinking scalar multiplication changes direction when $c > 0$ . A positive scalar changes only the length. Direction remains identical. Only negative scalars flip direction (

c < 0

) or collapse to the origin (

c = 0

Recap + Bridge

Scalar multiplication scales a vector. Vector addition combines two vectors. A linear combination does both. These two operations are the atoms of linear algebra — everything else is built from them. Next: the set of all points reachable via linear combinations — the span.

Domain Connection

In gradient descent (the optimisation algorithm that trains every neural network), the weight update is a linear combination: $w_{new} = w_{old} - η \nabla L$ . It scales the gradient vector (scalar multiplication by the learning rate $η$ ) and adds it to the current weight vector (vector addition). The entire training of a billion-parameter model is repeated linear combinations.

1.12 Span, Basis, and Dimension (Preview)

Hook

You have two vectors. By scaling each and adding them, you can reach some points in the plane — but can you reach every point? What if the two vectors point in exactly the same direction?

Intuition + Analogy

The span of a set of vectors is every point you can reach through linear combinations. If you have two arrows, their span is the set of all places you can get to by walking along the first arrow (any distance, forward or backward) and then along the second. Everyday analogy — the two-stick puzzle: You have two sticks of different lengths, lying on the floor in different directions. You can pick up each stick and lay it end-to-end with the other. The span is every spot on the floor you can touch by arranging both sticks end-to-end from a fixed starting point. If the sticks are not parallel, you can reach anywhere on the floor (span = the whole room). If they are parallel, you can only reach points along a single line (span = a line). If both sticks have zero length, you can only stay at the starting point (span = a point). A basis is a minimal set of sticks that spans the whole room. The dimension is how many sticks you need. This is covered fully in Lecture 2.

Formalize

Preview definitions (formal treatment in Lecture 2): - Span:

span {v_{1}, \dots, v_{k}} = {c_{1} v_{1} + \dots + c_{k} v_{k} ∣ c_{1}, \dots, c_{k} \in R}

. The set of all linear combinations. - Basis: A set of vectors that is linearly independent (no vector can be written as a linear combination of the others) and spans the entire space. - Dimension: The number of vectors in any basis for a given vector space. For

R^{n}

, the dimension is

n

Visual Intuition

In the interactive linear combination tool from Section 12: as you vary both sliders from $- \infty$ to $+ \infty$ , the orange vector's tip traces out the span. If $v_{1}$ and $v_{2}$ point in different directions, the orange tip covers the whole plane — span = $R^{2}$ . If they point in the same direction, the orange tip traces only a single line — span = $R^{1}$ (a line through the origin). The takeaway: the span is the reachable territory. The dimension of that territory is the number of truly independent directions you have.

Recap + Bridge

Span = all reachable points via linear combinations. Basis = minimal spanning set. Dimension = size of a basis. Full treatment in Lecture 2. Next: the final synthesis — what happens when a matrix is singular.

1.13 Collapsing Dimensions and Singular Matrices

Hook

A matrix can take a 2D plane and squash it into a 1D line. It can take a 3D volume and flatten it into a 2D shadow. Once squashed, you can never unsquash — the lost dimension is gone forever. Why does this happen, and what does it mean for solving equations?

Intuition + Analogy

A singular matrix is one whose determinant is zero. Its transformation collapses the input space into a lower-dimensional output space. Think of a hydraulic press crushing a 3D soda can into a 2D disc. The height dimension is destroyed. From the disc alone, you cannot reconstruct the original 3D can — every point on the disc could have come from any point along the vertical line above it. The information in the height dimension is permanently lost. Everyday analogy — the shadow: Hold a 3D object under a light. Its shadow on the wall is 2D. The shadow tells you the outline but not the depth. If I give you only the shadow, can you reconstruct the exact 3D shape? No — infinitely many 3D shapes cast the same 2D shadow. The projection from 3D to 2D is a singular transformation (

det = 0

). You cannot invert a shadow. This is why singular matrices have no inverse — you cannot recover what was lost.

Formalize

A square matrix

A \in R^{n \times n}

is singular (non-invertible) if

det (A) = 0

. Consequences of singularity: 1. Dimension collapse: The transformation sends

R^{n}

into a subspace of dimension

rank (A) < n

. At least one dimension is destroyed. 2. No inverse:

A^{- 1}

does not exist. The transformation is many-to-one. 3. The system $A X = B$ has either no solution (if

B

lies outside the collapsed subspace) or infinitely many solutions (if

B

lies inside it). It never has exactly one unique solution. In terms of systems of linear equations, a singular coefficient matrix corresponds to the "no solution" or "infinitely many solutions" cases. When the RREF reveals a row of the form

[00 \dots 0 ∣ 1]

, the system is inconsistent. When the RREF has free variables (fewer pivots than columns), there are infinitely many solutions — the solution set is a line, plane, or higher-dimensional subspace parameterised by the free variables.

Worked Example

Singular $2 \times 2$ matrix:

A = [1224], det (A) = 0

This matrix projects every point in

R^{2}

onto the line

y = 2 x

. Take two different input points: -

(1, 0) \mapsto (1, 2)

(- 1, 1) \mapsto (- 1 + 2, - 2 + 4) = (1, 2)

Both map to the same output. The transformation is 2-to-1 (actually

\infty

-to-1 — every point on a line perpendicular to

y = 2 x

maps to the same point on

y = 2 x

). No inverse can decide which of the infinitely many inputs produced a given output. System of equations interpretation:

x + 2 y 2 x + 4 y = 5 = 10

The second equation is just

2 \times

the first — it provides no new information. The system has infinitely many solutions: any point on the line

x + 2 y = 5

. The augmented matrix

[1224 ∣ ∣ 510]

reduces to

[1020 ∣ ∣ 50]

— one pivot, one free variable. Sense-check: the RREF has a row of all zeros (no contradiction) and a free variable (

y

), confirming infinitely many solutions.

Real-World Picture

In computed tomography (CT scans), the problem of reconstructing a 3D image from 2D X-ray projections is mathematically an inverse problem. The system matrix can be nearly singular, meaning small measurement noise gets massively amplified in the reconstruction. Regularisation techniques (Tikhonov, total variation) add constraints to make the system effectively invertible — they turn a singular problem into a non-singular one by imposing prior knowledge.

Visual Intuition

In the interactive application, select "Projection onto a line." Watch the 2D grid collapse — every vertical stack of grid intersections merges into a single point on the diagonal line. The entire rainbow of points across the plane concentrates onto one thin line. The determinant reads 0. You cannot "un-collapse" it — there is no slider, no button, no matrix that can recover the perpendicular information. The takeaway: singular = irreversible dimension loss.

Pitfalls

- Thinking "singular" means "zero matrix." A singular matrix can have all non-zero entries. The matrix

[1224]

is singular because its rows are linearly dependent, not because it contains zeros. - Assuming $det (A) = 0$ always means no solution for $A X = B$ . It depends on

B

. If

B

happens to lie in the column space of

A

, there are infinitely many solutions. If not, there are none. The three-case analysis (unique / none / infinite) is essential here. - Ignoring near-singular matrices. A matrix with

det \approx 1 0^{- 15}

is technically invertible, but numerically unstable. In practice, near-singular matrices cause algorithms to fail due to floating-point errors.

Recap + Bridge

A singular matrix (

det = 0

) collapses the space — dimensions are permanently lost, and the transformation cannot be undone. This is why inverses require non-zero determinants. This concludes Lecture 1. Lecture 2 picks up with span, basis, dimension, and the formal structure of vector spaces.

Domain Connection

In machine learning model training, the Hessian matrix (second derivative matrix of the loss function) becoming singular signals that the optimisation landscape has flat directions — parameters that can change without affecting the loss. This is a degeneracy that slows or prevents convergence. Newton-type optimisation methods require inverting the Hessian; near-singular Hessians require damping (Levenberg-Marquardt) to become numerically invertible.

Exam Guidance Summary

Topic	Importance	Key to remember
What is linear algebra	Medium	Built from scalar multiplication and vector addition; studies linear transformations
Vectors	High	Magnitude + direction; position vector = arrow from origin; data point = vector
Matrices	High	m × n shape; rows = samples, columns = features; also a transformation
$A X = B$ / matrices as transformations	Very High	Core concept; geometric interpretation; row picture vs. column picture
Linear transformations	High	Grid-line property: parallel and equidistant preserved; algebraic conditions
Determinant	Very High	Area/volume scaling factor; $det = 0$ means singular (dimension collapse)
Inverse	High	Undo button; exists only when $det \neq = 0$ ; $A A^{- 1} = I$
Rank	Medium	Dimension of output space; number of surviving independent directions
Eigenvectors and eigenvalues	Medium (preview)	Directions that only stretch; $A v = λ v$ ; full treatment in Lectures 4–5
Five transformation types	Medium	Identity, stretch, shear, rotation, projection — know each matrix form and determinant
Scalar ×, vector +, linear combos	Very High	The two atomic operations; everything else is built from these
Span, basis, dimension	Medium (preview)	Covered fully in Lecture 2

Study Advice

Stick to the companion PDF documents. They are the primary resource. Do not spread yourself thin across multiple textbooks.
Use the interactive web applications. The visual, hands-on approach is the intended learning method. Watching the geometry change in real time builds intuition that formulas alone cannot.
Practice problems are provided. In a previous offering, two exam questions appeared directly from the practice set. Do them all.
For deeper reading: Gilbert Strang's Introduction to Linear Algebra (MIT Press) is recommended for the first half of the course.
For interactive supplement: Interactive Linear Algebra (Georgia Tech) covers topics in the same visual style.
Maximise marks through quizzes and assignments. Submit everything on time. These are the areas where you have the most control.
The subject is "slightly difficult" — allocate study time accordingly. The instructor promises that staying with the companion documents and interactive tools will get you through.

Key Industry Applications

Application Area	Linear Algebra Concept Used
LLMs (Transformers)	Every attention head and feed-forward layer is $A X = B$ at massive scale
Loan/Credit Approval	Applicants are N-dimensional vectors; classification clusters separate approved from rejected
House Price Modelling	Houses are vectors; linear regression = solving $A X = B$
Recommendation Systems	User-item matrices; similarity via vector distances
Computer Vision	Images are pixel matrices; convolutions are matrix operations
PageRank (Google Search)	Principal eigenvector of the web link matrix
GPS Navigation	Solving systems of linear equations for position fix
Portfolio Theory	Portfolios are linear combinations of asset vectors
CT/MRI Reconstruction	Inverse problems with near-singular system matrices
PCA (Dimensionality Reduction)	Eigenvectors of covariance matrix; covered in Module 7

References

Strang, Gilbert. Introduction to Linear Algebra. Wellesley-Cambridge Press.
Interactive Linear Algebra (Georgia Tech) — Web-based visual textbook.
MLU Explain (Amazon Machine Learning University) — Interactive explanations of ML algorithms.
Polo Club Transformer Visualization — Interactive walkthrough of matrix math inside transformers.
Course Interactive Web Application — Custom-built for this course; covers vectors, transformations, determinants, eigenvectors, PCA.

MFML Lecture 01 notes · Introduction to Linear Algebra

Mathematical Foundations for Machine Learning· postgraduate· 2026-06-24

Summary

This lecture establishes linear algebra as the operating system of machine learning. It begins by defining linear algebra as the study of vectors, matrices, and linear transformations built from exactly two operations: scalar multiplication and vector addition. Vectors are introduced as geometric objects with magnitude and direction, serving as the universal data representation in ML. Matrices are presented both as data storage (rows = samples, columns = features) and as transformation machines. The central equation AX = B is explored through row and column pictures, showing how a matrix deforms the entire coordinate space. Linear transformations are formalised through additivity and homogeneity conditions, with the geometric interpretation of preserving parallel and equidistant grid lines. The determinant is revealed as the area/volume scaling factor of a transformation, with zero determinant signalling irreversible dimension collapse. The inverse is the undo operation, existing only when the determinant is non-zero. Rank measures how many dimensions survive a transformation. Eigenvectors and eigenvalues are previewed as the special directions that only stretch under a transformation. Five fundamental transformation types (identity, stretch, shear, rotation, projection) are catalogued with their matrix forms and geometric signatures. The lecture culminates by showing that singular matrices permanently destroy dimensions, making inversion impossible, and connects every concept to real-world ML applications: transformers, PCA, recommendation systems, computer vision, PageRank, and gradient descent.

Learning Objectives

1Define linear algebra and distinguish linear from nonlinear transformations using the two algebraic conditions

2Represent data points as vectors with magnitude and direction in N-dimensional space

3Store and manipulate datasets as matrices where rows are samples and columns are features

4Interpret matrix-vector multiplication AX = B as a geometric transformation of space

5Test whether a transformation is linear using additivity and homogeneity

6Compute the determinant and interpret it as the area/volume scaling factor of a transformation

7Determine when a matrix is invertible and compute the inverse for 2x2 matrices

8Calculate the rank of a matrix and relate it to the dimension of the output space

9Identify eigenvectors as directions that only stretch under a transformation

10Recognise the five fundamental transformation types from their matrix structure

Sections Breakdown

1What Is Linear Algebra

Definition, two fundamental operations, linearity conditions, and the blueprint analogy for high-dimensional data

2Vectors

Magnitude and direction, position vectors, data points as vectors, worked house-price examples, and the loan-approval geometric intuition

3Matrices

Row and column structure, matrices as spreadsheets, image representation, and the dual role of matrices as storage and transformations

4Matrices as Transforming Agents: AX = B

Row picture vs column picture, shape compatibility rules, neural network layers as AX = B, and the rubber-sheet analogy

5Linear Transformations

Additivity and homogeneity conditions, grid-line preservation property, translation as affine not linear, and PCA as a linear transformation

6Determinant

Area/volume scaling factor, ad-bc for 2x2, diagonal matrix product rule, zero determinant = singular, and Jacobian determinants in probability

7Inverse

Undo operation, existence condition det != 0, 2x2 formula, GPS positioning application, and why dimension collapse prevents inversion

8Rank

Dimension of output space, pivot columns in RREF, full rank vs rank-deficient, and low-rank structure in recommender systems

9Eigenvectors and Eigenvalues (Preview)

Directions that only stretch, characteristic polynomial preview, PageRank application, and the wind-on-a-flag analogy

10Types of Matrix Transformations

Identity, stretch, shear, rotation, and projection matrices with their geometric effects and determinants

11Fundamental Operations

Scalar multiplication, vector addition, linear combinations as the atoms of linear algebra, and gradient descent as repeated linear combinations

12Span, Basis, and Dimension (Preview)

Reachable points via linear combinations, minimal spanning sets, and the two-stick puzzle analogy

13Collapsing Dimensions and Singular Matrices

Irreversible dimension loss, many-to-one mappings, system-of-equations interpretation, and near-singular matrices in CT reconstruction

Postgraduate students in machine learning, data science, and AI who need a rigorous but intuitive foundation in linear algebra. Also suitable for engineers and practitioners who want to understand the mathematical engine behind modern ML systems.

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

What Is Linear Algebra

Must-know: Linear algebra is built from exactly two operations: scalar multiplication and vector addition. A transformation $T$ is linear iff $T (u + v) = T (u) + T (v)$ and $T (c v) = c T (v)$ . The function $f (x) = x^{2}$ is nonlinear; only $f (x) = k x$ (straight line through origin) is linear in the strict sense.

$T (u + v) T (c \cdot v) = T (u) + T (v) (additivity) = c \cdot T (v) (homogeneity)$

⚠️ Top pitfall: Confusing the high-school "linear function" $y = m x + b$ (which is affine) with the linear-algebra definition. In linear algebra, a linear function must pass through the origin: $T (0) = 0$ .

Self-check: Is $T (x) = 3 x + 5$ linear in the linear-algebra sense? Test $T (0)$ .

Connects to: Vectors, Linear Transformations, Fundamental Operations

Vectors

Must-know: A vector has exactly two properties: magnitude (norm) and direction. Every data point in ML is a vector in N-dimensional space. Magnitude: $∥ v ∥ = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2}$ . Direction in 2D: $θ = arctan (v_{2} / v_{1})$ ; in higher dimensions captured by the unit vector $v^= v /∥ v ∥$ .

$∥ v ∥ = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2}, v^= \frac{v}{∥ v ∥}$

⚠️ Top pitfall: Ignoring scale differences across dimensions. If one feature is in thousands and another in single digits, the magnitude is dominated by the larger-scale feature. Always normalise before comparing vector magnitudes.

Self-check: What are the two defining properties of any vector?

Connects to: Matrices, $A X = B$ , Fundamental Operations

Matrices

Must-know: A matrix $A \in R^{m \times n}$ has $m$ rows and $n$ columns. In ML convention: rows = samples, columns = features. Entry $A_{ij}$ is at row $i$ , column $j$ . A matrix is both a data container and a transformation machine.

$A = a_{11} a_{21} ⋮ a_{m 1} a_{12} a_{22} ⋮ a_{m 2} \dots \dots ⋱ \dots a_{1 n} a_{2 n} ⋮ a_{mn} \in R^{m \times n}$

⚠️ Top pitfall: Mixing up rows and columns. Always verify: is data stacked as rows (one row per sample) or columns? Check the shape: $m \times n$ means $m$ rows, $n$ columns. Most ML libraries use rows-as-samples.

Self-check: If you have 500 houses each described by 8 features, what is the shape of the data matrix?

Connects to: Vectors, $A X = B$ , Rank

$A X = B$ : Matrices as Transformations

Must-know: $A X = B$ is the core equation. The matrix $A$ transforms input vector $X$ into output vector $B$ . Row picture: each row of $A$ computes one component of $B$ via dot product. Column picture: $B$ is a linear combination of the columns of $A$ weighted by $X$ . Shape rule: $A$ is $m \times n$ , $X$ is $n \times 1$ , $B$ is $m \times 1$ .

$B_{i} = j = 1 \sum n A_{ij} X_{j}, B = x_{1} \cdot (col 1) + x_{2} \cdot (col 2) + \dots + x_{n} \cdot (col n)$

⚠️ Top pitfall: Dimension mismatch. If $A$ is $3 \times 2$ and $X$ is $3 \times 1$ , $A X$ is undefined. Always check: columns of $A$ must equal rows of $X$ . Also, $A X \neq = X A$ — matrix multiplication is not commutative.

Self-check: If $A$ is $4 \times 3$ and $X$ is $3 \times 1$ , what is the shape of $B = A X$ ?

Connects to: Vectors, Matrices, Linear Transformations, Determinant

Linear Transformations

Must-know: A transformation is linear iff it preserves parallel and equidistant grid lines. Algebraically: $T (u + v) = T (u) + T (v)$ and $T (c v) = c T (v)$ . Quick test: does $T (0) = 0$ ? If not, it is not linear. Every linear transformation can be written as $T (x) = A x$ for some matrix $A$ .

$T (u + v) = T (u) + T (v), T (c \cdot v) = c \cdot T (v)$

⚠️ Top pitfall: Assuming all matrix transformations are linear — they are. The confusion is the reverse: not all functions that transform vectors are linear. Translation $T (x) = x + b$ with $b \neq = 0$ is affine, not linear, because $T (0) = b \neq = 0$ .

Self-check: Is $T ([x, y]) = [x + 2, y]$ linear? Apply the $T (0)$ test.

Connects to: What Is Linear Algebra, $A X = B$ , Types of Matrix Transformations

Determinant

Must-know: The determinant is the factor by which a transformation scales area (2D), volume (3D), or hypervolume (nD). For a $2 \times 2$ matrix: $det = a d - b c$ . $det = 0$ means the matrix is singular — at least one dimension collapses. $det (I) = 1$ . $det (A B) = det (A) det (B)$ . $det (A^{- 1}) = 1/ det (A)$ .

$det [a c b d] = a d - b c, det (diag (d_{1}, \dots, d_{n})) = d_{1} d_{2} \dots d_{n}$

⚠️ Top pitfall: Thinking the determinant is "the value of the matrix." A matrix does not have a value. The determinant is a derived number that measures area/volume scaling. Also: $det (A + B) \neq = det (A) + det (B)$ in general.

Self-check: What is the determinant of a 90-degree rotation matrix $[01 - 1 0]$ ? What does the value tell you?

Connects to: Inverse, Rank, Collapsing Dimensions and Singular Matrices

Inverse

Must-know: $A^{- 1}$ undoes the transformation applied by $A$ : $A A^{- 1} = A^{- 1} A = I$ . Exists only when $det (A) \neq = 0$ . For $2 \times 2$ : $A^{- 1} = \frac{1}{a d - b c} [d - c - b a]$ . Singular matrices ( $det = 0$ ) have no inverse because they are many-to-one — information is destroyed.

$A^{- 1} = \frac{1}{a d - b c} [d - c - b a], provided a d - b c \neq = 0$

⚠️ Top pitfall: Thinking $(A + B)^{- 1} = A^{- 1} + B^{- 1}$ . This is false. Also, $(A B)^{- 1} = B^{- 1} A^{- 1}$ — the order reverses. Last transformation undone first.

Self-check: Why can the matrix $[1224]$ not be inverted?

Connects to: Determinant, Rank, Collapsing Dimensions and Singular Matrices

Rank

Must-know: Rank is the dimension of the output space after transformation — the number of independent directions that survive. $rank (A) \leq min (m, n)$ . Full rank means $rank = min (m, n)$ and $det \neq = 0$ (for square). Rank $< n$ means $det = 0$ and the matrix is singular. Rank 0 means the zero matrix.

$rank (A) = number of pivot columns in RREF (A) = dim (column space of A)$

⚠️ Top pitfall: Confusing rank of a vector with rank of a matrix. A vector has no rank in this sense. Also, rank is not the number of non-zero rows — $[1224]$ has two non-zero rows but rank 1.

Self-check: What are the possible ranks of a $2 \times 2$ matrix? Give an example of each.

Connects to: Matrices, Determinant, Inverse, Collapsing Dimensions and Singular Matrices

Eigenvectors and Eigenvalues (Preview)

Must-know: Eigenvectors are special directions that do not rotate under a transformation — they only stretch or shrink. The equation is $A v = λ v$ . $λ$ is the eigenvalue, the scaling factor. Diagonal matrices have eigenvectors along the coordinate axes. Full treatment in Lectures 4 and 5.

$A v = λ v, det (A - λ I) = 0$

⚠️ Top pitfall: Expecting real eigenvalues for every matrix. Rotation matrices (e.g. 90° rotation) have complex eigenvalues because every direction changes. Also, eigenvectors are not required to be unit length — any non-zero scalar multiple of an eigenvector is also an eigenvector.

Self-check: For the diagonal matrix $[2003]$ , what are the eigenvectors and eigenvalues?

Connects to: Determinant, Rank, Types of Matrix Transformations

Fundamental Operations

Must-know: Everything in linear algebra is built from two operations: scalar multiplication ( $c v$ scales a vector by $∣ c ∣$ , flips if $c < 0$ ) and vector addition ( $u + v$ via the parallelogram rule). A linear combination $c_{1} v_{1} + c_{2} v_{2} + \dots + c_{k} v_{k}$ combines both. Matrix multiplication, determinants, and solving systems all reduce to linear combinations.

$c \cdot v = c v_{1} c v_{2} ⋮ c v_{n}, u + v = u_{1} + v_{1} u_{2} + v_{2} ⋮ u_{n} + v_{n}, i = 1 \sum k c_{i} v_{i}$

⚠️ Top pitfall: Adding vectors of different dimensions — you cannot add a 2D vector to a 3D vector. Also, thinking positive scalar multiplication changes direction — it only changes length; direction stays the same.

Self-check: If $v = [18003]$ (house vector), compute $2 v$ , $- v$ , and $2 v + (- v)$ . What should the last result equal?

Connects to: Vectors, $A X = B$ , Span, Basis, and Dimension

Practice Quiz

Test your understanding of MFML Lecture 01 notes. Select an answer for each question — results are instant.

Which condition must a linear transformation satisfy?

What does a zero determinant mean geometrically?

Why can a singular matrix not be inverted?

In the equation AX = B, what does A represent geometrically?

Which matrix type has a determinant of 1 but changes the shape of objects?

Previous LectureLecture 17 · Support Vector Machines — Soft Margin, Non-Linear SVM, and Model Fairness

Next LectureLecture 02 · Solving Systems of Linear Equations

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

1.1 What Is Linear Algebra

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.2 Vectors

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.3 Matrices

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.4 Matrices as Transforming Agents: AX=B

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.5 Linear Transformations

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.6 Determinant

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.7 Inverse

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.8 Rank

Hook

Intuition + Analogy

Formalize

Worked Example

Real-World Picture

Visual Intuition

Pitfalls

Recap + Bridge

Domain Connection

1.4 Matrices as Transforming Agents: $A X = B$

$A X = B$ : Matrices as Transformations