Mathematical Foundations for Machine Learning

Vector Spaces, Linear Independence, Basis and Dimension

📅 Published: 2026-07-08

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Scalar Multiplication, Vector Addition, Linear Combinations — introduced in Lecture 1 (section 1.11).
Span, Basis, and Dimension (Preview) — introduced in Lecture 1 (section 1.12).
Gaussian Elimination and Row Echelon Form — covered in Lecture 2 (section 2.7), used here to test linear combinations.

Vector Spaces, Linear Independence, Basis and Dimension

4.1 Review of Groups

Why can you add any two integers and stay an integer, but you cannot multiply two whole numbers and always divide back? The answer sits in four simple rules that govern all of algebra. A group is the simplest mathematical structure with enough rules to be useful. Every vector space you will study builds on top of it.

4.1.1 What Is a Group

A group is a set of elements together with a single binary mathematical operation. Think of it as a self-contained club. The club has members — the elements — and one rule for combining any two members — the operation. For the club to be a group, it must obey four rules. Every single one must hold. If one fails, it is not a group.

Everyday analogy: a lock and key system. Imagine a set of keys, each with a unique shape. The operation is "try two keys in the same lock." Closure means every pair of keys produces some defined outcome. Associativity means the order you test three keys does not change the final result. The identity is a "dummy key" that, when paired with any key, leaves the outcome unchanged. The inverse is the partner key that, when paired, resets the system to the identity state.

Where the analogy breaks: in a real lock, a key paired with itself might jam. In a group, every element always has a well-defined inverse and operations always succeed. Groups are more orderly than physical locks.

The four rules are closure, associativity, identity element, and inverse element.

4.1.2 Symbol Registry

Symbol	Name	Meaning	Type	Domain
$G$	Group set	The collection of elements forming the group	Set	—
$\circ$	Binary operation	The operation combining two elements	Operation	—
$e$	Identity element	The neutral "do-nothing" element	Element	$e \in G$
$a^{- 1}$	Inverse of $a$	The element that combines with $a$ to give $e$	Element	$a^{- 1} \in G$
$\forall$	"For all"	Universal quantifier	Quantifier	—
$\exists$	"There exists"	Existential quantifier	Quantifier	—

4.1.3 The Four Group Properties

Closure. Take any two elements from the set and apply the operation. The output must stay inside the set. If the result ever lands outside, closure fails.

\forall a, b \in G, a \circ b \in G

Think of it like this: you have a set of numbers. Your operation is addition. Adding any two integers always gives an integer. The result never escapes.

Associativity. When you have three or more elements, the way you group them does not change the outcome. The chain of operations matters, but the grouping does not.

\forall a, b, c \in G, (a \circ b) \circ c = a \circ (b \circ c)

Take the numbers 2, 3, and 4 with addition. Group $(2 + 3) + 4$ and you get 9. Group $2 + (3 + 4)$ and you also get 9. Every time.

Identity element. There is always a special neutral element in the set. When you combine any element with this identity, nothing changes. For addition, the identity is 0. Just like $5 + 0 = 5$ .

\exists e \in G, \forall a \in G : a \circ e = e \circ a = a

Inverse element. Every element in the set has a partner — its inverse. When combined with its inverse, it gives back the identity element. For addition, the inverse of 5 is $- 5$ , because $5 + (- 5) = 0$ .

\forall a \in G, \exists a^{- 1} \in G : a \circ a^{- 1} = a^{- 1} \circ a = e

A binary operation takes exactly two inputs. A unary operation (like negation) takes one. The group operation is always binary — you always combine two elements, never one or three at a time.

4.1.4 Worked Example: Is $(Z, \times)$ a Group?

Take the set of all integers $Z$ with the operation of multiplication. Let us check each property.

- Closure: The product of any two integers is an integer. For example, $3 \times 7 = 21 \in Z$ . ✓ - Associativity: $(a \times b) \times c = a \times (b \times c)$ for integers. $(2 \times 3) \times 4 = 6 \times 4 = 24$ and $2 \times (3 \times 4) = 2 \times 12 = 24$ . ✓ - Identity: The identity for multiplication is 1, since $a \times 1 = a$ . And 1 is an integer. $5 \times 1 = 5$ . ✓ - Inverse: For the element 5, the inverse must satisfy $5 \times a^{- 1} = 1$ . That means $a^{- 1} = \frac{1}{5}$ . But $\frac{1}{5}$ is not an integer — it is a rational number, and rationals are not in $Z$ . The inverse fails. ✗

One failure is enough. Integers under multiplication is NOT a group. Contrast this with integers under addition. That IS a group, because every integer has an additive inverse (its negative). We have $5 + (- 5) = 0$ , and $- 5 \in Z$ .

Sense-check: Only the elements 1 and $- 1$ have multiplicative inverses inside $Z$ . For every other integer, the inverse is a fraction outside the set. That is why this structure fails.

4.1.5 Assumptions and Scope

Scope: The group concept assumes the operation is binary (two inputs, one output) and total (defined for every pair). If the operation is partial, the structure is not a group. Division is one example. It is not defined when the divisor is zero.

When groups apply: Any time you have a set and one way to combine elements, group theory gives you a framework. This covers addition on numbers, composition of rotations, permutations of a list, and modular arithmetic on a clock face.

When groups break: Groups only model one operation. If your structure needs two operations (like addition AND multiplication), you need a ring or a field — richer structures. Vector spaces need a field of scalars, which is why we cannot stop at groups.

4.1.6 Visual Intuition

Picture a circle with labeled points. Each group element is a point. The operation tells you how to jump from one point to another. The identity is a point that, when you jump from it to any other point and back, leaves you where you started. The inverse of a point is the point you jump to in order to return to the identity.

For the group of integers under addition: draw the number line. Zero is the identity. Any integer $n$ has its inverse at $- n$ , the mirror point across zero. Adding $n$ then adding $- n$ brings you back to zero. The takeaway: a group is a playground where every move is reversible and you never fall off the edge.

4.1.7 Pitfalls

- Confusing the set with the operation. A group is the pair $(G, \circ)$ , not just the set $G$ . The same set $Z$ is a group under addition but NOT a group under multiplication. Always state both the set and the operation. - Forgetting to check every element for inverses. The inverse property says every element needs an inverse, not just some. For $(Z, \times)$ , 1 and $- 1$ have inverses — but 2, 3, 5, and all others do not. One failure topples the whole structure. - Assuming commutativity is required. Commutativity ( $a \circ b = b \circ a$ ) is not part of the four group axioms. Many groups are non-commutative — matrix multiplication, for example. If a group has commutativity on top of the four axioms, it gets a special name (abelian). - Confusing closure with "output is in some bigger set." Closure means the result stays inside $G$ itself — not inside some larger superset. For $(Z, \times)$ , the output is in $Z$ , so closure holds. But for natural numbers under subtraction, $3 - 5 = - 2$ leaves the naturals — closure fails.

The professor flagged integers under multiplication as the classic exam trap. Students see closure, associativity, and identity all passing, and mistakenly conclude it is a group. Always check inverses last; that is where most counterexamples live.

4.1.8 Student Q&A

Q: Are natural numbers under addition a group?

A: Natural numbers are ${1, 2, 3, \dots}$ . Under addition: closure holds (sum of two naturals is a natural), associativity holds, but what is the identity? For addition, the identity is 0. But 0 is not a natural number (in the definition used here). So the identity property fails. Even if you included 0, the inverse would fail. The inverse of 5 would be $- 5$ , which is not a natural number. So natural numbers under addition is not a group.

Several students asked variations of "why is $(Z, \times)$ not a group?" — the professor's canonical answer is above in the worked example. The key: the inverse of any integer other than $\pm 1$ is a fraction, and fractions are not integers.

4.1.9 Exam Guidance

Exam note: You will get quiz questions of this form. You are given a set and an operation, and must determine whether it forms a group. Check all four properties in order: closure, associativity, identity, inverse. One failure disqualifies it. The classic trap is integers under multiplication (inverse fails). Also watch for natural numbers under addition (identity fails unless 0 is included, and inverse fails either way).

4.1.10 Recap and Bridge

A group is a set with one operation obeying four rules. You cannot escape the set. Grouping does not matter. A neutral element exists. Every element has an undo button. If the order of combination also does not matter, you get an abelian group. That is the structure that vector addition is built on.

Groups appear wherever symmetry lives. Crystallographers use groups to classify crystal structures by their rotational symmetries. Cryptographers build public-key systems on the structure of modular arithmetic groups. Physicists use Lie groups to describe continuous symmetries in particle physics. In machine learning, groups arise in equivariant neural networks. These architectures guarantee a model's prediction transforms correctly when the input is rotated or translated. The group axioms you learn here are the same ones that guarantee a self-driving car works. It recognizes a stop sign regardless of viewing angle.

4.2 Abelian Groups — Commutativity

What if the order of doing things did not matter? 3 + 4 is the same as 4 + 3. But putting on socks then shoes is not the same as shoes then socks. An abelian group is the special case where order never matters — and this is the exact structure that vector addition obeys.

4.2.1 Definition

If a group has a fifth property on top of the four — commutativity — it is called an abelian group. Commutativity means the order of the elements does not matter.

\forall a, b \in G, a \circ b = b \circ a

For addition: $3 + 4$ equals $4 + 3$ . The result is the same regardless of which comes first. You can swap the operands and the output never changes.

The name comes from Niels Henrik Abel, a Norwegian mathematician. He showed that certain polynomial equations cannot be solved by radicals. That discovery was tied to commutativity in permutation groups. Today, "abelian" is simply shorthand for "commutative."

Everyday analogy: adding money to a wallet. You put in a \$5 bill then a \$10 bill — your total is \$15. Put in the \$10 first then the \$5 — still \$15. Addition of money is commutative. But consider a sequence of physical actions. "Open the door" then "walk through" is not the same as "walk through" then "open the door." Many real-world operations are non-commutative. Matrix multiplication is the canonical non-commutative operation in linear algebra: $A B \neq = B A$ in general. Abelian groups are the nice, orderly case where everything commutes.

4.2.2 Why Study Groups Before Vector Spaces

You cannot understand vector spaces without first understanding groups. A vector space is an abelian group — it obeys all five properties — but it has two operations instead of one. The group properties give you the inner operation (vector addition). The second operation (scalar multiplication) is what makes it a vector space rather than just an abelian group. These are abstract mathematical ideas. You do not use them directly in ML code every day. But they are the foundation on which everything else sits.

4.2.3 Assumptions and Scope

Scope: Commutativity is an extra property — a group can be non-abelian and still be a valid group. Matrix multiplication groups, permutation groups on three or more objects, and quaternion multiplication are all non-abelian. If a structure fails commutativity but passes the four group axioms, it is still a group — just not an abelian one.

When commutativity holds: Addition of real numbers, integers, rationals. Vector addition (which is component-wise addition). Modular addition on a clock. Function addition $(f + g) (x) = f (x) + g (x)$ . Multiplication of non-zero real numbers. These are all abelian.

When commutativity fails: Matrix multiplication. Composition of rotations in 3D (rotate around X then Y gives a different result than Y then X). String concatenation. These are non-abelian groups (or not even groups, depending on the set).

4.2.4 Pitfalls

- Assuming all groups are abelian. The four group axioms do not include commutativity. Many important groups are non-abelian. Do not assume $a \circ b = b \circ a$ unless it is explicitly stated or proven. - Confusing abelian groups with fields or vector spaces. An abelian group has one operation. A field has two operations (addition and multiplication, both forming abelian groups, with distributivity linking them). A vector space is an abelian group under addition plus scalar multiplication from a field. Each builds on the previous. - Thinking commutativity means "no structure." Even abelian groups carry rich structure — subgroups, cosets, quotient groups, and torsion. Commutativity simplifies analysis but does not trivialize it.

4.2.5 Recap and Bridge

An abelian group is a group where swapping the order never changes the result. Vector addition inherits all five abelian group properties. The only thing missing from an abelian group is scalar multiplication. A vector space can stretch and shrink vectors, not just add them.

Abelian groups underpin digital signal processing (DSP). The set of all digital filters forms an abelian group under convolution. In error-correcting codes — used in every QR code and satellite transmission — codewords live in abelian groups under bitwise XOR. The commutative property guarantees that encoding then decoding returns the original message regardless of transmission order.

4.3 Vector Spaces: Inner and Outer Operations

You stand on a flat field. You can walk in any direction — that is vector addition. Now imagine a giant hand reaches down from the sky and stretches all your steps by a factor of 3. The hand is not standing on the field — it is an outside force. That is scalar multiplication. A vector space is simply the field plus the hand. You can walk anywhere and a force from outside can stretch every movement, but you never leave the field.

4.3.1 Definition

A vector space is an abelian group with two operations:

1. Vector addition — called an inner operation because everything happens within the space. You take two vectors from the space, add them, and the result is still in the space. No outside elements are involved.

2. Scalar multiplication — called an outer operation because you bring in something from outside the space. You take a vector from the space and multiply it by a scalar — a real number. The result must still be in the space. The scalar itself is not part of the vector space.

Formally, a vector space $(V, R)$ is:

- An abelian group $(V, +)$ — so vector addition is closed, associative, commutative, has identity $0$ , and every $v$ has an inverse $- v$ . - A scalar multiplication operation $\cdot : R \times V \to V$ satisfying four distributive and compatibility axioms: 1. $α (v + w) = α v + α w$ (scalar distributes over vector addition) 2. $(α + β) v = α v + β v$ (vector distributes over scalar addition) 3. $α (β v) = (α β) v$ (scalar multiplication is associative with field multiplication) 4. $1 v = v$ (multiplication by the scalar 1 leaves the vector unchanged)

The first three are the group axioms for vector addition. The last four are the scalar multiplication rules. Together they make eight axioms total.

4.3.2 Symbol Registry

Symbol	Name	Meaning	Type	Domain
$V$	Vector space	The set of all vectors under consideration	Set	—
$v, w$	Vectors	Elements of the vector space	Vector	$v, w \in V$
$α, λ$	Scalar	A real number used for scaling	Scalar	$α \in R$
$R$	Real numbers	The field of scalars	Field	—
$0$	Zero vector	The additive identity	Vector	$0 \in V$

4.3.3 Inner Operation — Vector Addition

Think of a two-dimensional plane $R^{2}$ . Every single point on this plane is a combination of two numbers — a vector. If you take any two points, say $(1, 2)$ and $(3, 5)$ , and add them component-wise, you get $(4, 7)$ . This new point is still in $R^{2}$ . You are doing everything within the space itself. That is why it is called an inner operation.

v + w \in V \forall v, w \in V

Vector addition is written as $V \times V \to V$ . You take two things from $V$ , produce one thing in $V$ . Everything stays inside — an inner (or internal) operation.

4.3.4 Outer Operation — Scalar Multiplication

Now take the vector $(1, 2)$ and multiply it by the scalar 8. You get $(8, 16)$ . The scalar 8 is not a vector. There is no special place in the two-dimensional plane where a single number like 8 lives on its own. Every point needs two coordinates. So 8 is an entity from outside the space. That is why scalar multiplication is called an outer operation.

α v \in V \forall α \in R, \forall v \in V

Scalar multiplication is written as $R \times V \to V$ . You take a scalar from $R$ and a vector from $V$ to produce a vector in $V$ . The scalar came from outside — an outer (or external) operation.

4.3.5 Assumptions and Scope

Scope: The scalar field is almost always $R$ (real numbers) in ML contexts. But it can also be $C$ (complex numbers) or $Q$ (rational numbers). The vector space changes subtly depending on the field. The space $C^{n}$ as a complex vector space has different dimension properties than $C^{n}$ as a real vector space.

Assumptions: Vector addition must form an abelian group. If addition is not commutative, the structure cannot be a vector space — even if scalar multiplication works fine. Also, the scalar multiplication distributive laws are what bind the inner and outer operations together. Without them, you have two unrelated operations, not a unified vector space.

4.3.6 Visual Intuition

Draw the infinite 2D plane. Mark the origin $(0, 0)$ . Every point is a vector. Vector addition is the "parallelogram rule": to add $v$ and $w$ , draw $w$ starting from the tip of $v$ . The diagonal of the parallelogram connecting origin to that final tip is $v + w$ . Scalar multiplication is the "zoom slider". Multiply by 2 and every point moves twice as far from the origin along the same ray. Multiply by $- 1$ and it flips to the opposite side. Multiply by 0 and it collapses to the origin.

The takeaway: vector addition lets you move anywhere. Scalar multiplication lets you stretch any step. Together they give you complete navigational freedom over the space.

4.3.7 Pitfalls

- Forgetting that scalars are outside the space. Students sometimes treat scalars as vectors — they are not. A scalar 5 in $R$ is not the same as the vector $[5]$ in $R^{1}$ (though they behave similarly). In higher dimensions, the distinction is obvious: a scalar is always one-dimensional. - Confusing scalar multiplication with the dot product. Scalar $\times$ vector produces a vector ( $α v$ ). Dot product of two vectors produces a scalar ( $v \cdot w$ ). They are entirely different operations. The notation $α \cdot v$ sometimes uses a dot, but the meaning is scalar multiplication, not dot product. - Assuming the scalar must be a real number. In this course, yes. But vector spaces can be defined over any field — complex numbers, rational numbers, even finite fields. Always note which field is being used.

4.3.8 Recap and Bridge

A vector space is an abelian group equipped with an extra operation. That operation is scalar multiplication. It lets you stretch, shrink, and flip vectors. Vector addition lives entirely inside the space. Scalar multiplication brings in a field element from outside but produces a vector still inside. This dual-operation structure is what separates vector spaces from plain groups.

The inner/outer operation distinction matters in software engineering too. When you design a class for linear algebra, vector addition is a method that takes two objects of the same type. It returns one of that type (inner). Scalar multiplication takes a float and a vector and returns a vector (outer). This type signature is exactly what the notation $V \times V \to V$ and $R \times V \to V$ encodes.

4.4 Vector Spaces in Machine Learning

A self-driving car sees a stop sign. Your phone transcribes your voice message. Netflix recommends a show. All three start with the same step: turning raw input into a vector. Every pixel, every word, every waveform becomes a list of numbers. The entire machinery of machine learning runs on vector spaces.

4.4.1 Data as Vectors

Every single data point — whether text, image, video, audio, or structured table data — can be represented as a vector. When you represent text or images numerically, those representations are called embeddings. An embedding is just a vector.

For an image: an image is a grid of pixel values. A 6-pixel image has 2 rows and 3 columns. Flatten it into a single vector by concatenating the rows — one after another. A 50-megapixel image becomes a 50-million-dimensional vector.

For text: you can use count-based methods. Count how many times each unique word appears. The word "sweet" might be one dimension, "are" another dimension, and so on. The vector records the frequency of each word.

Everyday analogy: a health report card. Your blood test gives you numbers: cholesterol, glucose, iron, white blood cell count. Each measurement is one dimension. Together they form a vector — your health profile. A doctor compares your vector against healthy ranges. A machine learning model does the same: it takes a data vector and compares it against learned patterns.

4.4.2 Symbol Registry

Symbol	Name	Meaning	Type	Domain
$α$	Model parameter	The coefficient being tuned (e.g., in salary = $α$ × age)	Scalar	$α \in R$
$p$	Position vector	Current (parameter, error) coordinate on the error landscape	Vector	$p \in R^{2}$

4.4.3 The Error Landscape and Gradient Descent Intuition

Machine learning models make predictions. We want those predictions to be as accurate as possible. So we minimize the error. The process of minimizing error uses vectors.

Imagine you have a simple model: $salary = α \times age$ . You have data — you know everyone's actual salary and age. The only thing you do not know is $α$ , the parameter. The model tries different values of $α$ and computes the error between predicted and actual salary.

Plot the error against the parameter value. This gives you an error landscape — a curve that might look like a valley. You start at some point on this curve, say parameter $= 8$ and error $= 35$ . Your goal is to reach the bottom of the valley where error is minimized, say parameter $= 4$ and error $= 0$ .

Your current position is a vector: $(parameter, error)$ . Your target position is another vector. You cannot jump directly to the bottom — that is like jumping off a cliff. Instead, you take small baby steps. Each step is computed by subtracting a scaled gradient vector from your current position vector:

p_{new} = p_{old} - η \nabla E (p_{old})

Here $η$ (eta, the learning rate) controls step size. The term $\nabla E$ is the gradient. It is the vector pointing in the direction where error increases fastest. Subtracting it points you downhill.

This entire process is called gradient descent. You are simply doing vector mathematics. You add and subtract scaled vectors. You move from one point to another iteratively. This is why vectors matter. Your position in parameter space is a vector. The direction of steepest descent is a vector. Each update step is vector addition.

4.4.4 Assumptions and Scope

Scope: The error landscape picture here is for a single parameter. Real models have millions or billions of parameters. The error landscape becomes an incomprehensibly high-dimensional surface. Vector operations generalize cleanly — the update rule $p_{new} = p_{old} - η \nabla E (p_{old})$ works for parameter vectors of any dimension.

Assumptions: The error landscape must be differentiable — the gradient $\nabla E$ must exist at every point. For models with non-differentiable components (ReLU at zero, absolute value loss), subgradients or other optimization methods are needed. Also, gradient descent assumes the step size $η$ is chosen appropriately. Too large and you overshoot the minimum. Too small and you crawl.

4.4.5 Visual Intuition

Draw a 2D graph. The horizontal axis is the parameter $α$ . The vertical axis is the error. The curve is U-shaped — a valley. Mark your starting point high on one wall: $(α = 8, error = 35)$ . Draw a small tangent arrow at that point pointing down the slope — that is the negative gradient. Then draw a new dot slightly lower: your position after one update step. Repeat several times, the dots step their way to the valley floor.

Takeaway: gradient descent is a blind hiker on a foggy hill. The hiker feels the slope underfoot and takes a small step downhill. Over and over, until the ground flattens.

4.4.6 Pitfalls

- Forgetting the origin is part of the vector space. In an error landscape, the point of zero error is the ideal — but it might not be reachable. The zero vector is always in the space. It is the coordinate $(0, 0)$ , not "zero error." - Confusing the data vector space with the parameter vector space. Your data (images, text) live in the input space. Your model parameters live in a different vector space. Gradient descent walks through parameter space, not data space. The dimensions are entirely different. - Thinking position vectors need a fixed origin. In $R^{n}$ , every vector implicitly starts at the origin. Your position on the error landscape IS the vector from the origin to your current coordinates. The origin is the reference. You always know where you are relative to $(0, 0)$ .

4.4.7 Student Q&A

Q: On the error landscape graph, how do we identify the position of the vector? Vectors do not have a fixed origin, so starting from the error = 40 point, how do we know where to go?

A: We know the starting point because we know two things. First, the error our model produced — say a deficit of 35. Second, the parameter value that produced it — say 8. The model is $salary = α \times age$ . Given the data — actual salaries and ages — we compute the error for each $α$ we try. That gives us a specific coordinate $(α, error)$ on the landscape. From there we take iterative baby steps toward the minimum. We know where we are because we know what the model predicted and what parameters it used.

Q: When we create embeddings for text and images, do they reside in the same coordinate space? How can we identify which is text and which is an image?

A: They typically reside in two different spaces. The dimensions come from different sources. For images, the dimensions are pixel brightness values — a 50-megapixel image is a 50-million-dimensional vector. For text, the dimensions are word frequencies — maybe 20,000-dimensional. The first entity measures brightness; the second measures word occurrence. They are fundamentally different things. However, there is a technique called Canonical Cross-Correlation (CCC) that projects both into a common space so you can correlate them. One application: subtitle generation from lip movements (models like LipNet). You project video frames and text into a shared space to match lip shapes with words.

Several students asked about gradient descent and optimization. The professor confirmed the valley-searching intuition maps to algorithms like gradient descent covered in Lectures 11–12.

4.4.8 Recap and Bridge

Every data point is a vector in some space. ML models search through parameter spaces — also vector spaces — using gradient descent, which is just iterative vector addition and subtraction. The link between abstract vector spaces and practical ML is direct: your position, your direction, and your updates are all vectors.

Vector representations power modern search engines. Google's BERT converts search queries and web pages into vectors in the same space. It then finds the closest matches using dot products. Recommendation systems at Netflix and Spotify embed users and items into a shared vector space. Your taste vector sits near the vectors of shows you would enjoy. Every time you get a relevant recommendation, a vector space just did its job.

4.5 Litmus Tests for Vector Spaces

You are given a set of vectors. Is it a vector space or an impostor? You do not need to verify all eight axioms every time. Three quick tests will expose any fake. If a space fails even one, it is not a vector space. These three tests are your litmus paper.

4.5.1 The Three Tests

For any space to be a vector space, it must pass three litmus tests. These are additional rules beyond the group and abelian group properties.

Test 1 — Contains the zero vector. The space cannot be empty. At minimum, the origin must be there.

0 \in V

Test 2 — Closed under vector addition. Take any two vectors from the space. Add them. The result must still be in the space.

\forall v, w \in V, v + w \in V

Test 3 — Closed under scalar multiplication. Take any vector from the space. Multiply it by any real number. The result must still be in the space.

\forall α \in R, \forall v \in V, α v \in V

Why just these three? Because the other axioms (associativity, commutativity, distributivity) are inherited from the parent space. If $V$ is a subset of $R^{n}$ and passes the three tests, it automatically inherits all remaining vector space axioms from $R^{n}$ . These three are the only ones that can fail for a subset.

Everyday analogy: airport security checkpoints. To board a flight, you do not need to prove your entire identity from scratch. Three checks — ID verification, security scan, boarding pass — are enough. If you fail one, you do not fly. The litmus tests work the same way for vector spaces: three targeted checks, one failure, and the set is rejected.

4.5.2 Symbol Registry

Symbol	Name	Meaning	Type	Domain
$V$	Vector space	The set of vectors under consideration	Set	—
$0$	Zero vector	The additive identity (origin)	Vector	$0 \in V$
$v, w$	Vectors	Elements of $V$	Vector	$v, w \in V$
$α$	Scalar	A real number for scaling	Scalar	$α \in R$

4.5.3 Worked Example: Is the Y-axis a Vector Space?

The Y-axis in $R^{2}$ is the set ${(0, y) ∣ y \in R}$ . It is a line passing through the origin. Let us check all three tests.

Test 1 — Zero vector: $(0, 0)$ is on the Y-axis. ✓

Test 2 — Closed under addition: Take any two vectors on the Y-axis, $(0, y_{1})$ and $(0, y_{2})$ . Their sum is $(0, y_{1} + y_{2})$ . Since $y_{1} + y_{2}$ is still a real number, the result is still on the Y-axis. ✓

v_{1} v_{1} + v_{2} = (0, y_{1}), v_{2} = (0, y_{2}) = (0, y_{1} + y_{2}) \in Y-axis

Test 3 — Closed under scalar multiplication: Take $(0, y)$ and multiply by any scalar $α$ . The result is $(0, α y)$ , which is still on the Y-axis. For example, $(0, 10)$ multiplied by 2 gives $(0, 20)$ . Multiplied by $- 4$ gives $(0, - 40)$ . Both are on the line. ✓

α \cdot (0, y) = (0, α y) \in Y-axis

Conclusion: The Y-axis IS a vector space. Answer: Yes. All three tests pass.

Counterexample: The line $y = x + 5$ is NOT a vector space. When $x = 0$ , $y = 5$ . When $y = 0$ , $x = - 5$ . The origin $(0, 0)$ is not on this line. Test 1 fails immediately. Any line or plane that does not pass through the origin cannot be a vector space.

Sense-check: A line through the origin is a 1-dimensional vector space sitting inside the 2-dimensional $R^{2}$ . A line shifted away from the origin loses the zero vector and can no longer support vector space structure.

4.5.4 Assumptions and Scope

Scope: The three litmus tests are sufficient only when checking whether a subset of a known vector space is a subspace. If you are building a vector space from scratch, you must verify all eight axioms. For example, defining a space of functions requires the full definition. The litmus tests are a shortcut for subspaces, not a replacement for the full definition.

What happens if a test fails: - Failing Test 1 (no zero vector): You cannot define direction. The identity element is missing. The space has no reference point. - Failing Test 2 (not closed under addition): Adding two valid elements can produce an invalid one. The space has "holes" or "edges" you can fall off. - Failing Test 3 (not closed under scalar multiplication): Scaling a valid element can push it outside the space. The space is bounded or has gaps.

4.5.5 Visual Intuition

Draw the 2D plane. Highlight the Y-axis as a bold vertical line through the origin. Now draw two arbitrary points on that line. Show their sum also landing on the line. Show scalar multiplication stretching a point but staying on the line. Then draw the line $y = x + 5$ — it crosses the Y-axis at $(0, 5)$ but misses the origin. Mark the origin $(0, 0)$ with a star — it is not on the line.

Takeaway: any line through the origin is a vector space. Any line that misses the origin is not. The origin is the gatekeeper.

4.5.6 Pitfalls

- Assuming "line" automatically means vector space. Only lines through the origin qualify. $y = 2 x$ is a vector space. $y = 2 x + 1$ is not. The constant term shifts the line off the origin. - Forgetting to check scalar multiplication for negative scalars. A set might be closed under addition and contain the zero vector, but multiplying by $- 1$ could produce a vector outside the set. Check negative scalars explicitly. - Assuming a square region is a subspace. The filled square $- 1 \leq x \leq 1, - 1 \leq y \leq 1$ contains the origin and is closed under addition for small vectors. But scaling a vector near the boundary by 10 breaks out. Any bounded set (other than ${0}$ ) fails Test 3. - Skipping Test 1 because "it is obvious." Always explicitly verify the zero vector is in the set. It catches shifted planes and lines instantly.

4.5.7 Student Q&A

Q: Why must a vector space contain the origin? What if it is a line that passes through the coordinate system's origin but in a different sense?

A: The origin is needed as a reference point. A vector has both direction and magnitude. If you have a house with 1 door and 1 window, it is $(1, 1)$ . You need to know you are measuring 1 unit from zero. Without zero, you cannot tell which way a vector points. The zero vector is essential for direction. Additionally, the group axioms require an identity element — the zero vector plays that role for vector addition. You need to trace back to something. You need an origin.

4.5.8 Exam Guidance

Exam note: Always check all three litmus tests when asked "is this a vector space?" The origin test catches counterexamples instantly. Lines or planes missing the origin cannot be vector spaces. The bounded-set trap (square, disc, sphere) catches those who forget scalar multiplication can explode a vector.

4.5.9 Recap and Bridge

Three tests expose any fake vector space. Does it have the origin? Can you add any two vectors without leaving? Can you scale any vector without leaving? Pass all three and you have a subspace. Fail one and you do not. The litmus tests connect to the next topic — linear combinations and span. Any span automatically passes all three tests.

The litmus tests appear in functional analysis too. The set of all continuous functions on $[0, 1]$ is a vector space. It contains the zero function. The sum of two continuous functions is continuous. Scaling a continuous function keeps it continuous. In contrast, the set of all functions with $f (0) = 5$ fails Test 1. The zero function does not satisfy $f (0) = 5$ . The origin principle is universal.

4.6 Linear Combinations

You have eggs, flour, and sugar. Mix them and you get cake batter. Change the proportions and you get cookie dough. A linear combination is the same idea: take vectors, scale each one by some amount, add them up. The recipe is the list of scalars. The dish is the new vector.

4.6.1 Definition

A linear combination is how you combine existing vectors to create a new vector. You scale each vector by a scalar coefficient and then add them all together.

w = α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} = i = 1 \sum n α_{i} v_{i}

Each $α_{i} \in R$ is a scalar coefficient. Each $v_{i} \in V$ is a vector from the set. The result $w$ is also in $V$ (by closure under addition and scalar multiplication — the vector space axioms guarantee this).

You are creating new things out of existing things. Think of ingredients in a kitchen: eggs, flour, and sugar. How you combine them — the proportions you choose — determines whether you get a cake or a tart. The ingredients are the original vectors. The amounts are the scalars. The new dish is the linear combination.

4.6.2 Symbol Registry

Symbol	Name	Meaning	Type	Domain
$v_{i}$	The $i$ -th vector	One of the vectors in the set being combined	Vector	$v_{i} \in V$
$α_{i}$	Scalar coefficient	How much to scale $v_{i}$	Scalar	$α_{i} \in R$
$w$	Resulting vector	The new vector created by the linear combination	Vector	$w \in V$

4.6.3 Geometric Intuition

In a two-dimensional plane, take two vectors that are not collinear. You can reach every single point by scaling and adding them. The interactive visualization at the linear algebra .info website demonstrates this: you have two dashed vectors. You scale the first by $c_{1}$ and the second by $c_{2}$ . The scaled versions appear as solid vectors. Their sum is shown in purple. By varying $c_{1}$ and $c_{2}$ , the purple vector can reach any point in the plane.

Even if you start from negative scalars, you can move in the opposite direction. The original vectors stay the same; only their scaling changes. And yet you can cover the entire space.

4.6.4 Worked Example: Simple 3D Linear Combination

Is the vector $[- 1, 4, 0]$ a linear combination of $[1, 0, 0]$ , $[0, 1, 0]$ , and $[1, 2, 0]$ ?

We need to find scalars $α, β, γ$ such that:

α 100 + β 010 + γ 120 = - 1 40

This gives the system:

α + 0 β + γ 0 α + β + 2 γ 0 α + 0 β + 0 γ = - 1 = 4 = 0 (1st component) (2nd component) (3rd component, always satisfied)

The third component of all three vectors is 0, so the third equation is $0 = 0$ — always true. We only need to solve the first two equations.

From equation 1: $γ = - 1 - α$ . Substitute into equation 2: $β + 2 (- 1 - α) = 4 ⟹ β - 2 - 2 α = 4 ⟹ β = 6 + 2 α$ . Let $α$ be free. Then $γ = - 1 - α$ and $β = 6 + 2 α$ .

One solution (set $α = - 2$ ): $β = 6 + 2 (- 2) = 2$ , $γ = - 1 - (- 2) = 1$ . Check: $- 2 [1, 0, 0] + 2 [0, 1, 0] + 1 [1, 2, 0] = [- 2, 0, 0] + [0, 2, 0] + [1, 2, 0] = [- 1, 4, 0]$ . ✓

Another solution (set $α = - 1$ ): $β = 4$ , $γ = 0$ . Check: $- 1 [1, 0, 0] + 4 [0, 1, 0] + 0 [1, 2, 0] = [- 1, 4, 0]$ . ✓

Conclusion: YES, the vector IS a linear combination. Answer: Yes. There are infinitely many solutions because the three given vectors are linearly dependent. One can be expressed in terms of the others.

Sense-check: All vectors have $z = 0$ , so any linear combination must also have $z = 0$ . The target $[- 1, 4, 0]$ has $z = 0$ — consistent. The system has 2 equations and 3 unknowns, so either infinitely many solutions or none.

4.6.5 Worked Example: 4D Linear Combination via Gaussian Elimination

Is $[4, 3, 1, 2]$ a linear combination of these four vectors?

v_{1} = [1, - 1, 0, 1], v_{2} = [0, 1, - 2, 2], v_{3} = [3, - 1, 0, 1], v_{4} = [1, 1, 1, 1]

We need scalars $α, β, γ, δ$ such that:

α v_{1} + β v_{2} + γ v_{3} + δ v_{4} = [4, 3, 1, 2]

Step 1 — Write the system component-wise:

α (1) + β (0) + γ (3) + δ (1) α (- 1) + β (1) + γ (- 1) + δ (1) α (0) + β (- 2) + γ (0) + δ (1) α (1) + β (2) + γ (1) + δ (1) = 4 = 3 = 1 = 2 (Eq 1) (Eq 2) (Eq 3) (Eq 4)

Step 2 — Form the augmented matrix:

1 - 1 01 01 - 2 2 3 - 1 01 11114312

Step 3 — Gaussian elimination.

$R_{2} \leftarrow R_{2} + R_{1}$ :

1001 01 - 2 2 320112114712

$R_{4} \leftarrow R_{4} - R_{1}$ :

1000 01 - 2 2 320 - 2 1210 471 - 2

$R_{3} \leftarrow R_{3} + 2 R_{2}$ :

10000102 324 - 2 1250 4715 - 2

$R_{4} \leftarrow R_{4} - 2 R_{2}$ :

10000100 324 - 6 125 - 4 4715 - 16

$R_{4} \leftarrow R_{4} + \frac{3}{2} R_{3}$ :

100001003240 125 \frac{7}{2} 4715 \frac{13}{2}

The system has a pivot in every column of the coefficient matrix. No row of all zeros on the left has a non-zero on the right. This means a unique solution exists.

Back-substitution: - From $R_{4}$ : $\frac{7}{2} δ = \frac{13}{2} ⟹ δ = \frac{13}{7}$ - From $R_{3}$ : $4 γ + 5 (\frac{13}{7}) = 15 ⟹ 4 γ = 15 - \frac{65}{7} = \frac{105 - 65}{7} = \frac{40}{7} ⟹ γ = \frac{10}{7}$ - From $R_{2}$ : $β + 2 (\frac{10}{7}) + 2 (\frac{13}{7}) = 7 ⟹ β = 7 - \frac{20}{7} - \frac{26}{7} = 7 - \frac{46}{7} = \frac{49 - 46}{7} = \frac{3}{7}$ - From $R_{1}$ : $α + 3 (\frac{10}{7}) + \frac{13}{7} = 4 ⟹ α = 4 - \frac{30}{7} - \frac{13}{7} = 4 - \frac{43}{7} = \frac{28 - 43}{7} = - \frac{15}{7}$

Solution: $α = - \frac{15}{7}, β = \frac{3}{7}, γ = \frac{10}{7}, δ = \frac{13}{7}$ .

Conclusion: YES, the vector IS a linear combination with a unique set of coefficients.

Interpreting outcomes in general: - Unique solution exists: The vector IS a linear combination via a unique set of coefficients. - Infinitely many solutions: The vector IS a linear combination — multiple ways exist. At least one vector in the set is redundant (dependent). - No solution (inconsistent): The vector is NOT a linear combination of the given set.

Sense-check: The target is in $R^{4}$ and we have 4 vectors in $R^{4}$ . The system is square, so we expect either a unique solution or none. The matrix reduced cleanly with 4 pivots — consistent system, unique solution.

This is a guaranteed type of exam question. You will be given vectors and asked whether a specific vector is a linear combination. Use Gaussian elimination on the augmented matrix.

4.6.6 Assumptions and Scope

Scope: Linear combination testing is solving a linear system $A x = b$ where the columns of $A$ are the given vectors and $b$ is the target. Gaussian elimination always works, regardless of matrix shape.

When it applies: For any finite set of vectors in any $R^{n}$ . Also applies to function spaces — a Fourier series is a linear combination of sine and cosine functions.

Limitations: If the coefficient matrix is square and the determinant is non-zero, you get a unique solution. If the matrix is rectangular (more equations than unknowns, or vice versa), row-echelon form reveals free variables or inconsistencies. Determinants do not work on non-square matrices.

4.6.7 Visual Intuition

In $R^{2}$ , draw vector $v_{1} = [2, 0]$ (a rightward arrow along the X-axis) and $v_{2} = [1, 1]$ (a diagonal arrow). Scaling $v_{1}$ by different values slides you left and right. Scaling $v_{2}$ slides you along the diagonal. Combining both lets you reach any point in the plane. Plot the target point $[5, 3]$ and find the scalars: $2 α + 1 β = 5$ and $0 α + 1 β = 3 ⟹ β = 3, α = 1$ .

Takeaway: two non-collinear vectors unlock the entire 2D plane through linear combinations. Every point corresponds to one unique $(α_{1}, α_{2})$ pair.

4.6.8 Pitfalls

- Forgetting that the coefficients can be zero. A zero coefficient just means "do not use that vector." It is perfectly valid. The only forbidden thing is the trivial combination. There, ALL coefficients are zero. That always gives the zero vector and tells you nothing. - Confusing "linear combination exists" with "linear combination is unique." A vector can be a linear combination in infinitely many ways if the given vectors are dependent. The 3D example above shows this. The classic $[1, 0] + [0, 1] + [2, 2]$ case shows it too. - Using determinants on non-square systems. Determinants require a square matrix. For example, 3 vectors in $R^{4}$ — the augmented matrix is $4 \times 3$ . You cannot use the determinant. Gaussian elimination is needed. - Not checking the augmented column. A row becomes $[00 \dots 0 ∣ c]$ with $c \neq = 0$ — the system is inconsistent. Always include the augmented column through the whole elimination.

4.6.9 Student Q&A

Q: For testing linear combination, can we use determinants instead of Gaussian elimination?

A: Yes, determinants can be used when the coefficient matrix is square. If the determinant is non-zero, there is a unique solution. When the matrix is large, Gaussian elimination (row-echelon form) is generally preferred because computing determinants becomes cumbersome for $n > 3$ . When the matrix is small ( $2 \times 2$ or $3 \times 3$ ), the determinant is the easier method. But determinants only apply to square matrices. For non-square systems, row-echelon form is required.

4.6.10 Exam Guidance

Exam note: This is a guaranteed exam question. Given a set of vectors and a target vector, determine whether the target is a linear combination. Form the augmented matrix. Use Gaussian elimination. Three outcomes: unique solution → yes (unique coefficients), infinitely many → yes (dependent vectors), no solution → no. Master this workflow.

4.6.11 Recap and Bridge

A linear combination scales vectors by scalars and adds them. Testing whether a target vector is a linear combination reduces to solving $A x = b$ via Gaussian elimination. The set of all vectors reachable this way is called the span — the topic of the next section.

Linear combinations are the backbone of computer graphics. Every 3D model in a game or film is a collection of triangles defined by vertices — vectors in $R^{3}$ . Rotating, scaling, and translating the model are all linear combinations (or affine combinations) of those vertex vectors. The GPU in your laptop spends its life computing billions of linear combinations per second.

4.7 Span and Subspaces

You have a train going North and a bus going East. Where can you get to? Every city you can reach by combining train rides and bus rides — that is the span. Now comes the beautiful part: the set of all reachable cities is itself a valid space. The span of any set of vectors is always a subspace.

4.7.1 Definition of Span

The span of a set of vectors is the collection of all possible points you can reach by linearly combining those vectors. It is the set of all vectors that are linear combinations of the given set.

span {v_{1}, \dots, v_{n}} = {i = 1 \sum n α_{i} v_{i} α_{i} \in R}

Read this as: "the span is the set of all $\sum α_{i} v_{i}$ such that each $α_{i}$ is a real number." The scalars $α_{i}$ can be any real numbers. They may be positive, negative, zero, or fractions. You have total freedom to mix the vectors in any proportion.

Think of the interactive visualization. Given two vectors in $R^{2}$ , the highlighted points on the screen are all the positions you can reach. You get there by scaling and adding those two vectors. That set of highlighted points is the span.

Everyday analogy: modes of transportation. You have a train line (vector $v_{1}$ ) and a bus route (vector $v_{2}$ ). The train goes North at 60 km/h. The bus goes East at 30 km/h. By riding the train for $t_{1}$ hours and the bus for $t_{2}$ hours, you arrive at $(30 t_{2}, 60 t_{1})$ . The span of ${v_{1}, v_{2}}$ is every location you can reach — the entire plane. If you only had the train, your span is just the North-South line (a 1D subspace). If you had both but they ran on the same track (collinear), your span is still that single line. Adding the bus adds nothing new.

Where the analogy breaks: real transportation has finite range. Trains do not go to infinity. Vectors do — you can scale by arbitrarily large scalars.

4.7.2 Symbol Registry

Symbol	Name	Meaning	Type
$span$	Span operator	The set of all linear combinations of a given set of vectors	Set
$α_{i}$	Scalar coefficient	Real-valued weight for vector $v_{i}$	Scalar

4.7.3 Theorem: Span Is a Subspace

Theorem: The span of any set of vectors is a subspace of the ambient vector space.

A subspace is a vector space contained within a larger vector space. It must satisfy the same three litmus tests: contains the zero vector, closed under addition, closed under scalar multiplication.

Why is the span always a subspace? Because any linear combination of linear combinations is still a linear combination of the original vectors:

- Contains zero vector: Set all $α_{i} = 0$ . The result is $0$ . ✓ - Closed under addition: Take two vectors in the span, $u = \sum α_{i} v_{i}$ and $w = \sum β_{i} v_{i}$ . Their sum is $\sum (α_{i} + β_{i}) v_{i}$ — another linear combination of the original vectors. ✓ - Closed under scalar multiplication: Take $u = \sum α_{i} v_{i}$ and multiply by $c \in R$ . The result is $\sum (c α_{i}) v_{i}$ — still a linear combination of the original vectors. ✓

All three tests pass automatically. No extra checking needed. The span is always a subspace.

This theorem reveals a deep insight. The set of all points reachable from a given set of vectors is always a vector space. It is a subspace of whatever larger space the original vectors lived in.

4.7.4 Span of Standard Basis Vectors

Take the three vectors $[1, 0, 0]$ , $[0, 1, 0]$ , and $[0, 0, 1]$ in $R^{3}$ . What is their span?

- $[1, 0, 0]$ sits on the X-axis. - $[0, 1, 0]$ sits on the Y-axis. - $[0, 0, 1]$ sits on the Z-axis.

By combining these three vectors with scalars $α, β, γ$ , you can reach any point $(α, β, γ)$ in $R^{3}$ . The span is the entire three-dimensional space.

span {[1, 0, 0], [0, 1, 0], [0, 0, 1]} = R^{3}

If you only had $[1, 0, 0]$ and $[0, 1, 0]$ , the span would be the XY-plane — a two-dimensional subspace. Any linear combination has $z = 0$ , so you cannot reach anything with a non-zero Z-component.

4.7.5 Assumptions and Scope

Scope: The span is always defined relative to an ambient space. $span {) tw o v ec t or s in (R^{3}}$ is a subspace of $R^{3}$ . It could be a line (if collinear) or a plane (if non-collinear). It can never be all of $R^{3}$ unless you have at least three vectors spanning three independent directions.

Key fact: In $R^{n}$ , the maximum possible dimension of a span from $k$ vectors is $min (k, n)$ . You cannot get more dimensions than the number of vectors you start with, and you cannot exceed the ambient dimension.

Degenerate case: The span of the empty set is ${0}$ — just the zero vector. The span of ${0}$ alone is also ${0}$ .

4.7.6 Visual Intuition

In $R^{3}$ , draw three basis vectors along X, Y, and Z axes as colored arrows. Shade the XY-plane (span of X and Y vectors) as a translucent sheet. Now draw an arbitrary point in 3D. Show you need a Z-component to reach it. The XY-plane alone cannot get there.

For two collinear vectors, draw both on the same line. Their span is just that line. Adding the second collinear vector adds no new reachable points. The takeaway: span grows only when new vectors point in genuinely new directions.

4.7.7 Pitfalls

- Confusing the span with the set of original vectors. The span is the infinite set of all linear combinations — not the finite original set. The original set might have 3 vectors, but the span contains infinitely many. - Thinking "more vectors = bigger span." Adding a vector that is already in the span of the others adds nothing. Two collinear vectors span only a line, not a plane. You need linearly independent vectors to increase the span's dimension. - Forgetting the span includes negative scalars. The span extends in all directions. Scaling by negative numbers means you can go opposite each vector. The span of $[1, 0]$ includes $[- 5, 0]$ .

4.7.8 Student Q&A

Q: If four vectors in 4D are linearly independent, can I reach any point in that 4D space?

A: Yes. This is exactly the concept of span: $n$ linearly independent vectors in $R^{n}$ span the entire $R^{n}$ . You can reach any point in the space. Four independent vectors in $R^{4}$ span all of $R^{4}$ .

4.7.9 Recap and Bridge

The span of a set of vectors is every point you can reach through linear combinations. It is always a subspace — automatically satisfying the three litmus tests. Independent vectors grow the span; dependent vectors do not. The span's dimension tells you how many truly different directions you have — leading directly to basis and dimension.

Span connects to recommendation systems. Netflix embeds movies into a 100-dimensional vector space using matrix factorization. The span of the top- $k$ rating vectors captures the most informative directions. These are the directions of viewer preference. Projecting onto this span is a form of dimensionality reduction. It retains the signal while discarding noise.

4.8 Linear Independence and Dependence

You are paying for three streaming services. But service A has all the shows of service B. Service B is redundant — you could cancel it and still watch everything. Linear independence is the same test: if one vector can be built from the others, it is dead weight. The formal test uses a single elegant equation.

4.8.1 Formal Definition

Vectors $v_{1}, v_{2}, \dots, v_{n}$ are linearly independent under one condition. The only way to combine them with scalars to get the zero vector is to set all scalars to zero. This is called the trivial solution.

λ_{1} v_{1} + λ_{2} v_{2} + \dots + λ_{n} v_{n} = 0

⎩ ⎨ ⎧ Linearly Independent Linearly Dependent ⟹ λ_{1} = λ_{2} = \dots = λ_{n} = 0 (only trivial solution) ⟹ \exists i such that λ_{i} \neq = 0 (non-trivial solution exists)

If the only way to "zero out" the combination is to make every $λ_{i} = 0$ , the vectors are independent. Each contributes something unique. If you can find any non-zero set of $λ_{i}$ that still produces $0$ , the vectors are dependent. At least one is redundant.

4.8.2 Symbol Registry

Symbol	Name	Meaning	Type	Domain
$λ_{i}$	Scalar coefficient	Weight for vector $v_{i}$ in the zero-vector test	Scalar	$λ_{i} \in R$
$0$	Zero vector	The vector of all zeros	Vector	$0 \in V$

4.8.3 Why Non-Zero Lambdas Mean Dependence

Suppose three vectors satisfy $λ_{1} v_{1} + λ_{2} v_{2} + λ_{3} v_{3} = 0$ with $λ_{1} \neq = 0$ . Then you can rearrange:

λ_{1} v_{1} v_{1} = - λ_{2} v_{2} - λ_{3} v_{3} = (- \frac{λ _{2}}{λ _{1}}) v_{2} + (- \frac{λ _{3}}{λ _{1}}) v_{3}

So $v_{1}$ is a linear combination of $v_{2}$ and $v_{3}$ . One vector is redundant — it can be expressed in terms of the others. That is what dependence means.

Everyday analogy: solving a puzzle with duplicate pieces. You have three jigsaw pieces. Piece A is a corner. Piece B is an edge. Piece C is the same corner as A, just colored differently. You can lay A and B and express C as "A flipped and repainted." Since C = A (up to a transformation), all three together do not help. You cannot cover more of the puzzle. So C is dependent on A.

Where math sharpens the analogy: in a puzzle, "A flipped" involves rotation — a non-linear operation. In vector spaces, dependence is purely about scaling and adding — linear operations. The vector $[2, 4]$ is exactly $2 \times [1, 2]$ . No rotation is needed.

4.8.4 Worked Example: Dependence in $R^{2}$

Are the vectors $[1, 0]$ , $[0, 1]$ , and $[2, 2]$ linearly independent?

Set up the zero-vector test:

λ_{1} [1, 0] + λ_{2} [0, 1] + λ_{3} [2, 2] = [0, 0]

This gives the system:

λ_{1} + 2 λ_{3} λ_{2} + 2 λ_{3} = 0 = 0

Two equations, three unknowns. Let $λ_{3} = t$ (free). Then $λ_{1} = - 2 t$ and $λ_{2} = - 2 t$ .

For any non-zero $t$ , say $t = - 1$ : $λ_{1} = 2$ , $λ_{2} = 2$ , $λ_{3} = - 1$ .

2 [1, 0] + 2 [0, 1] + (- 1) [2, 2] = [2, 0] + [0, 2] + [- 2, - 2] = [0, 0]

Non-trivial solution exists → Linearly dependent. Answer: Dependent.

Geometric meaning: The third vector lies in the span of the first two. It is redundant. You only need two vectors to span $R^{2}$ .

Sense-check: In $R^{2}$ , any set of 3 or more vectors is guaranteed to be dependent. There are only 2 dimensions — you cannot have more than 2 independent directions.

4.8.5 Key Fact: Maximum Independent Vectors

In $R^{n}$ , you can have at most $n$ linearly independent vectors. Having $n$ independent vectors means they span the entire space. Having more than $n$ vectors in $R^{n}$ guarantees linear dependence. At least one vector must be expressible as a linear combination of the others.

4.8.6 Span of Independent vs Dependent Vectors

- Two linearly independent vectors in $R^{2}$ span the entire $R^{2}$ — you can reach any point. - Two linearly dependent vectors in $R^{2}$ (e.g., $[1, 2]$ and $[2, 4]$ ) span only a line — a one-dimensional subspace. No matter what scalars you pick, the result always lies on that same line. You cannot break out into the second dimension.

This is the crucial link between linear independence and span. Independent vectors give you the full space. Dependent vectors restrict you to a subspace.

4.8.7 Assumptions and Scope

Scope: Linear independence is a property of a set of vectors, not of individual vectors. It makes no sense to say "this vector is independent". Only "this set is independent" is correct. A single non-zero vector is always independent on its own. The zero vector alone makes any set dependent. The reason: $1 \cdot 0 = 0$ is a non-zero combination, since $λ = 1 \neq = 0$ .

Key fact: Any set containing the zero vector is automatically linearly dependent. The non-trivial combination is simply $1 \cdot 0 = 0$ — the coefficient on the zero vector is non-zero.

When independence matters in ML: Independent features in a dataset mean no feature is a linear combination of the others. Dependent features — like "income" and "tax paid" — cause multicollinearity, which destabilizes linear regression coefficients.

4.8.8 Visual Intuition

In $R^{2}$ , draw two independent vectors: $[2, 0]$ (horizontal) and $[0, 3]$ (vertical). Their span is the full plane — every point is reachable. Now draw two dependent vectors: $[1, 2]$ and $[2, 4]$ — both lie on the same line through the origin. Their span is just that line. Adding more vectors on the same line does not expand the span.

In $R^{3}$ , draw three vectors: two in the XY-plane and one with a Z-component. If the first two span the XY-plane and the third has a non-zero Z, all three are independent and span $R^{3}$ . If the third also lies in the XY-plane (its Z = 0), the set is dependent — span is only the XY-plane.

Takeaway: independence = each vector opens up a genuinely new direction. Dependence = at least one vector is just walking down a path already available.

4.8.9 Pitfalls

- Confusing dependence with similarity. The sugar-and-honey analogy is imprecise. Dependence is a precise algebraic condition: $\sum λ_{i} v_{i} = 0$ with at least one $λ_{i} \neq = 0$ . Two vectors can be "completely different things" and still be dependent if one is a scalar multiple of the other. - Thinking "all-zero solution" means dependence. The opposite. The trivial solution $λ_{i} = 0$ for all $i$ always works. Independence means it is the ONLY solution. If you find the trivial solution and stop looking, you may wrongly declare dependent vectors independent. - Forgetting that more-than- $n$ vectors in $R^{n}$ are always dependent. 4 vectors in $R^{3}$ ? Automatically dependent. No calculations needed. This is a one-line answer on a test. - Including the zero vector and calling the set independent. Any set containing $0$ is dependent. Full stop.

4.8.10 Student Q&A

Q: Two vectors overlap — they are collinear. Then we add a third vector in a different direction. Are the three vectors dependent or independent?

A: The three vectors are linearly dependent because one of them (the collinear one) is a scalar multiple of the other. Even with the third vector pointing in a new direction, the set is still dependent. The collinear pair already guarantees a non-trivial solution. For three vectors in $R^{3}$ , the set could be dependent even if the third is in a new direction. This happens as long as any one vector is a combination of the others. You only need to find one non-trivial solution.

Q: The sugar and honey analogy — how can they be dependent when they are completely different things?

A: The professor acknowledged the analogy was imprecise. The intended idea: if multiple items serve the same function, one is redundant. A better analogy: synonyms are linearly dependent because they are essentially different expressions of the same meaning. The formal mathematical definition (non-trivial solution to $\sum λ_{i} v_{i} = 0$ ) is what ultimately matters. In the vector world, dependence means one vector can be written as a combination of others. It does not mean two things are "similar" in a loose sense. The math is precise: $[1, 0]$ and $[0, 1]$ and $[2, 2]$ are dependent because $2 [1, 0] + 2 [0, 1] - 1 [2, 2] = [0, 0]$ .

4.8.11 Exam Guidance

Exam note: Set up $λ_{1} v_{1} + \dots + λ_{n} v_{n} = 0$ and solve. Only trivial solution → independent. Any non-trivial solution → dependent. In $R^{n}$ , more than $n$ vectors are always dependent — you can answer without calculation. Any set containing $0$ is dependent. These are rapid-fire quiz material.

4.8.12 Industry Application: RGB Color Space

Any color on a digital screen can be expressed as a linear combination of Red, Green, and Blue channels. These are the three primary additive colors. You combine them with different intensities (scalars) to produce any color.

Color = r \cdot R + g \cdot G + b \cdot B

The R, G, and B vectors are linearly independent — you cannot combine Green and Blue to get Red. Together they span the entire RGB color space. Each of the 16.7 million colors on a standard display is one point in this 3D vector space.

Another analogy: the 26 letters of the alphabet A–Z are linearly independent — no letter can be formed by combining others. Words are linear combinations of letters. "Cat" is $1 \times C + 1 \times A + 1 \times T + 0 \times (all other letters)$ .

This same concept appears in convolutional neural networks (CNNs). An input image is first decomposed into its Red, Green, and Blue channels. Each channel is a matrix of pixel intensities. These are the basic primary pixel matrices that are extracted, and then the mathematical operations of convolution begin on them.

4.8.13 Recap and Bridge

Linear independence means no vector in the set can be built from the others. The only way to zero the combination is to zero every coefficient. Dependent vectors carry redundancy. In $R^{n}$ , at most $n$ vectors can be independent. Independent vectors span the full space; dependent ones span only a subspace. Independence plus spanning = basis — the next concept.

In finance, portfolio diversification uses linear independence. Assets whose returns are linearly dependent (e.g., two S&P 500 index funds) provide no diversification benefit. They move together. Independent (or at least uncorrelated) assets — stocks and bonds, domestic and international — span a wider range of risk-return profiles. Modern Portfolio Theory, which won a Nobel Prize, is built on this linear algebra insight.

4.9 Basis and Dimension

How many coordinates do you need to locate a point in a room? Three: one for each direction — length, width, height. Those three directions form a basis. The number 3 is the dimension. A basis is the minimal set of directions that can describe the entire space. Its size is fixed. Every basis for the same space has the same number of vectors.

4.9.1 Definition of Basis

A basis is a set of vectors that satisfies two conditions:

1. The vectors are linearly independent. 2. The vectors span the entire space.

B = {b_{1}, \dots, b_{n}} is a basis for V ⟺ {B is linearly independent span (B) = V

Think of a basis as a "coordinate system." Every vector in $V$ can be written uniquely. It is a linear combination of the basis vectors.

v = c_{1} b_{1} + c_{2} b_{2} + \dots + c_{n} b_{n}

The scalars $c_{1}, \dots, c_{n}$ are the coordinates of $v$ in the basis $B$ . If the basis vectors are independent, the representation is unique — every point gets exactly one set of coordinates.

4.9.2 Symbol Registry

Symbol	Name	Meaning	Type
$B$	Basis	A set of linearly independent vectors that span $V$	Set of vectors
$∣ B ∣$	Cardinality	Number of vectors in the basis	Integer
$dim (V)$	Dimension	Cardinality of any basis of $V$	Integer

4.9.3 Why Both Conditions Are Necessary

Someone might argue: if linearly independent vectors span the entire space, why do we need to state both conditions? Because linear independence alone does not guarantee spanning the entire space.

Consider $R^{3}$ . The vectors $[1, 0, 0]$ and $[0, 1, 0]$ are linearly independent. You cannot create one from the other. But can they span $R^{3}$ ? No. The third component is always 0. You can only reach points on the XY-plane. You cannot reach anything with a non-zero Z component. These two vectors are linearly independent. But they do NOT form a basis for $R^{3}$ . They fail the spanning condition.

You need at least three vectors to span $R^{3}$ . Those three must be linearly independent. If one is a combination of the other two, you are back to spanning only a plane.

Everyday analogy: choosing dimensions to describe a box. To fully describe a box, you need height, width, and depth. If you only report height and width, you cannot specify the volume — you are missing a dimension. Those two measurements are "independent" (height does not determine width), but they do not "span" the description of a 3D box. You need all three. A basis is the minimal set of measurements that captures everything about the object.

4.9.4 Standard Basis for $R^{2}$ and $R^{3}$

The standard basis for $R^{2}$ is:

B_{std} (R^{2}) = {[10], [01]}

The standard basis for $R^{3}$ is:

B_{std} (R^{3}) = ⎩ ⎨ ⎧ 100, 010, 001 ⎭ ⎬ ⎫

These are the familiar $i^, j^, k^$ unit vectors from earlier education. Any vector in $R^{3}$ can be written as $x i^+ y j^+ z k^$ — a linear combination of the basis vectors. The scalars $x, y, z$ are the coordinates. The unit vectors $i^, j^, k^$ are the basis vectors.

4.9.5 Definition of Dimension

The dimension of a vector space is the number of vectors in any basis — the cardinality of the basis set.

dim (V) = ∣ B ∣ = number of vectors in any basis of V

For $R^{n}$ : $dim (R^{n}) = n$ . For $R^{2}$ it is 2. For $R^{3}$ it is 3. Simple and direct.

A fundamental theorem: every basis for the same vector space has the same size. You cannot have one basis with 3 vectors and another with 4 for the same space. The dimension is an invariant — it belongs to the space, not to the particular basis you choose.

4.9.6 Assumptions and Scope

Scope: Dimension is defined for a vector space, not for an arbitrary set of vectors. The space of all polynomials of degree $\leq n$ has dimension $n + 1$ (basis: ${1, x, x^{2}, \dots, x^{n}}$ ). The space of all $m \times n$ matrices has dimension $m \cdot n$ . These are finite-dimensional vector spaces. Some spaces — like the space of all continuous functions on $[0, 1]$ — are infinite-dimensional and have no finite basis.

Key properties: - Any basis of $V$ has exactly $dim (V)$ vectors. - Any set with $> dim (V)$ vectors is dependent. - Any set with $< dim (V)$ vectors cannot span $V$ . - Any LI set with exactly $dim (V)$ vectors is automatically a basis. - Any spanning set with exactly $dim (V)$ vectors is automatically a basis.

4.9.7 Visual Intuition

In $R^{2}$ , draw the standard basis vectors $i^= [1, 0]$ and $j^= [0, 1]$ as perpendicular arrows. Mark an arbitrary point $(3, 2)$ . Show that this point is $3 i^+ 2 j^$ . Now show that any other pair of independent vectors — say $[1, 1]$ and $[1, - 1]$ — also works as a basis. The coordinates of $(3, 2)$ change in the new basis, but the point remains the same physical location.

Takeaway: basis = coordinate system. Dimension = the number of coordinates you need. The coordinates depend on the basis you pick, but the dimension is fixed.

4.9.8 Pitfalls

- Confusing "linearly independent" with "basis." Two vectors in $R^{3}$ can be independent but fail to span the space. Basis requires BOTH independence and spanning. The professor emphasized this distinction heavily. - Confusing the dimension of a matrix space with its row/column count. $dim (R^{3 \times 3}) = 9$ , not 3. Each entry is an independent degree of freedom. This is a classic short-answer trap. - Assuming a basis must be orthogonal. Orthogonality is nice but not required. The vectors in $B$ just need to be independent and spanning — they can be at any angle to each other. - Forgetting that the coordinates are unique. In a basis, every vector has exactly one representation. If you find two different coordinate sets for the same vector, the basis condition is violated — the vectors are dependent.

4.9.9 Student Q&A

Q: Is there a quick formula to figure out how many maximum independent vectors exist for an $m \times n$ matrix space?

A: The dimension is $m \cdot n$ . You need exactly $m \cdot n$ basis vectors — the "unit matrices" with a 1 in one position and 0 elsewhere. There can be infinitely many matrices in the space, but any set of more than $m \cdot n$ matrices is guaranteed to be linearly dependent. For example, the space of $3 \times 3$ matrices has dimension 9 — you can have at most 9 independent $3 \times 3$ matrices.

4.9.10 Exam Guidance

Exam note: Basis requires BOTH linear independence AND spanning. Two LI vectors in $R^{3}$ are NOT a basis — they span only a 2D plane. The dimension $dim (R^{m \times n}) = m \cdot n$ , NOT $m$ or $n$ or $m + n$ . A $6 \times 6$ matrix space has dimension 36. These appear in every quiz.

4.9.11 Recap and Bridge

A basis is the Goldilocks set of vectors — just enough to span the space (spanning), with no wasted ones (independent). The dimension is the size of any basis — an invariant property of the space. In $R^{n}$ , the dimension is $n$ ; for $m \times n$ matrix spaces, it is $m \cdot n$ . This leads to the final topic: matrices themselves form vector spaces.

Basis and dimension concepts appear in quantum computing. A quantum state is a vector in a Hilbert space. The computational basis states $∣0 ⟩$ and $∣1 ⟩$ for a single qubit form a basis for a 2D complex vector space. Multi-qubit systems use tensor products of basis states, giving spaces of dimension $2^{n}$ for $n$ qubits. Quantum algorithms manipulate these basis vectors through unitary transformations — the same linear algebra you are learning now, just over complex numbers.

4.10 Matrices as Vector Spaces

A matrix looks like a grid. But if you snip each row and lay the rows end-to-end, you get one long column. That column is a vector. The grid was just a different way to arrange the numbers. So the space of all $3 \times 3$ matrices is really a 9-dimensional vector space in disguise. That fact powers everything from CNN flattening layers to dataset representations.

4.10.1 Flattening Matrices into Vectors

Any $m \times n$ matrix can be decomposed into a vector by concatenating its rows (or columns) one after another.

A = [a c b d] ⟷ v_{row} = a b c d or v_{col} = a c b d

Row-wise concatenation: take row 1, then row 2, and stack them vertically. Column-wise: take column 1, then column 2. Both produce valid vectors in $R^{mn}$ . The choice is a convention — what matters is that you use the same convention consistently.

This flattening is exactly what happens at the final layer of a convolutional neural network (CNN) before classification. All the values from the feature maps are lined up into a single one-dimensional vector. This is the flattening layer. Then the prediction is made.

4.10.2 Symbol Registry

Symbol	Name	Meaning	Type
$R^{m \times n}$	Matrix space	The set of all $m \times n$ matrices with real entries	Set
$dim$	Dimension	Number of vectors in any basis	Integer
$m, n$	Matrix dimensions	Number of rows ( $m$ ) and columns ( $n$ )	Integer

4.10.3 Worked Example: Is the Space of All $2 \times 2$ Matrices a Vector Space?

Consider the set of all $2 \times 2$ matrices with real entries: ${[a c b d] ∣ a, b, c, d \in R}$ .

Test 1 — Zero matrix: $[0000]$ exists. This corresponds to the vector $[0, 0, 0, 0]$ . ✓

Test 2 — Closed under addition:

[a_{1} c_{1} b_{1} d_{1}] + [a_{2} c_{2} b_{2} d_{2}] = [a_{1} + a_{2} c_{1} + c_{2} b_{1} + b_{2} d_{1} + d_{2}]

The result is still a $2 \times 2$ matrix with real entries. For example:

[1324] + [5768] = [610812]

✓

Test 3 — Closed under scalar multiplication: $α [a c b d] = [α a α c α b α d]$ . Result is a $2 \times 2$ matrix.

3 \cdot [1324] = [39612]

✓

Conclusion: The space of all $2 \times 2$ matrices IS a vector space. Answer: Yes. Dimension = 4.

4.10.4 Basis and Dimension of Matrix Spaces

The standard basis for the space of $2 \times 2$ matrices is:

B = {[1000], [0010], [0100], [0001]}

These are the four "unit matrices" — each has a 1 in exactly one position and 0 everywhere else. Any $2 \times 2$ matrix can be written as:

a [1000] + b [0010] + c [0100] + d [0001]

The dimension is 4 — there are four unique entries $(a, b, c, d)$ , each needing its own direction.

Generalization: The space of all $m \times n$ matrices is a vector space of dimension $m \cdot n$ .

dim (R^{m \times n}) = m \cdot n

So a $3 \times 3$ matrix space has dimension 9. A $6 \times 6$ matrix space has dimension 36. A $100 \times 100$ matrix space has dimension 10,000.

4.10.5 Matrices as Arrangements of Vectors

A matrix can be viewed in multiple ways. A $2 \times 2$ matrix $[a c b d]$ can be seen as:

1. A single 4-dimensional vector $[a, b, c, d]$ . 2. Two row vectors: $[a, b]$ and $[c, d]$ , each in $R^{2}$ . 3. Two column vectors: $[a, c]^{T}$ and $[b, d]^{T}$ , each in $R^{2}$ .

All three views are valid. Which one you use depends on the problem. The key insight: a matrix is an arrangement of vectors. Think of a dataset of houses. Each row could represent one house. Columns hold square footage, price, bedrooms, bathrooms. The dataset is a matrix. Each row is a vector for one data point. A matrix is a convenient way to organize many vectors together.

4.10.6 The Fundamental Equation $A X = B$ Revisited

The equation $A X = B$ is the most important equation in linear algebra. Here $A$ is a matrix of coefficients, $X$ is the unknown vector, and $B$ is the output vector.

Think of it this way: the matrix $A$ is like a force being applied to the vector $X$ . The force transforms $X$ into a new vector $B$ . $A$ is a transformation — it takes vectors from one position and moves them to another position. The direction of this force is given by eigenvectors, and the magnitude of this force is given by eigenvalues. These will be covered in later lectures.

This is the intuition to hold onto. When you multiply a matrix by a vector, you are transforming that vector. You might be scaling, rotating, shearing, or reflecting it. The matrix is the transformer.

4.10.7 Assumptions and Scope

Scope: The dimension of the matrix space is $m \cdot n$ . This is different from the row dimension ( $m$ ) or column dimension ( $n$ ). When you hear "a $4 \times 4$ matrix," the word "dimension" could refer to two things. First, the size of the row/column vectors: each row/column is in $R^{4}$ . Second, the dimension of the matrix space: all $4 \times 4$ matrices live in a 16-dimensional vector space.

Context disambiguates which is meant. In the exam, $dim (R^{m \times n}) = m \cdot n$ always refers to the matrix space dimension.

Key fact: Any $m \times n$ matrix can be flattened to a vector in $R^{mn}$ . The flattening is a vector space isomorphism — it preserves addition and scalar multiplication. The matrix space and $R^{mn}$ are structurally identical as vector spaces; only the arrangement of numbers differs.

4.10.8 Visual Intuition

Draw a $2 \times 2$ grid with entries $a, b, c, d$ . Now draw an arrow from the grid to a tall 4-element column vector $[a, b, c, d]^{T}$ . Show the reverse arrow — stacking four numbers back into a $2 \times 2$ grid. This bidirectional mapping is the flatten/unflatten operation.

For the basis, draw four small $2 \times 2$ grids, each with a 1 in a different position. Label them as the four basis matrices. Show that adding them with coefficients $a, b, c, d$ rebuilds the original matrix.

Takeaway: a matrix is just a vector arranged in a rectangle. The linear algebra works the same either way.

4.10.9 Pitfalls

- Confusing $dim (R^{m \times n}) = m \cdot n$ with $dim (R^{m}) = m$ . The matrix space dimension is the product, not the row count or column count or their sum. $m \cdot n$ is the number of independent entries. - Assuming row-flattening and column-flattening produce the same vector. They do not. $[a, b, c, d]^{T}$ (row-wise) versus $[a, c, b, d]^{T}$ (column-wise) are different vectors. But both are in $R^{4}$ and the dimension is the same. Just pick one convention and stick to it. - Thinking that only square matrices form a vector space. Any $m \times n$ matrix space is a vector space. Rectangular matrices work fine — the three litmus tests still pass. - Forgetting that flattening matters in practice. The CNN flattening layer connects convolutional outputs to dense layers. It is not just abstract math — it runs in every image classifier.

4.10.10 Student Q&A

Q: When we say $R^{3}$ is 3-dimensional but a $3 \times 3$ matrix has 9 dimensions — why the discrepancy?

A: $R^{3}$ refers to the space of 3-dimensional column vectors (like $[x, y, z]^{T}$ ) — its standard basis has 3 vectors. A $3 \times 3$ matrix has 9 unique entries. When flattened into a vector, it lives in a 9-dimensional space. You can also view a $3 \times 3$ matrix as an arrangement of three 3D vectors (rows or columns). But the matrix space itself has dimension 9. These are two different vector spaces — the former is the space of 3D vectors, the latter is the space of $3 \times 3$ matrices.

Q: So the confusion is: when we see a $4 \times 4$ matrix, the dimension looks like 4. Each row is a 4D vector. But mathematically, the matrix space has dimension 16?

A: Exactly. Suppose you have a $4 \times 4$ matrix. Each row represents one data point with 4 attributes — square footage, price, bedrooms, bathrooms. Then each row is a vector in $R^{4}$ . The matrix is a collection of such vectors. But consider the space of all possible $4 \times 4$ matrices as a vector space. The dimension is 16. You need 16 numbers to specify any $4 \times 4$ matrix uniquely. The two uses of "dimension" refer to different things. First, the dimension of the row/column vectors. Second, the dimension of the matrix space itself.

Several students asked about the dimension of matrix spaces. The professor clarified that $dim (R^{m \times n}) = m \cdot n$ . He also clarified that this is a distinct concept from the dimension of the individual row or column vectors.

4.10.11 Exam Guidance

Exam note: $dim (R^{m \times n}) = m \cdot n$ — NOT $m$ , $n$ , or $m + n$ . A $6 \times 6$ matrix space has dimension 36. A $2 \times 2$ matrix space has dimension 4. This is a common short-answer quiz question. Also, the basis for the matrix space is the set of $m \cdot n$ "unit matrices" with a single 1 and zeros elsewhere.

4.10.12 Recap and Bridge

Matrices are vectors in disguise. Any $m \times n$ matrix flattens into $R^{mn}$ . The space of all $m \times n$ matrices is a vector space of dimension $m \cdot n$ , with unit matrices as the standard basis. A matrix is also an arrangement of row or column vectors — multiple valid views, one underlying algebraic structure. The equation $A X = B$ ties it all together: the matrix acts as a transformation on vectors, a preview of eigenvalues and eigenvectors.

Matrix flattening is ubiquitous in deep learning. Every image classifier (ResNet, EfficientNet, ViT) ends with a flattening layer. It takes the final convolutional feature maps. These form a 3D tensor of shape $(H, W, C)$ . The layer reshapes it into a 1D vector. That vector feeds into a fully-connected layer that outputs class probabilities. The CNN's "understanding" of the image compresses into a single vector. It is the same linear algebra you just learned, scaled to millions of dimensions.

Exam Guidance Summary

Group Axioms

- Group identification quiz questions: You will get a set and an operation and must determine whether it is a group. Check all four properties: closure, associativity, identity, inverse. One failure disqualifies the entire structure. The classic trap is integers under multiplication (inverse of 5 is $1/5 \in / Z$ ). Natural numbers under addition often appear too — identity (0) is missing and inverses (negatives) are missing.

Vector Space Litmus Tests

- Vector space litmus tests: Always check three things. (1) Contains zero vector — any line/plane missing the origin fails instantly. (2) Closed under addition. (3) Closed under scalar multiplication — watch for bounded sets (squares, discs) that fail when scaled by large numbers. Any line or plane not passing through the origin cannot be a vector space.

Linear Combination Testing

- Linear combination via Gaussian elimination: Guaranteed exam question. Given a set of vectors and a target vector, determine whether the target is a linear combination. Form the augmented matrix. Use Gaussian elimination. Three outcomes: - Unique solution → yes. The coefficients are unique and the vectors are independent. - Infinitely many solutions → yes. The vectors are dependent, so there are multiple ways to form the target. - No solution (inconsistent) → no.

- Determinant shortcut: Determinants can be used for square matrices. If determinant is non-zero, there is a unique solution. For large matrices, Gaussian elimination is preferred. Determinants only work for square matrices — for non-square systems, row-echelon form is required.

Linear Independence

- Linear independence questions: Set up $λ_{1} v_{1} + \dots + λ_{n} v_{n} = 0$ and solve. Only trivial solution → independent. Any non-trivial solution → dependent. In $R^{n}$ , more than $n$ vectors are ALWAYS linearly dependent — answer in one line, no calculation needed. Any set containing the zero vector is automatically dependent.

Basis and Dimension

- Basis requires BOTH conditions: Linearly independent AND spanning the entire space. Two independent vectors in $R^{3}$ are NOT a basis — they span only a 2D plane and cannot reach anything with a non-zero Z component. A spanning set that is dependent is also not a basis (redundant vectors). Only exactly $dim (V)$ independent vectors in $V$ form a basis.

- Dimension of matrix space: $dim (R^{m \times n}) = m \cdot n$ , NOT $m$ or $n$ or $m + n$ . A $6 \times 6$ matrix space has dimension 36. A $3 \times 3$ matrix space has dimension 9. A $2 \times 2$ matrix space has dimension 4. This is a common short-answer quiz question.

- Standard basis: For $R^{3}$ : $[1, 0, 0]$ , $[0, 1, 0]$ , $[0, 0, 1]$ — these are the $i^, j^, k^$ unit vectors. For $m \times n$ matrix spaces: the $m \cdot n$ unit matrices, each with a single 1 in one position and zeros elsewhere.

Quick Reference

Concept	Formula/Key Fact
Group axioms	Closure, associativity, identity, inverse — one failure = not a group
Abelian group	Group + commutativity: $a \circ b = b \circ a$
Vector space tests	$0 \in V$ , closed under $+$ and scalar $\times$
Linear combination	$w = \sum α_{i} v_{i}$ ; test via Gaussian elimination
Span	$span {v_{i}} = {\sum α_{i} v_{i} ∣ α_{i} \in R}$ — always a subspace
Linear independence	$\sum λ_{i} v_{i} = 0 ⟹$ only trivial solution
Basis	LI + spanning; unique coordinates for every vector
Dimension	$dim (R^{n}) = n$ ; $dim (R^{m \times n}) = m \cdot n$
Matrix flattening	$m \times n$ matrix $\leftrightarrow$ vector in $R^{mn}$

Key Industry Applications

Data as Vectors — Text and Image Embeddings

Text, images, video, and audio are all represented as vectors (embeddings) for machine learning. Each data type lives in its own vector space. Images are pixel value vectors. Text is word frequency or token embedding vectors. Canonical Cross-Correlation (CCC) projects different modalities into a shared space, enabling multi-modal tasks. One production application is subtitle generation from lip movements. Models like LipNet project video frames and text into a shared vector space. They match lip shapes with words. Modern multi-modal models (GPT-4V, Gemini, CLIP) all rely on embedding text and images into comparable vector spaces.

Gradient Descent and Optimization

The error landscape in ML is navigated using vectors. Your current position in parameter space is a vector. The gradient direction — where the error increases fastest — is a vector. Each optimization update step is vector addition or subtraction. Specifically:

p_{new} = p_{old} - η \nabla E (p_{old})

This is covered in detail in Lectures 11–12. Every neural network training run, from a 3-layer perceptron to GPT, executes this vector update billions of times.

RGB Color Space

Every pixel color on a digital screen is a linear combination of Red, Green, and Blue basis vectors:

Color = r \cdot R + g \cdot G + b \cdot B

R, G, B are linearly independent — you cannot mix Green and Blue to get Red. Together they span the entire 16.7-million-color RGB space. CNNs decompose input images into R, G, B channels before applying convolution operations. Each channel is a matrix of pixel intensities — the primary input to the feature extraction pipeline.

CNN Flattening Layer

The flattening step at the end of every image classifier (ResNet, EfficientNet, ViT) takes the final convolutional feature maps. These form a 3D tensor of shape $(H, W, C)$ . The step reshapes it into a single 1D vector. That vector feeds into a fully-connected layer for classification. This is exactly the matrix-to-vector flattening discussed in Section 4.10. The CNN's learned visual understanding compresses into a single high-dimensional vector.

Portfolio Diversification (Finance)

Linear independence and span concepts directly apply to Modern Portfolio Theory. Assets whose returns are linearly dependent (two S&P 500 index funds) provide no diversification benefit. Independent assets span a wider range of risk-return profiles, enabling more efficient portfolios.

Recommendation Systems

Netflix and Spotify embed users and items into a shared vector space. Your taste vector sits near the vectors of content you would enjoy. The span of top rating vectors captures the most informative directions of viewer preference. This is a form of dimensionality reduction via basis selection.

Tools

- Linear algebra .info website: An interactive resource for visualizing span, linear combinations, and vector operations. Essential for building geometric intuition alongside the algebraic derivations in these notes.

MFML Lecture 04 notes · Vector Spaces, Linear Independence, Basis and Dimension

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-08

Sections Breakdown

1Review of Groups

Core concepts of Review of Groups in linear algebra for machine learning.

2Abelian Groups — Commutativity

Core concepts of Abelian Groups — Commutativity in linear algebra for machine learning.

3Vector Spaces: Inner and Outer Operations

Core concepts of Vector Spaces: Inner and Outer Operations in linear algebra for machine learning.

4Vector Spaces in Machine Learning

Core concepts of Vector Spaces in Machine Learning in linear algebra for machine learning.

5Litmus Tests for Vector Spaces

Core concepts of Litmus Tests for Vector Spaces in linear algebra for machine learning.

6Linear Combinations

Core concepts of Linear Combinations in linear algebra for machine learning.

7Span and Subspaces

Core concepts of Span and Subspaces in linear algebra for machine learning.

8Linear Independence and Dependence

Core concepts of Linear Independence and Dependence in linear algebra for machine learning.

9Basis and Dimension

Core concepts of Basis and Dimension in linear algebra for machine learning.

10Matrices as Vector Spaces

Core concepts of Matrices as Vector Spaces in linear algebra for machine learning.

11Exam Guidance Summary

Core concepts of Exam Guidance Summary in linear algebra for machine learning.

12Key Industry Applications

Core concepts of Key Industry Applications in linear algebra for machine learning.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review. If something does not make sense, go back to the full explanation in the main content.

Groups

Must-know: A group is a set with one binary operation obeying four rules. They are closure, associativity, identity, and inverse. One failure disqualifies it. Integers under multiplication fail the inverse test.

\forall a,b\in G,\; a\circ b\in G \quad\text{(closure)};\quad \exists e,\; \forall a,\; a\circ e=e\circ a=a \quad\text{(identity)}

⚠️ Top pitfall: Assuming integers under multiplication form a group. Closure, associativity and identity all pass, but the inverse of 5 is 1/5, which is not an integer. Always check inverses last.

Self-check: Given the set of natural numbers under addition, is it a group? Why or why not?

Connects to: Abelian groups, Vector spaces, Litmus tests

Abelian Groups

Must-know: An abelian group adds commutativity on top of the four group axioms: a∘b = b∘a. Vector addition is abelian; matrix multiplication is not.

\forall a,b\in G,\quad a\circ b = b\circ a

⚠️ Top pitfall: Assuming every group is abelian. Matrix multiplication, 3D rotations and permutations on 3+ objects are non-abelian yet still valid groups.

Self-check: Name one operation you use daily that is commutative and one that is not.

Connects to: Groups, Vector spaces

Vector Spaces

Must-know: A vector space is an abelian group under vector addition plus scalar multiplication from a field. Eight axioms total: four for addition, four for scaling.

\mathbf{v}+\mathbf{w}\in V,\quad \alpha\mathbf{v}\in V,\quad \alpha(\mathbf{v}+\mathbf{w})=\alpha\mathbf{v}+\alpha\mathbf{w}

⚠️ Top pitfall: Treating a scalar as a vector. A scalar lives outside the space; scalar multiplication is an outer operation that produces a vector still inside V.

Self-check: Why is scalar multiplication called an 'outer' operation while vector addition is 'inner'?

Connects to: Abelian groups, Litmus tests, Matrices as vector spaces

Vector Spaces in Machine Learning

Must-know: Every data point is a vector (an embedding). Gradient descent walks through parameter space by repeatedly subtracting a scaled gradient vector from the current position.

\mathbf{p}_{\text{new}} = \mathbf{p}_{\text{old}} - \eta \nabla E(\mathbf{p}_{\text{old}})

⚠️ Top pitfall: Confusing the data vector space with the parameter vector space. Gradient descent moves through parameter space, not the space of images or text.

Self-check: In the update rule, what does η control and what does ∇E point toward?

Connects to: Vector spaces, Span, Linear combinations

Vector Space Litmus Tests

Must-know: To test whether a subset of ℝⁿ is a subspace, check only three things. It contains the zero vector. It is closed under addition. It is closed under scalar multiplication.

\mathbf{0}\in V;\quad \forall\mathbf{v},\mathbf{w}\in V,\; \mathbf{v}+\mathbf{w}\in V;\quad \forall\alpha\in\mathbb{R},\; \alpha\mathbf{v}\in V

⚠️ Top pitfall: Assuming any line is a vector space. Only lines through the origin qualify. A line like y = x + 5 misses the origin and fails Test 1 instantly. Bounded sets fail scalar multiplication.

Self-check: Is the filled square −1≤x≤1, −1≤y≤1 a vector space? Test it against the three rules.

Connects to: Vector spaces, Span, Subspaces

Linear Combinations

Must-know: A linear combination scales vectors by scalars and adds them. Testing whether a target is a combination reduces to solving A𝐱 = 𝐛 via Gaussian elimination.

\mathbf{w} = \sum_{i=1}^{n} \alpha_i \mathbf{v}_i

⚠️ Top pitfall: Using determinants on non-square systems. Determinants only work for square matrices. For rectangular systems use row-echelon form.

Self-check: Given vectors and a target, what are the three possible outcomes of Gaussian elimination and what does each mean?

Connects to: Span, Linear independence, Vector spaces

Span and Subspaces

Must-know: The span of a set of vectors is every point reachable by linear combination. It is always a subspace — it automatically passes the three litmus tests.

\operatorname{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_n\} = \left\{\sum_{i=1}^{n}\alpha_i\mathbf{v}_i \;\middle|\; \alpha_i\in\mathbb{R}\right\}

⚠️ Top pitfall: Thinking more vectors always mean a bigger span. Adding a vector already in the span of the others adds nothing; you need independent vectors to grow the dimension.

Self-check: What is the span of the standard basis vectors [1,0,0], [0,1,0], [0,0,1] in ℝ³?

Connects to: Linear combinations, Linear independence, Basis

Linear Independence

Must-know: Vectors are independent if the only way to combine them to the zero vector is the trivial (all-zero) solution. Any non-trivial solution means dependence, that is, redundancy.

\lambda_1\mathbf{v}_1+\cdots+\lambda_n\mathbf{v}_n=\mathbf{0}\implies \text{independent iff } \lambda_1=\cdots=\lambda_n=0

⚠️ Top pitfall: Forgetting that in ℝⁿ, more than n vectors are ALWAYS dependent — answer in one line, no calculation. Any set containing the zero vector is automatically dependent.

Self-check: Are the vectors [1,0], [0,1], [2,2] independent? Set up the zero-vector test and solve.

Connects to: Span, Basis, Dimension

Basis and Dimension

Must-know: A basis is a set that is both linearly independent AND spans the space. It gives every vector a unique coordinate representation. Dimension is the size of any basis — an invariant of the space.

\dim(\mathbb{R}^n)=n;\quad \dim(\mathbb{R}^{m\times n})=m\cdot n

⚠️ Top pitfall: Two independent vectors in ℝ³ are NOT a basis. They span only a 2D plane. And dim(ℝ^{m×n}) = m·n, never m or n. A 6×6 matrix space has dimension 36.

Self-check: Why must a basis satisfy BOTH independence and spanning? Give a counterexample using ℝ³.

Connects to: Linear independence, Span, Matrices as vector spaces

Matrices as Vector Spaces

Must-know: Any m×n matrix flattens into a vector in ℝ^{mn}. The space of all m×n matrices is a vector space of dimension m·n. Unit matrices form the standard basis.

\dim(\mathbb{R}^{m\times n}) = m\cdot n;\quad \mathbf{A}=\begin{bmatrix}a&b\\c&d\end{bmatrix}\longleftrightarrow\begin{bmatrix}a\\b\\c\\d\end{bmatrix}

⚠️ Top pitfall: Confusing the dimension of the matrix space (m·n) with the dimension of its row/column vectors (m or n). A 4×4 matrix has 16-dimensional matrix space even though each row is in ℝ⁴.

Self-check: What is the dimension of the space of all 3×3 matrices, and what is its standard basis?

Connects to: Basis, Dimension, Vector spaces

Previous LectureLecture 03 · Groups, Vector Spaces, and Linear Combinations

Next LectureLecture 05 · Dot Products and Inner Products

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Vector Spaces, Linear Independence, Basis and Dimension

4.1 Review of Groups

4.1.1 What Is a Group

4.1.2 Symbol Registry

4.1.3 The Four Group Properties

4.1.4 Worked Example: Is (Z,×) a Group?

4.1.5 Assumptions and Scope

4.1.6 Visual Intuition

4.1.7 Pitfalls

4.1.8 Student Q&A

4.1.9 Exam Guidance

4.1.10 Recap and Bridge

4.2 Abelian Groups — Commutativity

4.2.1 Definition

4.2.2 Why Study Groups Before Vector Spaces

4.2.3 Assumptions and Scope

4.2.4 Pitfalls

4.2.5 Recap and Bridge

4.3 Vector Spaces: Inner and Outer Operations

4.3.1 Definition

4.3.2 Symbol Registry

4.3.3 Inner Operation — Vector Addition

4.3.4 Outer Operation — Scalar Multiplication

4.3.5 Assumptions and Scope

4.3.6 Visual Intuition

4.3.7 Pitfalls

4.3.8 Recap and Bridge

4.4 Vector Spaces in Machine Learning

4.4.1 Data as Vectors

4.4.2 Symbol Registry

4.4.3 The Error Landscape and Gradient Descent Intuition

4.4.4 Assumptions and Scope

4.4.5 Visual Intuition

4.4.6 Pitfalls

4.4.7 Student Q&A

4.4.8 Recap and Bridge

4.5 Litmus Tests for Vector Spaces

4.5.1 The Three Tests

4.5.2 Symbol Registry

4.5.3 Worked Example: Is the Y-axis a Vector Space?

4.5.4 Assumptions and Scope

4.5.5 Visual Intuition

4.5.6 Pitfalls

4.5.7 Student Q&A

4.5.8 Exam Guidance

4.5.9 Recap and Bridge

4.6 Linear Combinations

4.6.1 Definition

4.6.2 Symbol Registry

4.6.3 Geometric Intuition

4.6.4 Worked Example: Simple 3D Linear Combination

4.6.5 Worked Example: 4D Linear Combination via Gaussian Elimination

4.6.6 Assumptions and Scope

4.6.7 Visual Intuition

4.6.8 Pitfalls

4.6.9 Student Q&A

4.6.10 Exam Guidance

4.6.11 Recap and Bridge

4.7 Span and Subspaces

4.7.1 Definition of Span

4.7.2 Symbol Registry

4.7.3 Theorem: Span Is a Subspace

4.7.4 Span of Standard Basis Vectors

4.7.5 Assumptions and Scope

4.7.6 Visual Intuition

4.7.7 Pitfalls

4.7.8 Student Q&A

4.7.9 Recap and Bridge

4.8 Linear Independence and Dependence

4.8.1 Formal Definition

4.8.2 Symbol Registry

4.8.3 Why Non-Zero Lambdas Mean Dependence

4.8.4 Worked Example: Dependence in R2

4.8.5 Key Fact: Maximum Independent Vectors

4.8.6 Span of Independent vs Dependent Vectors

4.8.7 Assumptions and Scope

4.8.8 Visual Intuition

4.8.9 Pitfalls

4.1.4 Worked Example: Is $(Z, \times)$ a Group?

4.8.4 Worked Example: Dependence in $R^{2}$

4.9.4 Standard Basis for $R^{2}$ and $R^{3}$

4.10.3 Worked Example: Is the Space of All $2 \times 2$ Matrices a Vector Space?

4.10.6 The Fundamental Equation $A X = B$ Revisited