Mathematical Foundations for Machine Learning

Dot Products and Inner Products

📅 Published: 2026-07-09

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures in this subject. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Rank — introduced in Lecture 1 (1.8 Rank), then used here to interpret matrix transformations geometrically.
Vector Spaces — Lecture 4 (4.3 Vector Spaces: Inner and Outer Operations) gives the structure an inner product lives in.
Linear Combinations — Lecture 4 (4.6 Linear Combinations) underpins how orthonormal bases recombine vectors.
Span and Subspaces — Lecture 4 (4.7 Span and Subspaces) explains the space a set of vectors covers, used in Gram-Schmidt.
Linear Independence and Dependence — Lecture 3 (3.5) and Lecture 4 (4.8) are essential: Gram-Schmidt requires independent input vectors, and rank counts them.
Basis and Dimension — Lecture 4 (4.9 Basis and Dimension) connects to orthonormal bases and the rank-nullity relationship.

Dot Products and Inner Products

This lecture introduces the dot product as a fundamental operation on vectors, establishing its geometric meaning through the cosine of the angle between vectors. It then generalises the dot product to the concept of an inner product, an abstract bilinear form satisfying symmetry, linearity, and positive definiteness. Key topics include the Cauchy–Schwarz inequality, orthogonality, orthogonal projection, the Gram–Schmidt orthogonalisation algorithm, and the rank of a matrix.

5.1 Introduction to Dot Products

Machines cannot "look" at two users and decide they are similar. They need numbers. They need a mathematically precise way to answer one question: how alike are these two things?

The Core Question: Given two users with different tastes, two products with different features, or two documents with different words — how do you turn "they seem similar" into a concrete number that a machine can compute and compare? This is the question the dot product answers.

Every recommendation you get on Netflix, every "People You May Know" suggestion on LinkedIn, every song Spotify queues for you — all of them start with the same first step: representing people and things as lists of numbers. Once you have those numbers, you need a way to compare them. That way is the dot product.

5.1.1 Intuitive Motivation: Users, Movies, and Rideshares

Before formulas, an analogy. Think about how you decide if two people share your taste in music. You might ask: do they like the same artists? Do they listen to the same genres? You compare, feature by feature. Their similarity is not a yes/no answer — it is a matter of degree.

Analogy: Comparing Shopping Lists. Imagine two shoppers at a grocery store. Shopper A buys 5 apples, 0 bananas, and 3 oranges. Shopper B buys 1 apple, 4 bananas, and 0 oranges. To compare them, you look at each item category — fruit by fruit. They differ on apples, differ on bananas, differ on oranges. If you could multiply their counts item-by-item and sum them up, you would get one number capturing their overlap. That is exactly what the dot product does. Where the analogy breaks: A shopping list comparison assumes every item matters equally. In real ML problems, some features (like "bedrooms" for house prices) matter far more than others (like "number of taps"). The dot product treats all dimensions equally. Later, the inner product fixes this by letting you weigh features differently.

Now replace shopping items with movie genres. Imagine two users on a movie platform. User A likes horror and sci-fi. User B likes romance. Can you represent this preference pattern as numbers and measure how similar these two users are?

Representation as Vectors. You encode preferences by picking dimensions — one for each feature you care about — and assigning a numeric value to each dimension. For movie preferences, choose three genre-dimensions: horror, romance, sci-fi. Rate each on a 5-point preference scale.

A vector is an ordered list of numbers: $u = [u_{1}, u_{2}, \dots, u_{n}]$ . Each number $u_{i}$ is a component — it tells you how much of feature $i$ the vector has.
A feature space is the imaginary grid spanned by your chosen dimensions. Here, the space is $R^{3}$ — all possible combinations of horror, romance, and sci-fi preferences.
The vector encodes an object (a user, a movie, a document) as a point in this space.

Represent each user as a vector across three genre-dimensions: horror, romance, sci-fi. Use a 5-point preference scale. Now you have two vectors. But how similar are they? You need a number.

Worked Example: Movie Preference Vectors

User A: $u_{A} = [5, 0, 5]$ — likes horror a lot (5), does not like romance (0), very interested in sci-fi (5).
User B: $u_{B} = [0, 5, 1]$ — likes romance (5), sometimes watches sci-fi (1).

Step 1 — Component-by-component comparison. Horror: 5 vs 0 — completely different. Romance: 0 vs 5 — completely different. Sci-fi: 5 vs 1 — some overlap. Qualitatively, these users share little.

Step 2 — Geometric interpretation. If you plot these vectors in 3D space, User A's vector points strongly along the horror and sci-fi axes. User B's vector points along the romance axis with a slight sci-fi tilt. They point in visibly different directions. The angle between them is large.

Step 3 — Computing the similarity number. Multiply matching components and add them up:

u_{A} \cdot u_{B} = (5 \times 0) + (0 \times 5) + (5 \times 1) = 0 + 0 + 5 = 5

That number — 5 — is the dot product. It captures the degree of overlap between these two users. The dot product is relatively small because these users have little in common. If both users loved the same genres, the dot product would be much larger.

A second motivating example reinforces the idea. Consider Uber trip data with two metrics: number of trips taken and amount of money spent. User A takes fewer trips and spends less. User B takes more trips and spends more. They are relatively similar — both lie along the same general spending trend. A very different User C takes many trips but spends little. How do you formalize this relative similarity? Again: represent each user as a vector, then compute a number that captures the relationship.

Scope: When Vector Representation Works

Vector representation works when:

You can list a fixed set of features (dimensions) that describe every object.
Each feature can be assigned a numeric value on a meaningful scale.
The features are comparable — a 1-point difference in horror means roughly the same as a 1-point difference in sci-fi.

Vector representation breaks down when:

Preferences are contextual — a user might love horror movies with friends but hate them alone. A single number cannot capture this.
Features are categorical with no natural order — movie genres have an order, but what about "favorite color"? You cannot rank red above blue without making arbitrary choices.
Relationships are non-linear — if enjoyment of a genre depends on time of day, season, or a specific actor, a simple vector cannot encode these interactions.

The dot product assumes a linear world. Real preferences are messier. But this linear first step is powerful enough to drive most recommendation engines in production today.

5.1.2 Geometric Intuition: Visual Proximity as Similarity

When vectors are plotted in 2D or 3D, those pointing in similar directions or lying close to one another suggest similarity. This is visually apparent in low dimensions.

Picture a 3D plot with three labeled axes: horror on the X-axis, romance on the Y-axis, and sci-fi on the Z-axis. User A's vector $[5, 0, 5]$ shoots into the corner where horror and sci-fi meet — far along X, zero along Y, far along Z. User B's vector $[0, 5, 1]$ climbs the romance axis with a slight forward lean into sci-fi. The angle between them is large. They point to completely different corners of the preference cube. You can see, with your own eyes, that these users are dissimilar. The geometric challenge is turning what your eyes see into a number a computer can use.

But real machine learning data lives in dozens, hundreds, or thousands of dimensions. You cannot draw a thousand-dimensional plot. You need an algebraic tool that gives you the same information — a number that encodes whether two vectors point together, point apart, or are completely unrelated. That tool is the dot product.

Common Pitfalls: Magnitude vs. Direction

Trap 1: Confusing magnitude with similarity. A vector with large component values (e.g., $[100, 50, 100]$ ) is not automatically similar to another vector with large values. Two vectors can both have enormous magnitudes but point in opposite directions — they would be completely dissimilar. The dot product cares about direction and magnitude together, not either in isolation.
Trap 2: Assuming large dot products always mean similarity. A large dot product can happen because both vectors genuinely point in the same direction, or because one vector is extremely long. A user who rates everything a 5 will have a large dot product with almost everyone — not because they are similar, but because their vector has large magnitude.
Trap 3: Relying on sign agreement alone. User A $[5, 0, 5]$ and User C $[5, 4, 1]$ both have positive horror ratings. But their romance preferences are opposite. Component-by-component comparison matters, not just whether signs match.

The solution to the magnitude problem is to measure only the angle between vectors, ignoring their lengths. That is what the geometric form of the dot product — involving the cosine of the angle — provides. But first you must master the algebraic form.

Q: Where do we use dot products in reality? Why does this matter? A: The geometric definition gives the answer. If two vectors have an angle close to zero, their dot product is maximized — they are nearly collinear, meaning highly similar. If the dot product equals zero, the vectors are orthogonal — not similar at all, pointing in completely different directions. This binary classification — similar vs. not similar — powers recommendation systems at Netflix, friend suggestions at Facebook, and connection recommendations at LinkedIn. Every time a platform decides "you might like this," a dot product was likely involved somewhere in the pipeline. The same principle extends to text embeddings in GPT and Gemini, where word and sentence meanings are compared through dot products in the transformer attention mechanism.

Recap + Bridge. Vectors let you encode objects — users, movies, songs, documents — as points in a numeric feature space. Once encoded, you need a single number that captures how similar two objects are. That number is the dot product: multiply matching components and add. A large positive value means alignment. A value near zero means no relationship. A negative value means opposition. Next: the precise mathematical definition of the dot product — its component-wise formula, its matrix notation, and its geometric interpretation via lengths and angles.

Netflix uses vector representations of users and movies to power its recommendation engine. Spotify does the same for songs and playlists. Uber uses them for rider-driver matching. LinkedIn represents your profile as a vector to suggest connections. In every case, the pipeline is identical: extract features, build vectors, compute dot products. This first step — turning messy real-world objects into clean numeric vectors — is called feature extraction. It is the entry point to every machine learning pipeline, from simple linear classifiers to deep neural networks. Without it, there is no data to learn from.

5.2 Mathematical Formulation of Dot Product

Hook: From Visual Intuition to Computation. You can represent two Netflix users as vectors in a high-dimensional space. You can even plot them and see that they point in different directions. But visual similarity is not enough for a machine. How do you turn that sense of "these two look similar" into a single number you can compute, store, and compare automatically?

The answer lies in a deceptively simple operation. Its name is ordinary, but its reach is vast. Welcome to the dot product.

Intuition: The Shopping List Analogy. The dot product is like comparing two shopping lists item by item. Suppose you and your roommate each list how many apples, bananas, and oranges you want. To compare your lists, you multiply the apples you want by the apples they want, do the same for bananas and oranges, then add everything up. The total tells you how much your preferences overlap. More overlap means a bigger number. No overlap means zero. Contradictory preferences push the total negative. That is exactly how the dot product quantifies similarity — one feature at a time, summed across all dimensions.

5.2.1 Component-wise Definition

Component-wise Dot Product. For two vectors

A

and

B

R^{n}

, the dot product multiplies corresponding components and sums the results:

A \cdot B = i = 1 \sum n A_{i} B_{i}

Every symbol has a name:

$A_{i}$ : the $i$ -th component of vector $A$
$B_{i}$ : the $i$ -th component of vector $B$
$\sum_{i = 1}^{n}$ : sum from the first component to the $n$ -th component
$A \cdot B$ : the dot product of $A$ and $B$ (pronounced "A dot B")

The output is a single scalar — not a vector, not a matrix. You feed in two vectors and get back one number.

Worked Example: User Similarity. Recall User A and User B from the introduction:

A = [5, 0, 5], B = [0, 5, 1]

A \cdot B = (5 \times 0) + (0 \times 5) + (5 \times 1) = 0 + 0 + 5 = 5

Both users have a slight overlap on sci-fi (User A rates it 5, User B rates it 1), but diverge completely on horror and romance. The dot product captures this partial alignment as a single scalar: 5.

What does this scalar mean? A positive value says the vectors point in roughly the same general direction. A negative value says they point in opposite directions. A value near zero says they are almost perpendicular — no strong alignment either way. The magnitude is influenced by both the lengths of the vectors and how well they align.

5.2.2 General Form for N-Dimensional Vectors

Dot Product in $R^{n}$ . For vectors

x, y \in R^{n}

x \cdot y = x_{1} y_{1} + x_{2} y_{2} + \dots + x_{n} y_{n}

The pattern never changes, no matter how many dimensions you have. Multiply the first components together. Multiply the second components together. Continue through every dimension. Add the results. The dot product scales cleanly to any finite number of dimensions — 3, 300, or 3 million. This is why it works for Netflix embedding vectors with hundreds of features.

5.2.3 Matrix Notation

Dot Product as Matrix Multiplication. The component-wise sum can be written compactly using the transpose operator. For

x, y \in R^{n}

x \cdot y = x^{T} y = y^{T} x

Why does $x^{T} y = y^{T} x$ ? Both produce $\sum_{i = 1}^{n} x_{i} y_{i}$ . The dot product is symmetric. Multiplying the components in order $(x_{1} y_{1} + \dots + x_{n} y_{n})$ or $(y_{1} x_{1} + \dots + y_{n} x_{n})$ gives the same scalar. Transposing one vector and multiplying by the other is just a computational rearrangement, not a different operation.

Shape compatibility: $x^{T}$ is a $1 \times n$ row vector. $y$ is an $n \times 1$ column vector. Their product has shape $1 \times 1$ — exactly one scalar. You can only multiply when the number of columns of the first equals the number of rows of the second. Here: $n = n$ . The result is always $1 \times 1$ .

Worked Example: Matrix Notation with Shape Annotation. For

x = [2, - 3]

and

y = [5, 1]

1 \times 2 x^{T} 2 \times 1 y = [2 - 3] [51] = (2 \times 5) + (- 3 \times 1) = 10 - 3 = 7

Inner dimensions match ( $2 = 2$ ), outer dimensions give the result shape ( $1 \times 1$ ).

Student Correction Note: During lecture, a student pointed out that the vector might have been $[- 2, 3]$ instead of $[2, - 3]$ . Let us verify:

[- 2, 3] \cdot [5, 1] = (- 2 \times 5) + (3 \times 1) = - 10 + 3 = - 7

Both versions are valid examples of the matrix formulation. With $[2, - 3]$ the dot product is $+ 7$ (acute angle, $cos θ > 0$ ). With $[- 2, 3]$ the dot product is $- 7$ (obtuse angle, $cos θ < 0$ ). The geometric principle — dot product sign tracks the angle class — holds either way. The professor acknowledged the correction in class and confirmed the core idea remains unchanged.

5.2.4 Geometric Definition: Angle Formulation

Dot Product via Length and Angle. The dot product connects algebra to geometry through the angle

θ

between two vectors:

x \cdot y = ∥ x ∥ ∥ y ∥ cos θ

Every symbol:

$∥ x ∥$ : Euclidean norm (length) of $x$
$∥ y ∥$ : Euclidean norm (length) of $y$
$θ$ : the angle between $x$ and $y$ , measured in radians or degrees
$cos θ$ : cosine of that angle, ranging from $- 1$ to $+ 1$

This single equation encodes everything about how two vectors relate geometrically. The dot product is the product of their magnitudes, scaled by how much they point in the same direction.

Visual Intuition. Picture a 2D coordinate plane. Draw vector $a = [2, - 3]$ . It starts at the origin, goes right 2 units, then down 3 units — pointing into the fourth quadrant. Now draw $b = [5, 1]$ from the origin. It goes right 5 units and up 1 unit, pointing into the first quadrant. The angle $θ$ between them sits in the plane, measured as the smaller rotation needed to align one with the other. For these vectors:

cos θ = \frac{a \cdot b}{∥ a ∥∥ b ∥} = \frac{7}{13 \cdot 26} \approx 0.381

θ \approx 67. 6^{\circ}

This is an acute angle — the vectors lean toward each other. Now visualize the three regimes:

$θ \approx 0^{\circ}$ (collinear, pointing same way): the vectors overlap almost perfectly. $cos θ \approx 1$ , so the dot product approaches its maximum possible value $∥ x ∥∥ y ∥$ .
$θ \approx 9 0^{\circ}$ (perpendicular): the vectors share no common direction. $cos 9 0^{\circ} = 0$ , so the dot product is exactly zero. This is the test of orthogonality — zero dot product means the vectors are geometrically independent, pointing in completely unrelated directions.
$θ > 9 0^{\circ}$ (obtuse, pointing apart): $cos θ$ becomes negative, so the dot product goes negative. The vectors work against each other instead of aligning.

5.2.5 Computing Vector Length via Self-Dot-Product

Length from Dot Product. The Euclidean norm (length) of a vector emerges naturally from the dot product of the vector with itself:

∥ x ∥ = x \cdot x = x^{T} x

Take the dot product of a vector with itself. You get the sum of squared components. Take the square root. The result is the length. This is why the dot product is sometimes called a "length engine" — it generates the Euclidean distance metric.

Worked Example: Length of $[3, 4]$ . Two equivalent paths:

Path 1 — Pythagoras:

∥ x ∥ = 3^{2} + 4^{2} = 9 + 16 = 25 = 5

Path 2 — Self-dot-product:

x^{T} x = [34] [34] = 9 + 16 = 25

∥ x ∥ = x^{T} x = 25 = 5

Both paths yield 5. The self-dot-product generalises Pythagoras to any number of dimensions.

Symbol Registry for Section 5.2:

Symbol	Meaning	Type	Shape
$x, y$	Vectors in $R^{n}$	Vector	$n \times 1$
$x_{i}, y_{i}$	$i$ -th components	Scalar	Scalar
$x^{T}$	Transpose of $x$	Row Vector	$1 \times n$
$\cdot$	Dot product operator	—	—
$∥ x ∥$	Euclidean norm (length)	Scalar	Scalar
$θ$	Angle between vectors	Scalar (radians/degrees)	Scalar
$cos θ$	Cosine of the angle	Scalar	Scalar

Scope and Assumptions. The dot product as defined here operates on vectors in

R^{n}

— the space of real numbers. Both vectors must have the same dimension. Components are real-valued. The geometric form

∥ x ∥∥ y ∥ cos θ

assumes a Euclidean space with the standard notion of angle and distance. The identity

x^{T} y = y^{T} x

holds because multiplication of real numbers commutes. In complex vector spaces (

C^{n}

), the conjugate transpose replaces the ordinary transpose — that is a different story for a different day.

Common Pitfalls.

Forgetting the square root. $x^{T} x$ gives the squared length, not the length. If you compute $x^{T} x = 25$ and claim the length is 25, you are off by the square root. Always remember $∥ x ∥ = x^{T} x$ .
Mixing row and column vectors. The dot product requires one row vector and one column vector: $1 \times n$ times $n \times 1$ . Writing a row times a row ( $1 \times n$ times $1 \times n$ ) has mismatched inner dimensions and produces garbage, not a scalar.
Confusing dot product magnitude with similarity. A large dot product does not always mean high similarity — it could mean one vector is simply very long. For similarity independent of scale, use the normalised dot product (cosine similarity): $\frac{x \cdot y}{∥ x ∥∥ y ∥}$ .
Misreading the sign. Positive dot product $⟹$ acute angle ( $< 9 0^{\circ}$ ). Negative dot product $⟹$ obtuse angle ( $> 9 0^{\circ}$ ). Zero dot product $⟹$ right angle (orthogonal, exactly $9 0^{\circ}$ ). The sign tells you the angle class — not the exact angle, but whether the vectors lean together, push apart, or are perpendicular.

Several students had questions about the practical details of dot products. Here are the most important ones, with the professor's answers.

Q: A student pointed out during the worked example — shouldn't the vector be

[- 2, 3]

instead of

[2, - 3]

? That would give -7 instead of +7. Which is correct? A: The professor acknowledged the point. Both versions are valid. With

[2, - 3] \cdot [5, 1] = 7

, the angle is acute — the vectors lean in the same general direction. With

[- 2, 3] \cdot [5, 1] = - 7

, the angle is obtuse. What matters is the principle: if the angle measured anticlockwise from the first vector does not exceed

18 0^{\circ}

from the first vector, the dot product remains positive. The geometric relationship — dot product sign tracks the angle class — holds regardless of which pair you use.

Q: Why is one vector written as a row and the other as a column in the matrix form

x^{T} y

? A: Matrix multiplication has a shape rule: you can only multiply when the inner dimensions match.

x^{T}

is a

1 \times n

row vector.

y

is an

n \times 1

column vector. Their inner dimensions are both

n

, so multiplication is valid. The result is

1 \times 1

— a scalar. If you tried to multiply two column vectors or two row vectors, the inner dimensions would not match. The transpose makes the shapes compatible.

Q: If there are multiple vectors — say four users — how do you compare them? Do you multiply two at a time and then combine? A: Dot product is a binary operation. You can only take the dot product between two vectors at a time. For multiple users, you compute a dot product similarity matrix: a square table where entry

(i, j)

X_{i} \cdot X_{j}

. The diagonal entries give squared lengths — generally not of primary interest. Correlation works the same way — it compares two variables at a time, not many at once. The similarity matrix captures every pairwise relationship in a single organized structure.

Recap and Bridge. The dot product compresses two vectors into a single similarity number using three equivalent forms: the component-wise sum

\sum x_{i} y_{i}

, the matrix product

x^{T} y

, and the geometric product

∥ x ∥∥ y ∥ cos θ

. Every form says the same thing in a different language. If the result is positive, the vectors lean together. If zero, they stand perpendicular. If negative, they push apart. What comes next? What if some features matter more than others? What if you want to say "bedrooms count four times as much as taps" when computing similarity? The answer lies in replacing the implicit identity matrix inside the dot product with a custom weight matrix — giving birth to the inner product. That is Section 5.3.

Real-World and Domain Connections. The dot product is not just classroom mathematics. It powers the recommendation engines you use every day. Netflix converts your watch history into a vector; each other user also becomes a vector. Pairwise dot products rank every other user by similarity to you, surfacing "Users Like You Also Watched." LinkedIn does the same for "People You May Know" — your profile vector is dotted against millions of others, and high-scoring matches appear in your feed. Google Photos embeds face images as vectors in a learned feature space; dot products group photos of the same person without ever knowing their name.

In information retrieval, cosine similarity — the normalised dot product — is the standard metric for ranking search results. A query vector is dotted against a corpus of document vectors. The documents with the highest cosine similarity appear at the top of your search results. Search engines, academic databases, and AI-powered Q&A systems all rely on this same operation. OpenAI embedding models use exactly this principle: their small embedding models produce vectors of about 300 dimensions, while large models produce vectors of about 768 dimensions — both compared via dot-product-based cosine similarity for semantic search and clustering.

In Transformer architectures, the dot product is the computational core of the attention mechanism. Every token in a GPT model computes dot products against every other token to produce attention scores — deciding which words in the sequence are relevant to each other. The scaled dot-product attention formula $Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$ is a direct descendant of the dot product you studied in this section, operating on millions of vectors in parallel across every Transformer layer. The dot product is the engine underneath the surface of modern information systems.

5.3 Inner Products as a Generalization

Hook: Why the dot product isn't always enough.

The dot product treats every feature equally. In $x^{T} y$ , each component gets exactly the same importance. But in the real world, features are rarely equal. When predicting house prices, the number of bedrooms matters way more than the number of taps. If you use a standard dot product, two houses identical in bedrooms but wildly different in taps would appear very similar — and that's a terrible model.

You need a similarity measure that weights features differently. That's exactly what an inner product gives you.

Take a moment to absorb that. The dot product is the "one-size-fits-all" ruler of geometry. It works perfectly when every dimension deserves equal attention. But you wouldn't judge a job applicant's resume by counting all words equally — you weight relevant experience higher than filler phrases. An inner product lets you build that weighting directly into your math.

5.3.1 From Identity Matrix to Arbitrary Matrix

Look at the dot product with fresh eyes. The identity matrix $I$ has been hiding in plain sight all along:

x^{T} y = x^{T} I y

For $x, y \in R^{2}$ :

x^{T} I y = [x_{1} x_{2}] [1001] [y_{1} y_{2}] = x_{1} y_{1} + x_{2} y_{2}

The identity matrix "does nothing" — multiply any vector by $I$ and it comes out unchanged. That's exactly why the standard dot product weights everything equally. If $I$ is the engine driving the dot product, then swapping in a different engine would give you a different product. That engine swap is the big idea.

Analogy: The custom ruler.

Think of the dot product as a standard ruler with equally spaced markings. Every inch is just like every other inch.

An inner product is a custom ruler where some markings are stretched apart (features you care about more) and others are squeezed together (features you care about less). The matrix $A$ is the blueprint for your custom ruler — it tells you exactly how much to stretch or squeeze each direction.

Every inner product defines its own private geometry, its own way of measuring "how far" and "how similar." When you change $A$ , you reshape space itself.

5.3.2 Replacing I with a Positive Definite Matrix A

What if you replace $I$ with any other matrix $A$ ? Not just any matrix will do. The replacement must preserve the geometric integrity that makes lengths real and distances meaningful. The class of matrices that works is the symmetric positive definite matrices.

Definition: Inner Product

⟨ x, y ⟩ = x^{T} A y

This is called an inner product. It generalizes the dot product. When $A = I$ , you get back the standard dot product.

Two conditions must hold for $A$ to define a valid inner product:

Symmetry: $A^{T} = A$ . The matrix equals its own transpose. Entry $a_{ij}$ always equals $a_{j i}$ . Flip the matrix across its main diagonal and you get the same matrix back.
Positive definiteness: $x^{T} A x > 0$ for every non-zero vector $x$ . The quadratic form must be strictly positive unless $x = 0$ .

Sandwiching $A$ between $x^{T}$ and $x$ and demanding the result be positive — that's the definition of a positive definite matrix.

Notice the angle-bracket notation $⟨ x, y ⟩$ versus the dot notation $x \cdot y$ . In formal writing, inner products always use angle brackets. In conversation, people often say "dot product" when they technically mean "inner product." They're interchangeable only when $A = I$ .

The matrix $A$ applies feature-specific weights. Put a larger entry on the diagonal for the dimension you want to emphasize. Put a smaller entry for dimensions you want to downplay. Off-diagonal entries encode interactions between features.

5.3.3 Positive Definite Matrix: Intuition and Worked Example

Worked Example: The simplest positive definite matrix.

Let $A = [4001]$ and $x = [x_{1}, x_{2}]^{T}$ :

x^{T} A x = [x_{1} x_{2}] [4001] [x_{1} x_{2}] = 4 x_{1}^{2} + x_{2}^{2}

Every term is a square multiplied by a positive coefficient. Feed in any value for $x_{1}$ and any value for $x_{2}$ . The result is always positive. Squares cannot be negative, and positive coefficients cannot make them flip sign. That's why this matrix is called positive definite:

x^{T} A x > 0 for all x \neq = 0

This property — always positive — is non-negotiable. It's what guarantees that the length of any vector, defined as $⟨ x, x ⟩$ , is a real, non-imaginary number. Without positive definiteness, you could get negative "squared lengths," which would make lengths imaginary. Geometry collapses.

Geometric intuition for positive definite matrices. Every matrix applies a transformation — stretching, shrinking, or rotating. When you multiply a vector by a positive definite matrix $A$ , the transformed vector $A x$ will never rotate beyond 180° from the original direction of $x$ . It can stretch, it can shrink, but it cannot flip into the opposite half-space. The transformed vector always lands on the same "side" of the origin as the original. That constraint is exactly what keeps $x^{T} (A x) > 0$ — the dot product of the original vector with its transformed version stays positive.

Worked Example: Stretching a vector.

Take $A = [4001]$ and $x = [2, 1]^{T}$ :

A x = [4001] [21] = [81]

The first dimension got amplified by a factor of 4. The second dimension stayed unchanged. Now compute the inner product:

⟨ x, x ⟩ = x^{T} (A x) = [21] [81] = 16 + 1 = 17 > 0

Always positive, as guaranteed. Now connect this to the house price example: $x = [2 bedrooms, 100 taps]$ . After applying $A$ , the weighted vector becomes $[8 bedrooms-worth, 100 taps-worth]$ . The bedroom dimension is now comparable in magnitude to the taps dimension. The matrix amplified what matters.

If you wanted to emphasize a different feature, you'd place a larger weight in that diagonal position. Want taps to matter twice as much as bedrooms? Use $A = [1002]$ . The diagonal entries are your feature-importance knobs.

Visual Intuition: Unit circle becomes unit ellipse. With the standard inner product $(A = I)$ , the set of all vectors of length 1 satisfies $x_{1}^{2} + x_{2}^{2} = 1$ — the familiar unit circle. Every point on that circle is exactly one unit from the origin in every direction.

Now impose $A = [4001]$ . The set of unit vectors satisfies:

4 x_{1}^{2} + x_{2}^{2} = 1 ⟹ \frac{x _{1}^{2}}{( 1/2 ) ^{2}} + \frac{x _{2}^{2}}{1 ^{2}} = 1

That's an ellipse. The x₁-axis is squashed — you reach unit length at $x_{1} = \pm \frac{1}{2}$ instead of $\pm 1$ . The x₂-axis remains unchanged. The inner product literally reshapes space. What counts as "equal distance" depends on which direction you travel.

Scope: Positive definite vs. positive semi-definite.

A matrix is positive definite if $x^{T} A x > 0$ for all $x \neq = 0$ . The inequality is strict.

A matrix is positive semi-definite if $x^{T} A x \geq 0$ for all $x$ , allowing zero even when $x \neq = 0$ . Covariance matrices in statistics are positive semi-definite — they can be singular.

Only positive definite matrices (strict inequality) define valid inner products. Semi-definiteness is a related but weaker condition. If $A$ is only semi-definite, the "length" of a non-zero vector could be zero, which breaks the geometry.

5.3.4 Proving Positive Definiteness: Class Exercise

Given a symmetric matrix, how do you prove it is positive definite? The most direct method: compute $x^{T} A x$ and rewrite it as a sum of squares. If every term is a square and at least one is non-zero whenever $x \neq = 0$ , the matrix is positive definite.

Fully Worked Example: Proving a 3×3 matrix is positive definite.

Prove that $A = 210 13 - 1 0 - 1 2$ is positive definite.

Let $x = [a, b, c]^{T}$ .

Step 1 — Compute $A x$ :

A x = 210 13 - 1 0 - 1 2 a b c = 2 a + b a + 3 b - c - b + 2 c

Step 2 — Compute the quadratic form $x^{T} A x$ :

x^{T} A x = a (2 a + b) + b (a + 3 b - c) + c (- b + 2 c) = 2 a^{2} + ab + ab + 3 b^{2} - b c - b c + 2 c^{2} = 2 a^{2} + 2 ab + 3 b^{2} - 2 b c + 2 c^{2}

Step 3 — Complete the square (decompose into sum of squares):

2 a^{2} + 2 ab + 3 b^{2} - 2 b c + 2 c^{2} = a^{2} + (a^{2} + 2 ab + b^{2}) + b^{2} + (b^{2} - 2 b c + c^{2}) + c^{2} = a^{2} + (a + b)^{2} + b^{2} + (b - c)^{2} + c^{2}

Step 4 — Analyze the sign. Every term is a perfect square: $a^{2}$ , $(a + b)^{2}$ , $b^{2}$ , $(b - c)^{2}$ , and $c^{2}$ . Each is non-negative. For the sum to be zero, every single square must be zero simultaneously. That forces $a = 0$ , $b = 0$ , and $c = 0$ . For any non-zero vector, at least one square term is positive.

Step 5 — Conclude. $x^{T} A x > 0$ for all $x \neq = 0$ . So $A$ is positive definite.

The sum-of-squares method is your primary direct proof tool. But there are two other equivalent ways to verify positive definiteness:

Eigenvalue test: All eigenvalues of $A$ are strictly positive ( $λ_{i} > 0$ ). For a symmetric matrix, real eigenvalues are guaranteed. This test connects positive definiteness to the spectral decomposition you will study next.
Symmetry verification: First confirm $A^{T} = A$ . For our 3×3 example, look at the off-diagonal entries: $a_{12} = a_{21} = 1$ , $a_{13} = a_{31} = 0$ , $a_{23} = a_{32} = - 1$ . Mirror entries match — the matrix is symmetric. A non-symmetric matrix cannot define an inner product.

Common Pitfalls:

Diagonal entries alone aren't enough. A diagonal matrix with positive entries like $[100100]$ is positive definite, but positive diagonal entries do not guarantee positive definiteness for matrices with off-diagonal terms. Always check the full quadratic form.
Positive definite vs. positive semi-definite. The distinction is the strict inequality. Positive definite means $> 0$ for non-zero vectors. Semi-definite means $\geq 0$ , which allows the quadratic form to vanish for a non-zero vector (e.g., a singular covariance matrix). Only the strict version defines a valid inner product.
Symmetry is mandatory. A non-symmetric matrix — even one satisfying $x^{T} A x > 0$ — cannot define an inner product. The symmetry axiom $⟨ x, y ⟩ = ⟨ y, x ⟩$ would be violated. Always verify $A^{T} = A$ before anything else.
The condition $x \neq = 0$ is essential. You only check positive definiteness for non-zero vectors. $x^{T} A x = 0$ when $x = 0$ is always true and tells you nothing.

Several students asked questions clarifying the inner product concept during the lecture. Here are the professor's answers to the most important ones.

Q: We already have the dot product for computing similarity. Where do we actually use the inner product instead? A: Think of inner product as a weighted dot product. When all features matter equally — say, every movie genre is equally important for recommendations — the standard dot product (where

A = I

) works fine. But reality is different. If you are predicting house prices, the number of bedrooms matters far more than the number of taps. With the inner product

⟨ x, y ⟩ = x^{T} A y

, you can set

A = [4001]

to give bedrooms four times the weight of taps. When all weights are equal,

A

reduces to the identity matrix — and you are back to the standard dot product. The inner product is the more general tool; the dot product is the special case where every feature is equally important.

Q: So if you wanted to increase the importance of the second feature instead of the first, you would place a larger weight in that position of the matrix? A: Exactly. The diagonal entries of

A

set the importance weight of each dimension. More weight to whatever dimension you want to emphasize. But the implications go deeper than feature weighting. Positive definite matrices ensure convex loss functions in optimization — a Hessian matrix that is positive definite guarantees your loss function has a unique global minimum. Gradient descent will converge to that minimum regardless of where you start. If the loss function is not convex, you risk getting trapped in a local minimum instead of finding the best solution.

Recap + Bridge. An inner product generalizes the dot product by inserting a symmetric positive definite matrix

A

between the vectors:

⟨ x, y ⟩ = x^{T} A y

. It's a weighted similarity measure — the diagonal of

A

dials up or down the importance of each feature. Positive definiteness (

x^{T} A x > 0

for

x \neq = 0

) ensures that lengths remain real and geometry stays coherent. Every valid inner product must satisfy three axioms: symmetry, bilinearity, and positive definiteness. Those are the rules of the game. Next, Section 5.4 unpacks each of those three properties in detail and shows you why they matter.

Real-world connection: Why positive definite matrices dominate machine learning. Positive definite matrices appear everywhere in ML practice:

Hessian matrices in optimization. The Hessian (matrix of second derivatives of a loss function) being positive definite means the loss function is locally convex. A convex function has a unique global minimum — your optimizer can't get trapped in a local minimum. This is the mathematical guarantee behind why gradient descent converges reliably on well-behaved problems.
Covariance matrices in statistics and PCA. Every covariance matrix $Σ$ is symmetric positive semi-definite. When $Σ$ is positive definite (full rank), it's invertible and defines a valid inner product $⟨ x, y ⟩ = x^{T} Σ^{- 1} y$ — the Mahalanobis distance inner product, which measures distance in units of standard deviation rather than raw coordinates.
Kernel matrices in SVMs. The Gram matrix (kernel matrix) in Support Vector Machines must be positive semi-definite for the optimization to be convex. This is Mercer's theorem in action — the kernel function implicitly defines an inner product in a high-dimensional feature space.

Symbol Registry for Section 5.3:

Symbol	Meaning	Type	Shape
$I$	Identity matrix	Matrix	$n \times n$
$A$	Symmetric positive definite matrix	Matrix	$n \times n$
$⟨ x, y ⟩$	Inner product of $x$ and $y$	Scalar	Scalar
$x^{T} A y$	Inner product in matrix form	Scalar	Scalar

5.4 Properties of Inner Products

Hook: Three Rules That Define Geometry

A dot product isn't just any formula someone invented. It follows three strict rules. These rules are so fundamental that ANY operation following them deserves the title "inner product." What are they?

Symmetry, bilinearity, and positive definiteness. Isolate these three — remove everything else about the standard dot product — and you have the blueprint for building entirely new ways to measure similarity, length, and angle. This is the bridge from "one ruler" to "infinitely many rulers."

The dot product `[5, 0, 5] · [0, 5, 1] = 5` gave us a similarity score. The inner product `⟨x, y⟩ = xᵀAy` gave us a weighted version of that score. But what makes these operations valid? What stops someone from inventing a nonsense formula and calling it an inner product? The answer: three axioms. Every inner product — whether the standard dot product or one with a custom matrix A — must satisfy them without exception.

Intuition: The "Laws of Physics" for Inner Products

Think of these three properties as the laws of physics for inner products. Just as any building must obey gravity, any inner product must obey:

Order doesn't matter (symmetry). The similarity from vector A to vector B must equal the similarity from B to A. Your Netflix similarity to a friend's profile should match their similarity to yours.
You can break things apart and reassemble (bilinearity). Stretching a vector or adding two vectors together before the inner product gives the same result as doing the inner product first and then stretching or adding. The gauge reads predictably.
Every real object has positive length (positive definiteness). A vector paired with itself must give a positive number — because that number is the squared length. Zero length means zero vector. There is no such thing as a negative-length stick.

Remove any one of these three, and you lose a fundamental geometric guarantee. This is why they are axioms — foundational, non-negotiable rules.

5.4.1 Symmetry

Symmetry: Order Does Not Matter

⟨ x, y ⟩ = ⟨ y, x ⟩

For any two vectors in the space, swapping their positions leaves the inner product unchanged. The measure of alignment from x to y is exactly the same as the alignment from y to x.

In real vector spaces (which this lecture uses), symmetry is exact equality: `⟨x, y⟩ = ⟨y, x⟩`
In complex spaces, symmetry becomes conjugate symmetry: `⟨x, y⟩ = ⟨y, x⟩̅` — the complex conjugate of the reversed pair. But we stay in the real domain here.
This property follows directly from the definition `⟨x, y⟩ = xᵀAy`. Since A is symmetric (`Aᵀ = A`), we have `xᵀAy = (xᵀAy)ᵀ = yᵀAᵀx = yᵀAx`.

Why it matters: The similarity from user A to user B must equal the similarity from user B to user A. A similarity measure that depended on ordering would be meaningless.

Compact check: for standard dot product with `x = [2, -3]`, `y = [5, 1]`:

`x·y = 2×5 + (-3)×1 = 10 − 3 = 7`
`y·x = 5×2 + 1×(-3) = 10 − 3 = 7` ✓

5.4.2 Bilinearity

Bilinearity: Linear in Both Arguments

Everything in linear algebra revolves around two operations: scalar multiplication and vector addition. Bilinearity says the inner product behaves predictably under both — and it does so in both slots. The prefix "bi" means the function is linear in the first argument and linear in the second argument.

(a) Scaling Property:

⟨ α x, y ⟩ = α ⟨ x, y ⟩

Think of the inner product as a gauge. If you stretch a vector by a factor α, the gauge reading stretches by exactly α. No randomness. No nonlinearity. Just proportional change. Scaling the vector first or scaling the result after — identical output.

(b) Distribution over Addition:

⟨ u + v, y ⟩ = ⟨ u, y ⟩ + ⟨ v, y ⟩

Add two vectors first, then take the inner product. Or take both inner products first, then add the results. Same answer either way. The inner product distributes cleanly across addition.

Worked Example: Scaling Property

Let `a = [2, 1]`, `b = [3, 4]`. Then `a·b = 6 + 4 = 10`.

Scale `a` by 5: `5a = [10, 5]`. Then `(5a)·b = 30 + 20 = 50 = 5 × 10`. ✓

Worked Example: Distribution over Addition

Let `u = [1, 1]`, `v = [2, 0]`, `y = [3, -4]`.

Addition first: `u + v = [3, 1]`, then `(u+v)·y = 9 − 4 = 5`.
Inner products first: `u·y = 3 − 4 = −1`, `v·y = 6 + 0 = 6`, sum = `−1 + 6 = 5`. ✓

The two paths converge to the same number. This is the distributive character that makes inner products so algebraically clean.

Together: Scaling and distribution combine into a single powerful statement. For any scalars α, β and vectors x, y, z:

⟨ α x + β y, z ⟩ = α ⟨ x, z ⟩ + β ⟨ y, z ⟩

The inner product is a bilinear form. And crucially, the linearity works in the second argument too — a point many students miss:

⟨ x, α y + β z ⟩ = α ⟨ x, y ⟩ + β ⟨ x, z ⟩

This follows from symmetry combined with linearity in the first argument. If `⟨x, αy + βz⟩ = ⟨αy + βz, x⟩ = α⟨y, x⟩ + β⟨z, x⟩ = α⟨x, y⟩ + β⟨x, z⟩`, the proof is done in two lines.

5.4.3 Positive Definiteness

Positive Definiteness: Lengths Must Be Real and Positive

⟨ x, x ⟩ > 0 for all x \neq = 0, and ⟨ x, x ⟩ = 0 ⟺ x = 0

A vector paired with itself produces its squared length (or squared norm). This number must be strictly positive for any real, non-zero vector. The only way to get zero is with the zero vector itself.

Why this matters:

The length of a vector is defined as `‖x‖ = √⟨x, x⟩`
Without positive definiteness, the value under the square root could be negative — and lengths would become imaginary numbers
An imaginary length is geometrically meaningless. You cannot have a stick whose measured length is `i` centimeters.
Positive definiteness is the axiom that keeps geometry real.

Positive Semi-Definite (Weaker Condition):

If `⟨x, x⟩ ≥ 0` (allowing zero for non-zero x), the inner product is only positive semi-definite. The distinction:

Positive definite: `xᵀAx > 0` for all `x ≠ 0` → lengths are always positive for non-zero vectors
Positive semi-definite: `xᵀAx ≥ 0` for all `x ≠ 0` → some non-zero vectors can have zero "length"

You will encounter positive semi-definite matrices often in machine learning. Covariance matrices, Gram matrices, and kernel matrices are all positive semi-definite. The distinction matters deeply in optimization: a positive definite Hessian guarantees a unique global minimum (strict convexity), while a positive semi-definite Hessian allows flat regions with multiple minima.

Geometric interpretation (from the professor): If A is positive definite, the transformation `Ax` never pushes a vector beyond the 180-degree mark from its original direction. The vector stays within the same half-plane. The dot product between the original vector and its transformed version remains positive — which is exactly what `xᵀAx = x·(Ax) > 0` encodes.

Scope: These Properties Define Real Inner Product Spaces

The three axioms — symmetry, bilinearity, and positive definiteness — are the complete definition of an inner product on any real vector space.

They apply to all vectors in the space, not just some convenient subset
They apply to the standard dot product (where `A = I`, the identity matrix) and to weighted inner products (where A is any symmetric positive definite matrix)
In complex spaces, symmetry is replaced by conjugate symmetry (`⟨x, y⟩ = ⟨y, x⟩̅`), but this lecture works exclusively in real vector spaces
The properties are axioms: they are assumed, not derived. Any operation satisfying all three defines a valid geometry on its vector space.

The entire machinery of lengths, distances, angles, orthogonality, and projections — everything covered in this lecture and the next — depends on these three properties holding.

Visual Intuition: The "Pick Any Two" Test

You can think of these three axioms as a checklist. Any operation that checks all three boxes is an inner product — and with it comes a full geometric toolkit (length, distance, angle, orthogonality). Remove one box and you lose a key geometric guarantee:

Drop this axiom	You lose
Symmetry	Similarity becomes directional; the notion of "angle" may not be well-defined
Bilinearity	You cannot decompose vectors or scale them predictably under the inner product
Positive definiteness	Lengths can be imaginary or zero for non-zero vectors; geometry breaks down

The standard dot product `x·y = Σxᵢyᵢ` passes all three. A weighted product `⟨x, y⟩ = xᵀAy` with symmetric positive definite A also passes all three. A pseudo-inner product (symmetric + bilinear but not positive definite) gives coherent algebra but broken geometry — lengths can be imaginary. Every inner product in machine learning kernels (RBF, polynomial, etc.) implicitly or explicitly satisfies these axioms.

Common Pitfalls

Forgetting the second argument. Bilinearity means the inner product is linear in both slots. Students often remember `⟨αx, y⟩ = α⟨x, y⟩` but forget `⟨x, αy⟩ = α⟨x, y⟩`. Both hold in real spaces (use symmetry to verify the second from the first).
Confusing positive definiteness with "all entries positive." A positive definite matrix A does NOT mean every entry of A is positive. The matrix `[[2, 1, 0], [1, 3, -1], [0, -1, 2]]` is positive definite yet contains zeros and a negative entry. The condition is on the quadratic form `xᵀAx`, not on individual matrix entries.
Symmetry + bilinearity without positive definiteness. This combination gives a "pseudo-inner product" or "indefinite inner product." You can still do algebra, but geometry fails: some non-zero vectors will have negative or zero squared length. Try `A = [[1, 0], [0, -1]]` — for `x = [0, 1]ᵀ`, the "squared length" is `−1`. That is not a real length.
Confusing `⟨x, x⟩` with `‖x‖`. The inner product of a vector with itself gives the squared length, not the length. The length is the square root: `‖x‖ = √⟨x, x⟩`. Forgetting the square root is one of the most common errors on exams (see Section 5.2.5).
Conflating bilinearity with distributivity alone. Bilinearity covers both scaling and addition. Verifying one without the other is not enough to claim a function is an inner product.

Recap: The Three Pillars — and What's Next

Every inner product — whether the familiar dot product or a custom weighted version using a matrix A — obeys exactly three axioms:

Symmetry: `⟨x, y⟩ = ⟨y, x⟩` — order doesn't change the answer
Bilinearity: `⟨αx + βy, z⟩ = α⟨x, z⟩ + β⟨y, z⟩` — scaling and addition behave predictably in both arguments
Positive definiteness: `⟨x, x⟩ > 0` for `x ≠ 0` — lengths are real, positive, and the zero vector is the only one with zero length

These three are what guarantee sensible geometry: you can measure lengths, compute distances, define angles, and test orthogonality. Without them, the entire structure collapses into contradictions like imaginary lengths or direction-dependent similarity.

Next: Where does all this math actually get used? Section 5.5 brings it down to earth — recommendation systems, text embeddings, image recognition, and the transformer attention mechanism that powers GPT.

Real-World & Domain Connection

These three axioms underpin all kernel methods in machine learning. Mercer's theorem states a profound result: any symmetric positive definite kernel function `K(x, y)` defines a valid inner product in some (possibly infinite-dimensional) feature space. This is the mathematical foundation of:

SVM kernels: RBF (radial basis function), polynomial, and sigmoid kernels all correspond to inner products in transformed feature spaces
Gaussian Processes: The covariance function must be a positive definite kernel — this is what guarantees the resulting covariance matrix is valid (positive semi-definite)
Kernel PCA: Instead of computing the inner product explicitly in a high-dimensional feature space, you compute the kernel function directly in the original space (the "kernel trick")
Transformer Attention: The attention mechanism at the heart of GPT and all modern transformer architectures computes dot products between query vectors (`Q`) and key vectors (`K`) as its core operation: `Attention(Q, K, V) = softmax(QKᵀ/√dₖ)V`. Each query-key pair undergoes an inner product to measure relevance — the same `⟨x, y⟩` operation defined in this section, scaled and run billions of times per forward pass. This connection makes the dot product arguably the most numerically expensive single operation in modern AI.

Whenever you choose a kernel in scikit-learn or any ML library, you are implicitly selecting an inner product that satisfies these three axioms. The library verifies none of them — it trusts you to pick a valid kernel. Understanding symmetry, bilinearity, and positive definiteness is what lets you distinguish a legitimate kernel from one that will produce NaN gradients or non-convergent training.

Symbol Registry for Section 5.4:

Symbol	Meaning	Type	Shape
`⟨x, y⟩`	Inner product of x and y	Scalar	Scalar
`α, β`	Scalars	Scalar	Scalar
`u, v, y`	Arbitrary vectors	Vector	`n × 1`
`‖x‖`	Norm (length) of x	Scalar	Scalar
`0`	Zero vector	Vector	`n × 1`

5.5 Applications in Machine Learning

The Hook. Everything you learned about dot products — component-wise multiplication, angles, cosine — powers the apps you use every day. Netflix doesn't guess what you'll like. It computes dot products. LinkedIn doesn't randomly suggest connections. It finds vectors with small angles from yours. The same math that gave you

5 \times 0 + 0 \times 5 + 5 \times 1 = 5

in the first worked example is running silently behind the recommendation engines, search results, and face recognition systems you interact with constantly.

Your Digital Self as a Vector. Think of your digital self as a vector in a high-dimensional space. Your coordinates are: how much you like action movies (0.7), how often you watch at night (0.3), your average rating strictness (-0.2), how many seconds you linger on thumbnails (0.5), and hundreds more features you probably never think about. Finding similar users means finding vectors with high dot products — people pointing in the same direction as you. If

cos θ

is close to 1, the angle is small, and the platform knows: "These two are alike."

5.5.1 Recommendation Systems and User Similarity

Users as Vectors, Similarity as Dot Product. Platforms like Netflix and LinkedIn represent users as vectors in an N-dimensional feature space — number of movies watched, watch time, genre preferences, interaction patterns, and so on. Computing the dot product between user vectors produces a similarity score. Users with high dot products (small angle between vectors) are considered similar.

For multiple users, you compute a dot product similarity matrix: for users $X_{1}, X_{2}, X_{3}, X_{4}$ , the entry $(i, j)$ is $X_{i} \cdot X_{j}$ . The diagonal entries ( $X_{i} \cdot X_{i}$ ) give squared lengths — generally not of primary interest. The off-diagonal entries are what matter: they tell you which users are similar to which.

Dot product is a binary operation. You can only take the dot product between two vectors at a time. For many users, you compute all pairwise dot products to build the full similarity matrix. This matrix then feeds into "People You May Know" on LinkedIn or "Because you watched..." on Netflix.

The professor made this concrete during the Q&A. When a student asked about Facebook's friend tagging in photos, the answer was direct: users are vectorized, and those with similar vectors get friend recommendations. Their vectorized representations have a high dot product. The same principle extends to content — if your vector is close to the vector of a movie you haven't watched, the platform recommends it.

Another student asked a sharp question: if there are multiple users, do you multiply two at a time and then somehow combine results? The answer: dot product is strictly binary. You compute every pair separately. The result is a square matrix where entry (i, j) tells you how similar user i is to user j. Correlation works the same way — it compares two variables, not many at once.

Concrete Example: Similarity Matrix for Four Users. Suppose four users are vectorized in a 3D feature space (movie genres: action, romance, comedy):

User 1: $[5, 1, 3]$
User 2: $[4, 2, 2]$
User 3: $[1, 5, 0]$
User 4: $[5, 0, 4]$

Compute pairwise dot products:

User 1 \cdot User 2 User 1 \cdot User 3 User 1 \cdot User 4 = (5 \times 4) + (1 \times 2) + (3 \times 2) = 20 + 2 + 6 = 28 = (5 \times 1) + (1 \times 5) + (3 \times 0) = 5 + 5 + 0 = 10 = (5 \times 5) + (1 \times 0) + (3 \times 4) = 25 + 0 + 12 = 37

The full similarity matrix:

\cdot 281037 28 \cdot 1828 1018 \cdot 5 37285 \cdot

Users 1 and 4 are most similar (dot product = 37). Users 1 and 3 are least similar among those compared (dot product = 10). The diagonal is omitted — it's just the squared length of each user's own vector.

5.5.2 Text Similarity and Word Embeddings

From Words to Vectors. Text embedding models — like those from OpenAI and Gemini — convert any text into a fixed-dimensional numeric vector. Each dimension is a floating-point number. You don't interpret any single dimension in isolation, but the collection of numbers captures meaning:

Small embedding model: ~300 dimensions
Large embedding model: ~768 dimensions

Once text becomes a vector, $v_{1} \cdot v_{2}$ measures semantic similarity. Two pieces of text that share concepts produce a high dot product.

Example. "I love Biryani" (3 words, 17 characters) gets embedded into a 300-dimensional vector. "Lucknow is known for Biryani" (5 words, 25 characters) gets embedded into another vector. Even though the sentences have different lengths and structures, their dot product is high because the shared concept — "Biryani" — dominates the vector representation. The model has learned that these sentences point in similar directions.

In transformer architectures — the foundation of GPT — dot products appear in the attention mechanism. The model computes dot products between query vectors and key vectors to decide which words should attend to which. This is the simple math of dot product combined with scaling. You will encounter key-query-value attention in more detail in your NLP class next semester.

The TensorFlow embedding projector is a freely available tool that visualizes high-dimensional word vectors projected into 2D or 3D space. Words with similar meanings — like "assassinated" and "murdered" — cluster close together because their vector representations have high dot products (small angles). You can rotate the visualization, zoom into clusters, and see the geometry of meaning emerge. It's worth exploring if you want an intuitive feel for what these embeddings actually do.

5.5.3 Image Recognition and Photo Clustering

Photos as Vectors. Google Photos extracts vectorized representations

V_{1}

and

V_{2}

from photos. Computing

V_{1} \cdot V_{2}

— or equivalently, the angle

θ

between them — determines whether two photos contain the same person or similar scenes. If the dot product is high (angle close to 0), the photos are grouped together.

You could build a simple photo clustering application yourself: take dot products between thousands of photo vectors, bucket similar ones together. Real production systems layer more sophisticated math on top — convolutional neural networks generate the vectors, then dot products and nearest-neighbor search cluster them. But the core operation, the one that decides "these two are the same person," is a dot product.

When a student asked during the lecture whether Google Photos matches the same concept, the professor confirmed: the same intuition applies. Extract vectors, compute angles, group by similarity. The idea is simple; the engineering underneath keeps it reliable at scale.

5.5.4 Everyday Physical Intuition

Why You Tilt Your Laptop Screen. When you open your laptop, you instinctively tilt the screen toward your eyes. You are optimizing a dot product. The light rays from the screen form a vector. Your eye line forms another vector. When the angle

θ

between them is 0° — when the screen faces you directly —

cos (0°) = 1

. The dot product is maximized:

∥ eye_vector ∥ ∥ light_vector ∥ cos (0°) = ∥ eye_vector ∥ ∥ light_vector ∥

You perceive maximum brightness. Tilt the screen away and $θ$ increases. $cos θ$ drops below 1. The dot product decreases. The screen looks dimmer. You didn't need a math class to figure this out — you learned it through experience. But the geometric form of the dot product is what you were intuitively optimizing every time you adjusted your screen.

Pitfalls of Raw Dot Product for Similarity.

Dot product alone can mislead. A user who gives extreme ratings (all 5s and 1s) will have a high-magnitude vector. Their raw dot product with everyone will be large — not because they're similar to everyone, but because their vector is long. Real systems use cosine similarity, the dot product divided by both magnitudes: $\frac{x \cdot y}{∥ x ∥∥ y ∥}$ . This normalizes out the lengths and isolates the angle.
Embedding dimensions are not individually interpretable. You cannot look at dimension 3 of an OpenAI embedding and say "this means sentiment." The dimensions work together as a whole. A high dot product means similarity; you cannot point to one dimension and explain why.
Dot product is not transitive. If A is similar to B and B is similar to C, A is not necessarily similar to C. Each pairwise comparison is independent.

Q: Facebook had automatic friend tagging in photos a few years back. Is that a real application of the dot product? A: Yes, absolutely. Facebook and LinkedIn represent every user as a vector. Users with similar vectors — meaning a high dot product between their representations — get friend recommendations. The system vectorizes you: your profile data, your activity, your connections. Then it searches for other vectors pointing in similar directions. The same principle applies to content tagging in photos — the face in the photo is vectorized and compared against your known face vector.

Q: Does Google Photos use the same dot product concept for face grouping? A: Yes, the same fundamental idea applies. Google Photos extracts vector representations from every photo. Computing the dot product — or equivalently, the angle between two photo vectors — determines if they contain the same person. If the dot product is high (angle close to 0), the photos likely show the same person and are grouped together. Production systems layer much more sophisticated math on top — convolutional neural networks generate the vectors, clustering algorithms handle scale — but the core comparison operation is a dot product.

Recap + Bridge. Dot products power recommendation engines, search results, face grouping in photos, and even how you tilt your laptop screen. The math is simple: multiply components and sum, or multiply lengths and cosine. The applications are everywhere. Recommendation systems vectorize you. Embedding models vectorize text. Image models vectorize photos. Once everything is a vector, dot products let you compare anything with anything. Next: what else can we build from inner products? Distance, projection, orthogonalization, and dimensionality reduction all emerge from this same foundation — the inner product.

5.6 Derived Concepts from Inner Products

5.6.1 Distance Between Vectors

To find how far apart two vectors are, subtract one from the other. That gives you a difference vector. Then find its length.

Formula:

d (x, y) = ∥ x - y ∥

For $x, y \in R^{n}$ :

d (x, y) = (x - y)^{T} (x - y) = i = 1 \sum n (x_{i} - y_{i})^{2}

This is a direct consequence of the dot product. Subtract first, then take the length. Do not take the lengths separately and then subtract them. Those two operations give different results.

Worked Example: Distance vs. Length Difference

Let $x = [4, 0]^{T}$ and $y = [0, 3]^{T}$ .

Correct distance (subtract first, then take length):

x - y = [4, - 3]^{T}, d (x, y) = 4^{2} + (- 3)^{2} = 16 + 9 = 5

Wrong approach (take lengths, then subtract):

∥ x ∥ = 4^{2} + 0^{2} = 4, ∥ y ∥ = 0^{2} + 3^{2} = 3, ∥ x ∥ - ∥ y ∥ = 1 \neq = 5

The two operations are not interchangeable. The distance formula uses the difference vector, not the difference of lengths.

Pitfall: Lengths Don't Commute with Subtraction

$∥ x - y ∥ \neq = ∥ x ∥ - ∥ y ∥$ in general. The triangle inequality guarantees:

∥ x ∥ - ∥ y ∥ \leq ∥ x - y ∥ \leq ∥ x ∥ + ∥ y ∥

Always compute the difference vector first, then measure its length.

5.6.2 Vector Projection Intuition

Taking a dot product is almost like projecting the first vector onto the second vector. The dot product tells you how much of $a$ is aligned in the direction of $b$ . This geometric interpretation underpins the Gram-Schmidt process.

If $a$ and $b$ are vectors:

Scalar projection of $a$ onto $b$ : $\frac{a \cdot b}{∥ b ∥}$
Vector projection of $a$ onto $b$ : $proj_{b} (a) = \frac{a \cdot b}{b \cdot b} b$

Projection = Shadow

Think of shining a light perpendicular to $b$ . The shadow cast by $a$ onto the line of $b$ is the projection. The length of that shadow is the scalar projection. The shadow itself — as a vector along $b$ — is the vector projection.

The part of $a$ that is perpendicular to $b$ is exactly $a - proj_{b} (a)$ . This remainder is orthogonal to $b$ . Gram-Schmidt exploits this: subtract the shadow to get perpendicularity.

5.6.3 Gram-Schmidt Orthogonalization

Purpose: Why Gram-Schmidt Exists

You are given a set of linearly independent vectors — say columns of a matrix or features in a dataset. They are not perpendicular. You need an orthogonal basis for the same subspace: vectors that are mutually perpendicular. Gram-Schmidt solves this problem. It turns any linearly independent set into an orthogonal set. If you then normalize each vector, the result is an orthonormal set — perpendicular vectors of unit length.

Inputs & Outputs

Input: A set of $n$ linearly independent vectors ${v_{1}, v_{2}, \dots, v_{n}}$ in an inner product space.
Output (orthogonal): A set ${u_{1}, u_{2}, \dots, u_{n}}$ where $⟨ u_{i}, u_{j} ⟩ = 0$ for all $i \neq = j$ .
Output (orthonormal): A set ${e_{1}, e_{2}, \dots, e_{n}}$ where $⟨ e_{i}, e_{j} ⟩ = 0$ for $i \neq = j$ and $∥ e_{i} ∥ = 1$ for all $i$ .

Algorithm Steps

Set the first vector unchanged: $u_{1} = v_{1}$ .
Rationale: The first vector defines the initial direction. There is nothing to subtract from it yet.
For each subsequent vector $k = 2, 3, \dots, n$ : subtract all projections onto previously computed $u_{i}$ : $u_{k} = v_{k} - i = 1 \sum k - 1 \frac{⟨ v _{k} , u _{i} ⟩}{⟨ u _{i} , u _{i} ⟩} u_{i}$ Rationale: The fraction $\frac{⟨ v _{k} , u _{i} ⟩}{⟨ u _{i} , u _{i} ⟩}$ is the scalar that scales $u_{i}$ to equal the projection of $v_{k}$ onto $u_{i}$ . Subtracting this projection removes the component of $v_{k}$ parallel to $u_{i}$ . What remains is perpendicular to all previous $u$ 's.
To get orthonormal vectors: normalize each $u_{i}$ : $e_{i} = \frac{u _{i}}{∥ u _{i} ∥}, i = 1, \dots, n$

Full Trace: Gram-Schmidt on Two Vectors

Given: $v_{1} = [1, 1, 0]^{T}$ and $v_{2} = [1, 2, 2]^{T}$ . Use the standard dot product.

Step 1: $u_{1} = v_{1} = [1, 1, 0]^{T}$ .

Step 2: Compute the projection of $v_{2}$ onto $u_{1}$ :

⟨ v_{2}, u_{1} ⟩ = (1) (1) + (2) (1) + (2) (0) = 3

⟨ u_{1}, u_{1} ⟩ = 1^{2} + 1^{2} + 0^{2} = 2

proj_{u_{1}} (v_{2}) = \frac{3}{2} u_{1} = [\frac{3}{2}, \frac{3}{2}, 0]^{T} = [1.5, 1.5, 0]^{T}

Step 3: Subtract the projection:

u_{2} = v_{2} - proj_{u_{1}} (v_{2}) = [1, 2, 2]^{T} - [1.5, 1.5, 0]^{T} = [- 0.5, 0.5, 2]^{T}

Verification: $u_{1} \cdot u_{2} = (1) (- 0.5) + (1) (0.5) + (0) (2) = - 0.5 + 0.5 + 0 = 0 ✓$ — they are orthogonal.

Step 4 (Normalization):

∥ u_{1} ∥ = 1^{2} + 1^{2} + 0^{2} = 2, e_{1} = \frac{u _{1}}{2} = [\frac{1}{2}, \frac{1}{2}, 0]^{T}

∥ u_{2} ∥ = (- 0.5)^{2} + (0.5)^{2} + (2)^{2} = 0.25 + 0.25 + 4 = 4.5 \approx 2.121

e_{2} = \frac{u _{2}}{4.5} = [\frac{- 0.5}{2.121}, \frac{0.5}{2.121}, \frac{2}{2.121}]^{T}

Result: ${e_{1}, e_{2}}$ is an orthonormal basis for $span {v_{1}, v_{2}}$ .

Geometric Intuition: We kept $v_{1}$ as is. Then we measured how much of $v_{2}$ lies along $v_{1}$ — that is the shadow. We subtracted that shadow. What remained had no overlap with $v_{1}$ — it became perpendicular. Then we shrank both vectors to unit length. Gram-Schmidt = subtracting shadows to make perpendicular.

Complexity & Cost

Gram-Schmidt runs in $O (n^{3})$ for $n$ vectors in $R^{n}$ . Each step computes $k - 1$ inner products and vector subtractions. For large datasets, the classical Gram-Schmidt suffers from numerical instability due to floating-point rounding errors: the computed $u_{k}$ may not be exactly orthogonal to earlier vectors. In practice, the modified Gram-Schmidt algorithm or Householder reflections (via QR decomposition) are preferred for numerical stability.

When to Use / Alternatives

Scenario	Recommendation
Small to medium matrices, exact orthogonal basis needed	Classical Gram-Schmidt (this section)
Large matrices, numerical stability matters	Modified Gram-Schmidt or Householder QR
Very large sparse matrices, only a few eigenvalues needed	Iterative methods: Arnoldi, Lanczos
You only need orthonormal columns of a matrix	Use `np.linalg.qr` in practice — it uses Householder reflections under the hood

Key exam point: Know the Gram-Schmidt sequence — project, subtract, normalize. Know the difference between orthogonal (perpendicular) and orthonormal (perpendicular + unit length).

5.6.4 Orthogonal and Orthonormal Vectors

Definitions:

Orthogonal: a \cdot b = 0 (perpendicular, any length) Orthonormal: e_{i} \cdot e_{j} = {10 if i = j if i \neq = j (perpendicular AND unit length)

If you rotate one vector until it becomes perpendicular to another, they become orthogonal. Normalize both to unit length, and they become orthonormal.

Worked Example (Normalization):

For $x = [3, 4]^{T}$ :

∥ x ∥ = 3^{2} + 4^{2} = 5

x^= \frac{x}{∥ x ∥} = [\frac{3}{5}, \frac{4}{5}]^{T}

Every entry is divided by the length. The resulting vector has unit length. Its direction is unchanged — only its magnitude is scaled to 1.

Normalization = Divide by Length

To normalize any non-zero vector $x$ :

x^= \frac{x}{∥ x ∥}

The hat notation $x^$ denotes a unit vector. The operation preserves direction; it only scales the length to exactly 1. This is the most common normalization — sometimes called Euclidean normalization or L² normalization.

Orthogonal vs. Orthonormal: The Distinction

Property	Orthogonal	Orthonormal
Condition	$a \cdot b = 0$	$e_{i} \cdot e_{j} = 0$ for $i \neq = j$
Length constraint	Any length	$∥ e_{i} ∥ = 1$ for all $i$
Example	$[3, 0]^{T}$ and $[0, 5]^{T}$	$[1, 0]^{T}$ and $[0, 1]^{T}$
Use as basis?	Yes, but coordinates require scaling factors	Yes — coordinates are just dot products: $z_{i} = x \cdot e_{i}$

Orthonormal bases simplify everything. Any vector can be expressed as a linear combination with coefficients equal to dot products with the basis vectors. This is why PCA and many ML algorithms prefer orthonormal bases.

Q: What does "normalize" mean exactly? A: Take a vector like

[3, 4]

. Its length is

3^{2} + 4^{2} = 5

. To normalize it, divide every component by the length:

[3/5, 4/5]

. The resulting vector has unit length — exactly 1. You can verify:

(3/5)^{2} + (4/5)^{2} = 9/25 + 16/25 = 25/25 = 1

. Normalization means dividing a vector by its own length so it becomes a unit vector. This is the most common normalization process for vectors, though there are other types of normalization for different purposes.

Q: Why do we normalize vectors? What is the significance? A: Normalization is what turns orthogonal vectors into orthonormal vectors. Remember: orthogonal = perpendicular; orthonormal = perpendicular AND unit length. The standard basis vectors

[1, 0]

and

[0, 1]

are orthonormal — they are perpendicular and each has length 1. Orthonormal vectors can serve as a new basis for the vector space. You can combine them using linear combinations to reach any point. This is exactly what PCA does: it finds new orthonormal axes (the principal components) along which the data has maximum variance. You project your data onto these axes for dimensionality reduction. The orthonormal basis vectors become your new coordinate system — cleaner, more informative, and often lower-dimensional.

5.6.5 Connection to PCA (Preview)

Principal Component Analysis (PCA) uses orthonormal vectors as a new basis for dimensionality reduction. High-dimensional data — credit scores, medical parameters, image pixels — often contains redundancy. Many dimensions are correlated or irrelevant.

What PCA Does

PCA finds eigenvectors of the data covariance matrix. These eigenvectors are sorted by their associated eigenvalues, which measure the variance captured along each direction. The eigenvectors with the largest eigenvalues become the principal components. They form an orthonormal basis for a lower-dimensional subspace.

Key idea: Project the original high-dimensional data onto only the top $M$ principal components. The projection coordinates are simply $z_{i} = b_{i}^{T} x$ — the dot product with each principal component. This works because the principal components are orthonormal.

Why this matters in practice:

You have a dataset with 1000 features (columns). You compute the covariance matrix. You find its eigenvectors. The top 100 eigenvectors might capture 99% of the total variance — the data barely varies along the other 900 directions. You project your data onto those 100 principal components. Now you have 100 new features instead of 1000. You train your model on this reduced representation. Benefits:

Less computation — fewer features means faster training and inference
Less storage — the transformed data matrix is smaller
Noise reduction — directions with negligible variance are often just noise
Better generalization — the model focuses on the signal, not the noise

Geometric intuition (2D analogue): You are predicting salary using age and experience. The data points are scattered, but they mostly lie along a diagonal line. Rotate your axes so that the first axis aligns with that diagonal — this is the first principal component. Project every data point onto that single axis. Now you have a 1D representation instead of 2D, capturing the dominant trend. PCA does exactly this in higher dimensions.

PCA Recap from MML (T1, Ch10)

PCA maximizes the variance of the projected data (maximum variance formulation), which is equivalent to minimizing the reconstruction error (projection formulation).
The projection of data point $x$ onto the principal subspace is: $x ~ = B B^{T} x$ , where $B = [b_{1}, \dots, b_{M}]$ contains the $M$ leading eigenvectors.
The coordinates in the reduced space are: $z = B^{T} x$ .
The variance captured by $M$ principal components equals the sum of the $M$ largest eigenvalues: $V_{M} = \sum_{m = 1}^{M} λ_{m}$ .
The lost variance (reconstruction error) is: $J_{M} = \sum_{j = M + 1}^{D} λ_{j}$ .

In practice, choose $M$ such that $\frac{V _{M}}{V _{D}} \geq 0.95$ or $0.99$ — retaining 95% or 99% of total variance.

CIBIL Example (from lecture): Building a credit score prediction model with age, demographics, blood parameters, and hundreds of other features. PCA would find that many of these features are correlated. Instead of 1000 original parameters, you might retain 100 principal components capturing 99% of variance. Those 100 directions are your new basis. Your model trains on this compressed representation.

The same principles of dot products and vector embeddings power modern deep learning at scale. OpenAI's text embedding models — `text-embedding-3-small` produces 512-dimensional vectors and `text-embedding-3-large` produces 3072-dimensional vectors — represent text as high-dimensional vectors where semantically similar sentences have high dot-product similarity. Google's Gemini embedding models similarly map text to dense vector representations, enabling semantic search, clustering, and retrieval. In the Transformer architecture (the foundation of GPT models), the scaled dot-product attention mechanism computes dot products between queries ( $Q$ ) and keys ( $K$ ) to determine how strongly each word attends to every other word: $Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$ . The division by $d_{k}$ (scaling) prevents the dot products from growing too large in high dimensions — a practical normalization that echoes the core ideas of this section.

Q: Out of those thousand parameters, you reduce to 100 — are those 100 the new basis or the new parameters? A: Exactly. Instead of using 1000 original features, PCA finds 100 directions (principal components) that capture the majority of the variance. These 100 become your new basis — a new coordinate system. Every data point is re-expressed as 100 coordinates (one per principal component) instead of 1000. These 100 coordinates are your new parameters for training the model. PCA consolidates all the original information into a compressed representation.

How Gram-Schmidt, Orthonormal Bases, and PCA Connect

Gram-Schmidt produces an orthonormal basis from any linearly independent set.
An orthonormal basis simplifies all computations: inner products become standard dot products, coordinates are just dot products with basis vectors.
PCA finds an orthonormal basis (the eigenvectors of the covariance matrix) where the first few basis vectors capture the most variance.
This PCA basis is an orthonormal basis for a subspace — the principal subspace where the data mostly lives.
Projecting onto this subspace reduces dimensionality while preserving information.

The entire chain: inner product → orthogonality → Gram-Schmidt → orthonormal basis → PCA projection.

Q: Several students asked for a summary of today's topics. Here is the full chain. A: We started with dot products —

x^{T} y

— used for measuring similarity between vectors. Then we generalized to inner products —

x^{T} A y

— where

A

is a symmetric positive definite matrix that lets you weight features differently. We covered the geometric definition through lengths and angles (

x \cdot y = ∥ x ∥∥ y ∥ cos θ

), vector length via self-dot-product (

∥ x ∥ = x^{T} x

), and distance between vectors (

d (x, y) = ∥ x - y ∥

). We studied the three axioms of inner products — symmetry, bilinearity, and positive definiteness. Then we explored Gram-Schmidt for orthogonalization and closed with the definition and computation of matrix rank.

Symbol Registry for Section 5.6:

Symbol	Meaning	Type	Shape
$d (x, y)$	Euclidean distance	Scalar	Scalar
$x - y$	Difference vector	Vector	$n \times 1$
$proj_{b} (a)$	Vector projection of $a$ onto $b$	Vector	$n \times 1$
$u_{i}$	i-th orthogonal vector from Gram-Schmidt	Vector	$n \times 1$
$e_{i}$	i-th orthonormal vector	Vector	$n \times 1$
$x^$	Normalized (unit) vector	Vector	$n \times 1$
$b_{i}$	i-th principal component (eigenvector of S)	Vector	$D \times 1$
$λ_{i}$	i-th eigenvalue of covariance matrix	Scalar	Scalar
$B$	Matrix of M principal components	Matrix	$D \times M$
$z$	Low-dimensional code (PCA coordinates)	Vector	$M \times 1$

5.7 Matrix Rank

A matrix can squash a 3D world into a 2D plane, or even collapse it to a single line. The rank tells you how many dimensions survive the transformation. How do you compute it?

Intuition: The Paper-Shredder Analogy. Think of a matrix as a paper-shredder with different settings. Rank = full means a 3D box stays 3D (just rotated). Rank = 2 means the box gets flattened into a sheet of paper. Rank = 1 means it becomes a toothpick. Rank = 0 means it is annihilated to nothing. The rank measures how many dimensions your shredder preserves.

5.7.1 Definition via Linear Independence

For a given matrix $A$ , rank tells you something specific. How many linearly independent vectors do you have? Either row vectors or column vectors. The row rank always equals the column rank — this is one of the most important theorems in linear algebra.

Definition:

rank (A)

is the number of linearly independent rows of

A

, which always equals the number of linearly independent columns of

A

rank (A) = dim (col (A)) = dim (row (A))

Row rank equals column rank (theorem). This is not obvious — for a random 10×12 matrix, nobody would guess that independent rows and independent columns come in the same number. Yet they always do.
Rank cannot exceed $min (rows, cols)$ .
Full rank: $rank (A) = min (m, n)$ . The matrix uses all its available dimensions.
Rank deficient: $rank (A) < min (m, n)$ . Some columns (or rows) are redundant — they are linear combinations of others.
For a square $n \times n$ matrix: full rank means rank = $n$ , which implies $A$ is invertible (nonsingular). Rank deficient means $det (A) = 0$ and $A$ has a nontrivial nullspace.

5.7.2 Worked Example: Detecting Linear Dependence

Consider a marks matrix for three users (A, B, C) across three subjects (M₁, M₂, M₃):

203040305030508070

Column vectors: $c_{1} = [20, 30, 40]^{T}$ , $c_{2} = [30, 50, 30]^{T}$ , $c_{3} = [50, 80, 70]^{T}$ .

Step 1 — Check for dependence. Add the first two columns:

c_{1} + c_{2} = 203040 + 305030 = 508070 = c_{3}

The third column is exactly the sum of the first two. That means $c_{3}$ is linearly dependent on $c_{1}$ and $c_{2}$ . It contributes no new direction — no new information.

Step 2 — Count independent columns. Only $c_{1}$ and $c_{2}$ are independent. So:

rank (A) = 2

Step 3 — Verify row rank. From the column perspective, it is evident the third is dependent on the first two. From the row perspective, you would need to do the echelon form reduction. But you will eventually find the row rank is also 2. They always match. They always match.

Even though this is a $3 \times 3$ matrix, its rank is only 2. Two subjects would have been enough to capture all the information; M₃ was redundant.

5.7.3 Computing Rank via Reduced Echelon Form

When dependence is not obvious by inspection, use systematic row reduction.

Worked Example: Start with a

3 \times 3

matrix. Apply elementary row operations (swap rows, multiply a row by a non-zero scalar, add a multiple of one row to another) to reach reduced row echelon form (RREF). Suppose you get:

100010 * * 0

The last row is all zeros. The pivot elements (leading 1s) appear only in the first and second rows. The columns containing those pivots — columns 1 and 2 — are the linearly independent columns.

Count: two non-zero rows $⟹ rank (A) = 2$ .

Equivalently: two pivot columns $⟹ rank (A) = 2$ . Both methods give the same answer.

Rules for computing rank:

Reduce the matrix to row echelon form (REF) or reduced row echelon form (RREF) using elementary row operations. Row operations do not change the rank.
Count the number of non-zero rows (rows with at least one non-zero entry).
Equivalently, count the number of pivot columns. If a row has a pivot, that row is non-zero.
Rank cannot exceed $min (number of rows, number of columns)$ .
The pivot columns of the original matrix $A$ (not just the echelon form) form a basis for the column space.

Scope: Rank is always an integer. Full-rank square matrices are invertible — their determinant is non-zero. Rank-deficient matrices have non-trivial nullspaces:

dim (N (A)) = n - rank (A)

. This is the Rank-Nullity Theorem. Over real numbers. The dimension of the column space and the dimension of the row space both equal the rank. This is a consequence of the Fundamental Theorem of Linear Algebra (Part I):

$dim (C (A)) = r$ (column space)
$dim (N (A)) = n - r$ (nullspace)
$dim (C (A^{T})) = r$ (row space)
$dim (N (A^{T})) = m - r$ (left nullspace)

5.7.4 Geometric Intuition: Rank as Dimension of Output Space

Rank is the dimension of the output vector after a matrix transformation. Multiply a matrix $A$ with a vector $x$ to get $y = A x$ . The number of dimensions $y$ carries — that is the rank.

Visualize the transformation for a 2D input (a plane):

Rank 2: Output is still a 2D plane — maybe rotated or stretched, but the grid lines stay distinct. Every output point comes from exactly one input.
Rank 1: Output collapses to a line. All points in the input plane squish onto a single direction. You lose one dimension.
Rank 0: Everything maps to the origin. The matrix is the zero matrix.

For a 3D input:

Rank 3: Volume is preserved (just transformed). The cube stays a 3D parallelepiped.
Rank 2: 3D volume is squished to a 2D plane — a flat sheet. The cube becomes a parallelogram.
Rank 1: Squished to a 1D line — a toothpick.
Rank 0: Annihilated to a single point at the origin.

Think of it this way: dimension is how many coordinates the input vector carries. Rank is how many dimensions survive the matrix transformation and remain in the output. A matrix can squash a 3D world into a 2D plane or a 1D line. Rank tells you how many dimensions remain.

When a matrix is a rotation matrix, the output keeps the same number of dimensions as the input. The rank equals the input dimension. Things get tricky only in one case: a higher-dimensional vector gets squished to a lower dimension — a 3D vector pushed to a 2D plane, or a 2D vector transformed to a 1D line.

Pitfalls:

Confusing rank with dimension of input. Dimension is what goes in; rank is what comes out. A 3×3 matrix with rank 2 takes 3-dimensional inputs but produces only 2-dimensional outputs — every output lies on a plane.
Thinking a 3×3 matrix with 3 rows always has rank 3. Dependencies can reduce it. Three rows can collapse to two independent rows (or even one).
Counting columns instead of independent columns. A matrix with 5 columns can easily have rank 2 — four of those columns are redundant.
Forgetting that row rank = column rank. You can check either perspective. If columns are easier to inspect, check columns. If rows are easier, check rows. The answer is the same.
Assuming pivot columns and non-zero rows might disagree. They never do. Each pivot gives exactly one non-zero row. Both counts equal the rank.

Recap + Bridge: Rank measures how many dimensions survive a matrix transformation. Compute it via echelon form — count non-zero rows or pivot columns. The row rank always equals the column rank. Next, we apply this to understand when systems have unique solutions (full rank) versus infinite solutions or no solutions (rank deficient).

Real-World & Domain Connection: Rank reveals redundancy in datasets. In PCA (Principal Component Analysis), the rank of the data matrix tells you the intrinsic dimensionality — how many features actually carry independent information. In recommender systems (Netflix, Amazon), low-rank matrix factorization via SVD exploits the fact that user-item matrices are often rank-deficient: the underlying structure is simpler than the data suggests. A matrix of millions of users and items might have an effective rank of only 50 or 100 — meaning all that preference data can be compressed into a small number of latent factors.

Q: What is the difference between rank and dimension? A: Dimension is how many coordinates the input vector carries. Rank is how many dimensions survive after the matrix transformation. If a 3D vector goes in and a 2D vector comes out (because A squishes one dimension away), the rank is 2. Q: Should I count pivot columns or non-zero rows? A: Both give the same answer. Each pivot corresponds to a non-zero row. Count whichever is easier in the echelon form you have. Q: How do row rank and column rank compare for the marks matrix example? A: From columns,

c_{3} = c_{1} + c_{2}

makes rank = 2 obvious. Row operations would also give rank = 2 — they always match. The row rank equals the column rank is a theorem; you can verify it by computing rank both ways on any matrix.

Exam note: For the quiz: reduce to echelon form and count non-zero rows (or pivot columns). Row rank always equals column rank. Full-rank square matrices are invertible. The dimension of the nullspace =

n - rank (A)

Symbol Registry for Section 5.7:

Symbol	Meaning	Type	Shape
$rank (A)$	Rank of matrix $A$	Integer	Scalar
$col (A)$	Column space of $A$	Subspace	—
$row (A)$	Row space of $A$	Subspace	—
$c_{i}$	$i$ -th column vector	Vector	$m \times 1$
$C (A)$	Column space of $A$ (alternate notation)	Subspace	—
$N (A)$	Nullspace of $A$	Subspace	—
$r$	Rank (common shorthand)	Integer	Scalar

5.8 Student Q&A Log

Session Summary: This lecture introduced the dot product as a similarity measure between vectors, then generalized it to inner products via symmetric positive definite matrices. Key properties — symmetry, bilinearity, and positive definiteness — were established, along with the geometric interpretation via $cos θ$ . Derived concepts included vector length, distance, normalization, and the Gram-Schmidt process. Applications spanned recommendation systems, text embeddings, transformer attention, and image clustering. The session closed with matrix rank — defined via linear independence, computed via echelon form, and interpreted geometrically as the dimension of the output space after transformation. PCA was previewed as a dimensionality reduction technique built on orthonormal bases.

5.8.1 Dot Product Computation and Notation

Q: (About the

[2, - 3]

and

[5, 1]

class example) The dot product comes out to 7, which is positive, but graphing the vectors makes them look like they point in different directions. Shouldn't it be

[- 2, 3]

yielding

- 7

? A: Yes — the vector should be

[- 2, 3]

, not

[2, - 3]

. With

[- 2, 3] \cdot [5, 1] = (- 2) (5) + (3) (1) = - 10 + 3 = - 7

, the sign correctly reflects the geometric relationship. The takeaway is that a positive dot product means the angle is less than

9 0^{\circ}

(

cos θ > 0

); a negative dot product means the angle exceeds

9 0^{\circ}

(

cos θ < 0

). Always sketch vectors to sanity-check your computed sign. Notation and sign interpretation are also covered in section 5.2.

Q: In the matrix form

x^{T} y

, why is one vector a row and the other a column? Does order matter? A: The transpose is a mechanical requirement for matrix multiplication. To multiply two matrices, the number of columns of the first must equal the number of rows of the second.

x^{T}

1 \times n

(a row vector) and

y

n \times 1

(a column vector) — their product is a

1 \times 1

scalar. If you tried to multiply two column vectors directly (

n \times 1

times

n \times 1

), the dimensions would not match. Order does not matter for the result:

x^{T} y = y^{T} x

(symmetry of the dot product), but

x^{T} y

is the standard convention for writing it as matrix multiplication. See section 5.2.3 for the full derivation.

5.8.2 Applications and Real-World Usage

Q: Where are dot products actually used in the real world? How do Facebook friend tagging, Google Photos face grouping, and recommendation systems all connect? A: All of these are instances of the same core idea: represent entities as vectors, then use the dot product to measure similarity.

Recommendation systems (Netflix, LinkedIn, Facebook): Each user is vectorized across an $n$ -dimensional feature space (movies watched, watch time, interests, demographics). $u_{A} \cdot u_{B}$ quantifies how similar two users are. High dot product → small angle → similar preferences → friend or content recommendations ("People You May Know").
Photo clustering (Google Photos, Facebook facial recognition): Each photo is converted into a vector via a neural network (embedding). $v_{photo1} \cdot v_{photo2}$ measures visual similarity. If the dot product exceeds a threshold, the photos are grouped — same person, same scene, same event. Real systems layer sophisticated models on top, but the dot product is the fundamental comparison operation.
Text similarity and transformers: Text embedding models (OpenAI, Gemini) convert sentences into 300- or 768-dimensional vectors. Two semantically similar texts have a high dot product. In GPT-style transformers, the attention mechanism computes dot products between query and key vectors to decide which tokens to attend to — this is the engine of modern NLP.
Everyday physics: Bending your laptop screen toward your eyes aligns the light rays with your line of sight (angle ≈ 0°), maximizing the dot product between the light-source vector and your viewing-direction vector for maximum perceived brightness.

The application details are expanded in section 5.5.

Q: If I have more than two vectors — say a thousand users — how do I compute similarity across all of them? Do I multiply two at a time and somehow combine? A: The dot product is a binary operation — it takes exactly two vectors and returns one scalar. It does not generalize to three or more vectors at once. For

n

users, you build an

n \times n

similarity matrix where entry

(i, j)

x_{i} \cdot x_{j}

S = x_{1} \cdot x_{1} x_{2} \cdot x_{1} ⋮ x_{n} \cdot x_{1} x_{1} \cdot x_{2} x_{2} \cdot x_{2} ⋮ x_{n} \cdot x_{2} \dots \dots ⋱ \dots x_{1} \cdot x_{n} x_{2} \cdot x_{n} ⋮ x_{n} \cdot x_{n}

Each entry comes from a pairwise dot product between two vectors.
The diagonal entries ( $x_{i} \cdot x_{i}$ ) give the squared length $∥ x_{i} ∥^{2}$ — typically not used for similarity scoring.
The off-diagonal entries are what you care about: they rank similarity between distinct users.
This is analogous to a correlation matrix in statistics — correlation is also a binary operation; you compute it pairwise for every pair of variables.

5.8.3 Inner Products and Positive Definite Matrices

Q: We already have the standard dot product

x^{T} y

for measuring similarity. Why do we need the inner product

x^{T} A y

? Where would we actually use it? A: The inner product is a weighted dot product. In the standard dot product, every dimension contributes equally:

x_{1} y_{1} + x_{2} y_{2} + \dots + x_{n} y_{n}

. But in real problems, features have different importance.

Consider predicting house price from two features: number of bedrooms and number of taps. Bedrooms matter far more than taps. With

A = [4001]

the inner product $x^{T} A y$ weights the bedroom dimension four times more heavily than the tap dimension. When $A = I$ (the identity matrix), all weights are 1 — that is just the standard dot product. When $A \neq = I$ (but still symmetric positive definite), you are applying feature-specific importance.

To emphasize a different feature, place the larger weight in that position. To emphasize the second dimension, use $A = [1004]$ . The diagonal entries control per-dimension importance; off-diagonal entries (when present) encode interactions between dimensions.

Beyond feature weighting, positive definite matrices have a critical theoretical role: if the Hessian (second-derivative matrix) of a loss function is positive definite, the loss function is convex — it has exactly one global minimum. Regardless of where you initialize an optimization algorithm, you are guaranteed to converge to that unique minimum. Without positive definiteness, you risk getting trapped in a local minimum.

See section 5.3 for the full definition and worked example.

Q: You said the dot product becomes negative when the angle exceeds 90° because

cos θ

goes negative. But you also said that for a positive definite matrix, the transformed vector never crosses the 180° line from the original. Which angle threshold matters? A: Both thresholds have different meanings:

90° ( $π /2$ ): The boundary between positive and negative dot product. $cos θ > 0$ for $θ < 9 0^{\circ}$ (positive dot product), $cos θ < 0$ for $θ > 9 0^{\circ}$ (negative dot product), and $cos 9 0^{\circ} = 0$ (orthogonal — dot product is exactly zero).
180° ( $π$ ): The boundary for a positive definite transformation. When a positive definite matrix $A$ multiplies a vector $x$ to produce $A x$ , the displacement from $x$ to $A x$ stays within the half-plane defined by $x$ . The transformed vector may land on either side of the 90° line (so $x \cdot A x$ could be negative in some cases with off-diagonal terms), but it never ends up pointing into the opposite half-plane — the angle between $x$ and $A x$ never reaches or exceeds 180°. This geometric constraint is a consequence of $x^{T} A x > 0$ for all non-zero $x$ .

In short: dot product sign changes at 90°; the positive definite "safe zone" extends all the way to 180°.

See section 5.3.3 for the geometric intuition of positive definite matrices.

5.8.4 Geometric Intuition and Normalization

Q: What does it mean to normalize a vector? How do you compute it, and why is it important in machine learning? A: Normalization means scaling a vector so its length becomes exactly 1 (unit length), without changing its direction.

Computation (for $x = [3, 4]$ ):

∥ x ∥ = 3^{2} + 4^{2} = 5 x^= \frac{x}{∥ x ∥} = [\frac{3}{5}, \frac{4}{5}]

Divide every component by the length. The resulting vector $x^$ has $∥ x^∥ = 1$ . Every point now lies on the unit circle (in 2D) or unit sphere (in higher dimensions).

Why it matters:

Orthonormal bases: Gram-Schmidt produces orthogonal (perpendicular) vectors. Normalizing them yields orthonormal vectors — perpendicular and unit-length. Like the standard basis ${[1, 0], [0, 1]}$ , orthonormal vectors can serve as a new coordinate system. Any vector can be expressed as a linear combination of orthonormal basis vectors, which is the definition of a basis.
PCA and dimensionality reduction: PCA finds eigenvectors (directions of maximum variance) of the data covariance matrix. These eigenvectors form an orthonormal basis. Suppose you have 1000 features. PCA ranks the 1000 directions by how much variance each captures. You keep only the top $k$ (say, 100) that together capture 99% of total variance. You then project the original data onto these $k$ orthonormal axes, reducing from 1000 dimensions to 100. Those 100 are your new basis — your new parameters. Training a model on 100 features instead of 1000 saves computation, reduces noise, and often improves generalization.
Feature scaling: In practice, normalizing input features (so each has comparable scale) prevents optimization algorithms from oscillating — a different but related normalization used throughout ML pipelines.

See section 5.6 for detailed coverage of normalization, Gram-Schmidt, and the PCA preview.

5.8.5 Summary, Gram-Schmidt, and Rank

Q: Can you summarize all the topics we covered today? A: The lecture progressed through six major blocks:

Dot products ( $x^{T} y$ ) — a scalar measure of similarity between two vectors. Defined component-wise ( $\sum x_{i} y_{i}$ ), in matrix notation ( $x^{T} y$ ), and geometrically ( $∥ x ∥∥ y ∥ cos θ$ ).
Inner products ( $x^{T} A y$ ) — a generalization where a symmetric positive definite matrix $A$ applies feature-specific weights. When $A = I$ , it reduces to the standard dot product.
Three axioms — symmetry ( $⟨ x, y ⟩ = ⟨ y, x ⟩$ ), bilinearity (scaling and distribution over addition work predictably in both arguments), and positive definiteness ( $⟨ x, x ⟩ > 0$ for $x \neq = 0$ ). These axioms define what qualifies as a valid inner product.
Derived concepts — vector length ( $∥ x ∥ = x^{T} x$ ), distance ( $d (x, y) = ∥ x - y ∥$ ), vector projection, Gram-Schmidt orthogonalization, and normalization.
Applications — recommendation systems, text embeddings, transformer attention, image clustering, and convex optimization (positive definite Hessians guarantee unique global minima).
Matrix rank — the number of linearly independent rows (or columns). Computed via echelon form (count non-zero rows). Interpreted geometrically as the dimension of the output space after the matrix transformation.

About Gram-Schmidt: we will work through a full example in the next session — finding the basis of a matrix's column space, then applying Gram-Schmidt to get the orthogonal basis, with visualization.

The detailed summary and exam tips are in sections 5.9 and 5.10.

Q: When checking rank, is it enough to look at column vectors? For the marks matrix, it was obvious the third column equals the sum of the first two. But what about the row vectors — do we need to check them separately? A: No, you never need to check both. Row rank always equals column rank — this is a fundamental theorem of linear algebra. Once you determine rank = 2 from the column perspective (by spotting that

c_{3} = c_{1} + c_{2}

), you know the row rank is also 2 — the three rows span only a 2-dimensional space. If you were to perform row reduction (Gaussian elimination) on the same matrix and count non-zero rows, you would again get 2. Both approaches converge to the same number. Choose whichever is computationally easier for the given matrix. See section 5.7.2 for the worked example and 5.7.3 for the echelon form method.

Q: What exactly is the difference between rank and dimension? I keep confusing them. A: They measure different things:

Dimension is a property of the input space. It counts how many coordinates (parameters) the input vector carries. A vector in $R^{3}$ has dimension 3 — it takes three numbers to specify it.
Rank is a property of the matrix transformation. It counts how many dimensions survive the transformation and appear in the output. When a matrix $A$ maps $x \mapsto y$ , rank = the dimension of the output vector $y$ 's space.

Scenario	Input dim	Transformation	Output dim (= rank)
Rotation matrix in 2D	2	Rotates but preserves all directions	2
Projection onto a plane in 3D	3	Squishes 3D onto 2D plane	2
Projection onto a line in 3D	3	Collapses 3D to 1D line	1
Zero matrix	any	Everything maps to $0$	0

Rank cannot exceed the minimum of (number of rows, number of columns). A matrix can "squash" a higher-dimensional input into a lower-dimensional output, but it can never create new dimensions. Rank tells you exactly how many dimensions remain in the output.

See section 5.7.4 for the geometric explanation with more examples.

Q: When computing rank via echelon form, do I count pivot columns or all non-zero columns? Do I count non-zero rows? A: All three are equivalent — they yield the same number:

Non-zero rows in echelon form: After row reduction, count rows that contain at least one non-zero entry. Each non-zero row has a leading (pivot) entry.
Pivot columns: A pivot column is one that contains a leading entry after row reduction. The number of pivot columns equals the number of non-zero rows, because each non-zero row houses exactly one pivot.
Non-zero columns: Be careful here — a column can be non-zero (contain some numbers) but still be linearly dependent on other columns. The correct method is to count pivot columns (or equivalently, non-zero rows), not just any column with numbers in it. In the echelon form $100010 * * 0$ , there are two non-zero rows, two pivot columns — rank = 2. The third column is non-zero yet linearly dependent; it does not contribute to rank.

Rule of thumb: reduce to echelon form and count non-zero rows. That is the most direct and least error-prone method.

See section 5.7.3 for the step-by-step echelon form computation.

5.9 Exam Guidance Summary

Here is what you need to know for the quiz based on the professor's guidance. Focus on the core topics below, and pay special attention to the pitfalls — these are the places where students most commonly lose marks.

Exam note: Dot Products and Inner Products

The quiz tests your ability to compute and interpret dot products. You must be able to:

Compute the dot product component-wise: $x \cdot y = \sum x_{i} y_{i} = x^{T} y$
Interpret the sign: positive $\Rightarrow$ angle < 90°, zero $\Rightarrow$ orthogonal, negative $\Rightarrow$ angle > 90°
Explain how the inner product generalizes the dot product: $⟨ x, y ⟩ = x^{T} A y$ with $A$ positive definite. This is a weighted dot product — different features get different importance.

Be ready for a question that asks you to compute a dot product from vector components, or to interpret what a given value means geometrically.

Exam note: Positive Definite Matrices

Two verification methods will be tested:

Eigenvalue check: A symmetric matrix is positive definite if and only if all its eigenvalues are strictly greater than zero.
Sum of squares decomposition: Write $x^{T} A x$ as a sum of squared terms. If it simplifies to an expression like $3 x_{1}^{2} + 2 x_{2}^{2}$ , the matrix is positive definite. If any term has a negative coefficient or the expression can be zero for a non-zero $x$ , it is not positive definite.

Given a $2 \times 2$ or $3 \times 3$ matrix, you should be able to apply either method. The sum-of-squares approach is often faster on a small matrix and avoids eigenvalue computation.

Exam note: Matrix Rank

To find the rank of a matrix:

Reduce the matrix to row echelon form using Gaussian elimination.
Count the number of non-zero rows — this is the rank.
Equivalent: count the number of pivot columns. Both give the same answer.

Remember: rank = dimension of the column space = dimension of the row space. It is always $\leq min (rows, columns)$ . If the rank equals the smaller dimension, the matrix has full rank.

Exam note: Gram-Schmidt Process

The Gram-Schmidt algorithm converts a set of linearly independent vectors into an orthogonal or orthonormal basis. The sequence is:

Project: project the current vector onto each previously computed vector.
Subtract: subtract those projections to make the current vector orthogonal to all previous ones.
Normalize (optional): divide by the vector's length to get unit length — this yields an orthonormal set.

Know the distinction: orthogonal = perpendicular ( $u_{i} \cdot u_{j} = 0$ for $i \neq = j$ ). Orthonormal = perpendicular and unit length ( $∣∣ u_{i} ∣∣ = 1$ ). The exam may ask for one or the other — read the prompt carefully.

Exam note: Next Session Preparation

The professor emphasized that the next lecture — covering eigenvectors and spectral decomposition — is essential preparation for the quiz. These concepts underlie how high-dimensional data is projected into lower dimensions (e.g., PCA). Pay close attention to:

How eigenvectors define directions of maximum variance
How eigenvalues quantify the importance of each direction
How spectral decomposition connects to the positive definiteness tests covered in this session

5.9.1 Common Pitfalls

These are the five most common mistakes students make on this material. Review them before the quiz.

1. Dot Product Sign Misinterpretation

A positive dot product means the angle between the two vectors is strictly less than 90° ( $cos θ > 0$ ). It does not mean the vectors are collinear (pointing in exactly the same direction), nor does it mean the angle is "less than 180°" — an angle of, say, 120° has a negative cosine. A dot product of zero means the angle is exactly 90° (the vectors are orthogonal). A negative dot product means the angle is greater than 90°. If you are unsure, think of the formula: $x \cdot y = ∣∣ x ∣∣ ∣∣ y ∣∣ cos θ$ . The sign depends entirely on $cos θ$ .

2. Rank and Dimension Confusion

Students frequently confuse the dimension of the input space (the number of columns) with rank (the dimension of the output/column space). Rank is the number of linearly independent directions spanned by the columns. A $3 \times 3$ matrix can have rank 2 if one column is a linear combination of the other two — it collapses a 3-dimensional input into a 2-dimensional subspace. Rank $\leq$ min(rows, columns). The echelon form method makes this explicit: rows that vanish reveal the dependencies.

3. Orthogonal vs. Orthonormal

After Gram-Schmidt, vectors are orthogonal by default (perpendicular to each other). They are not orthonormal unless you also divide each vector by its norm. An orthonormal set satisfies both $u_{i} \cdot u_{j} = 0$ (for $i \neq = j$ ) and $∣∣ u_{i} ∣∣ = 1$ . If the question says "orthonormal," do not stop after the subtraction step — normalize as well. Many students lose points by submitting an orthogonal set when orthonormal was requested.

4. Positive Definite vs. Positive Semi-Definite

Positive definite means $x^{T} A x > 0$ for all non-zero $x$ . Positive semi-definite means $x^{T} A x \geq 0$ . The distinction matters crucially in optimization: only a positive definite Hessian guarantees a unique global minimum. A positive semi-definite Hessian may have infinitely many minima (a flat basin). On the exam, if you decompose the quadratic form and get an expression like $x_{1}^{2} + 0 \cdot x_{2}^{2}$ , the matrix is semi-definite, not definite — because a non-zero vector $x = (0, x_{2})$ yields zero.

5. Vector Length: Squared vs. Actual

$x^{T} x = \sum x_{i}^{2}$ gives the squared length, not the length itself. The actual Euclidean length is $∣∣ x ∣∣ = x^{T} x$ . This is essential when normalizing vectors (step 3 of Gram-Schmidt) and when computing distances. Never submit a squared length as the final answer unless the question explicitly asks for $x^{T} x$ — the square root is not optional.

5.10 Key Industry Applications

Dot products and inner products are not just abstract mathematical constructs — they are the computational engines behind many of the machine learning systems we interact with every day. From the recommendation algorithms that shape what we watch to the attention mechanism inside large language models, these ideas are deployed at massive scale in production systems.

5.10.1 Recommendation and Search

Recommendation Systems: Collaborative Filtering and Content-Based Approaches

Platforms such as Netflix, LinkedIn, and Facebook represent users as vectors in a high-dimensional feature space. These vectors encode user behavior — movies watched, watch time, likes, shares, connection patterns, and countless other interaction signals. Computing the dot product between two user vectors produces a similarity score. When the dot product is high (the angle between the vectors is close to zero), the users are considered similar.

This is the foundation of two major recommendation paradigms:

Collaborative filtering: Users who have interacted with similar items in the past are assumed to have similar tastes. User vectors are compared via dot products, and items liked by similar users are recommended. LinkedIn's "People You May Know" and Facebook's friend suggestions use this approach — your vectorized profile is compared against millions of others.
Content-based filtering: Each item (movie, article, product) is itself vectorized based on its attributes (genre, description, metadata). Recommendations are made by computing the dot product between a user vector and item vectors — items with the highest scores are surfaced.

For multiple users, the pairwise dot products are computed across all pairs to build a dot product similarity matrix. For users $X_{1}, X_{2}, X_{3}, X_{4}$ , the entry $(i, j)$ is $X_{i} \cdot X_{j}$ . The diagonal entries ( $X_{i} \cdot X_{i}$ ) give the squared length of each vector — a self-similarity measure that is generally not of primary interest.

The Cold-Start Problem

A persistent challenge in recommendation systems is the cold-start problem: what do you recommend to a brand-new user about whom you have no historical data? With no interaction history, the user vector is essentially empty, and dot products with other vectors produce unreliable similarity scores. Platforms address this through several strategies:

Using demographic information (age, location, sign-up method) to initialize a rough vector.
Recommending globally popular items until the system gathers enough behavioral signals.
Asking new users to rate a few seed items during onboarding to bootstrap their vector.

As the user interacts with the platform, their vector gets refined — the dot product becomes a more reliable measure of their preferences.

5.10.2 Embeddings, Vision, and Optimization

Text Embeddings and Semantic Search

Text embedding models convert arbitrary text — a word, a sentence, a document — into a fixed-dimensional numeric vector where semantically similar texts map to vectors that are close together (high dot product).

Model	Vector Dimension
OpenAI text-embedding-3-small	512
OpenAI text-embedding-3-large	3072
Older small embedding models (OpenAI, Gemini)	~300
Older large embedding models (OpenAI, Gemini)	~768

These vectors are dense — every component carries a non-zero value. The dot product between two text vectors quantifies their semantic similarity. For instance:

Text 1: "I love Biryani" → vector $v_{1}$
Text 2: "Lucknow is known for Biryani" → vector $v_{2}$

Their dot product $v_{1} \cdot v_{2}$ would be high, correctly capturing the semantic overlap around "Biryani." At scale, this powers semantic search, document clustering, and retrieval-augmented generation (RAG) pipelines.

The TensorFlow embedding projector is a widely used visualization tool that projects high-dimensional embedding vectors into 2D or 3D for inspection. Words with similar meanings (e.g., "assassinated" and "murdered") cluster close together in the projection space because their vector representations have high dot products.

Transformer Attention: The Dot Product at the Heart of GPT

The attention mechanism that powers GPT and virtually all modern large language models depends on dot products. In scaled dot-product attention, each token in a sequence generates three vectors: a query ( $Q$ ), a key ( $K$ ), and a value ( $V$ ). The attention score between any two tokens is computed as the dot product of their query and key vectors:

Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V

Here is what each term means:

$Q K^{T}$ — a matrix of dot products between every query vector and every key vector. A high dot product means the model should "pay attention" to that token.
$d_{k}$ — the scaling factor. $d_{k}$ is the dimension of the key vectors. Dividing by $d_{k}$ prevents the dot products from growing too large, which would push the softmax into regions with extremely small gradients.
$softmax$ — converts the raw scores into a probability distribution (summing to 1) across all tokens.
Multiplying by $V$ — produces the final attention-weighted representation for each token.

This single formula, grounded in the elementary operation $Q \cdot K$ , is what allows transformers to capture long-range dependencies, translate languages, generate code, and answer questions. Every time you use ChatGPT, billions of dot products are computed across layers of attention heads.

Image Recognition and Photo Clustering

Google Photos extracts vectorized representations from every uploaded photo using convolutional neural networks. Each photo becomes a vector $V$ in a high-dimensional embedding space. Computing $V_{1} \cdot V_{2}$ — or equivalently the angle $θ$ between them — determines whether two photos contain the same person or similar scenes. When the dot product is high (the angle is close to zero), the photos are grouped into the same cluster.

This is how Google Photos performs face clustering: photos of the same person, even taken years apart under different lighting conditions, map to nearby vectors. The same principle powers object recognition, scene classification, and reverse image search.

Dimensionality Reduction with PCA

Principal Component Analysis (PCA) is one of the most widely deployed dimensionality reduction techniques in industry. Its goal is to project high-dimensional data onto a lower-dimensional subspace while preserving as much variance (information) as possible. The math relies on eigenvectors and orthonormal bases — concepts built directly on inner products.

The process works as follows:

Center the data by subtracting the mean of each feature.
Compute the covariance matrix — each entry is a dot product (or inner product) between centered feature columns.
Find the eigenvectors and eigenvalues of the covariance matrix. The eigenvectors define the directions of maximum variance (principal components), and the eigenvalues quantify how much variance each direction captures.
Sort eigenvectors by descending eigenvalues and retain the top $k$ components that collectively capture the desired variance — typically 95% or 99%.
Project the original data onto these $k$ orthonormal directions to get the reduced representation.

In practice, a dataset with 1000 features might be reduced to just 50 or 100 principal components while retaining 99% of the total variance. This yields dramatic savings in storage, computation time, and model complexity, and often improves generalization by eliminating noise and redundant features. PCA is used across domains: finance (portfolio optimization, risk modeling), genomics (gene expression analysis), computer vision (eigenfaces for face recognition), and natural language processing (latent semantic analysis).

Convex Optimization and Positive Definite Matrices

In machine learning, models are trained by minimizing a loss function — a measure of how far the model's predictions are from the true values. Optimization algorithms such as gradient descent iteratively adjust model parameters to reduce this loss.

The Hessian matrix — the matrix of second-order partial derivatives of the loss function — determines the curvature of the loss surface at a given point. If the Hessian is positive definite everywhere (all its eigenvalues are positive), the loss function is convex. A convex loss function has a single, unique global minimum. This is a powerful guarantee: start your optimization algorithm from any initialization, and you are mathematically assured of converging to the same optimal solution.

If the Hessian is not positive definite, the loss surface may contain local minima — dips where gradient descent could get stuck without ever reaching the best possible solution. Many modern deep learning architectures (neural networks, transformers) have non-convex loss surfaces with many local minima, which is why techniques like stochastic gradient descent with momentum, learning rate schedules, and careful initialization are essential.

The inner product provides the mathematical language for reasoning about these properties. A positive definite matrix $A$ satisfies $x^{T} A x > 0$ for all non-zero vectors $x$ — the exact condition that guarantees convexity when $A$ is the Hessian.

From recommendation engines that decide what you watch next to the attention layers inside GPT that generate coherent responses, dot products and inner products form the computational backbone of modern machine learning. The same formula — multiply corresponding components and sum — appears in collaborative filtering, semantic search, photo clustering, dimensionality reduction, and convex optimization. What began as a simple geometric operation on two vectors has become one of the most consequential mathematical tools in artificial intelligence.

MFML Lecture 05 notes · Dot Products and Inner Products

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-09

Sections Breakdown

15.1 Introduction to Dot Products

Why we need a number for similarity between vectors, with user, movie, and rideshare examples.

25.2 Mathematical Formulation of Dot Product

Component-wise, matrix, and geometric definitions of the dot product, plus vector length from self-dot-product.

35.3 Inner Products as a Generalization

Generalizing the dot product with a symmetric positive definite matrix A to weight features differently.

45.4 Properties of Inner Products

The three axioms every inner product must satisfy: symmetry, bilinearity, and positive definiteness.

55.5 Applications in Machine Learning

Recommendation systems, text embeddings, image clustering, and the transformer attention mechanism.

65.6 Derived Concepts from Inner Products

Distance between vectors, projection, Gram-Schmidt orthogonalization, and the bridge to PCA.

75.7 Matrix Rank

Rank via linear independence and echelon form, with the geometric view of rank as output dimension.

85.8 Student Q&A Log

Clarifications on notation, real-world usage, inner products, normalization, Gram-Schmidt, and rank.

95.9 Exam Guidance Summary

What to know for the quiz plus the five most common pitfalls students hit.

105.10 Key Industry Applications

Recommendation, search, embeddings, vision, dimensionality reduction, and convex optimization in production.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Dot Product

Must-know: The dot product compresses two vectors into one number that measures how alike they are. Positive means the vectors lean in the same direction; zero means perpendicular; negative means they point apart.

$x \cdot y = i = 1 \sum n x_{i} y_{i} = x^{T} y = ∥ x ∥ ∥ y ∥ cos θ$

Top pitfall: A large dot product does not always mean high similarity. A user who rates everything 5 will have a large dot product with almost everyone. Use cosine similarity to remove the effect of vector length.

Self-check: For [1,2] and [3,4], compute the dot product and say what its sign tells you about the angle between them.

Connects to: Inner Product, Vector Length, Cosine Similarity.

Inner Product (Weighted Dot Product)

Must-know: An inner product generalizes the dot product by inserting a symmetric positive definite matrix A between the vectors. The diagonal of A sets how much each feature matters, so you can weight bedrooms more than taps.

$⟨ x, y ⟩ = x^{T} A y$

Top pitfall: A is not allowed to be any matrix. It must be symmetric (Aᵀ = A) and positive definite (xᵀAx > 0 for every non-zero x). A non-symmetric matrix cannot define an inner product.

Self-check: If A = I, what does the inner product become, and why is that just the standard dot product?

Connects to: Dot Product, Positive Definite Matrices, Three Axioms.

Three Axioms of Inner Products

Must-know: Every inner product must satisfy symmetry, bilinearity, and positive definiteness. These three rules are what make lengths real and angles well-defined. Drop any one and the geometry breaks.

$⟨ x, y ⟩ = ⟨ y, x ⟩, ⟨ α x + β y, z ⟩ = α ⟨ x, z ⟩ + β ⟨ y, z ⟩, ⟨ x, x ⟩ > 0 for x \neq = 0$

Top pitfall: Students remember linearity in the first argument but forget the second. Both slots are linear. Also, positive definiteness is about the quadratic form xᵀAx, not about individual entries of A being positive.

Self-check: A matrix has positive diagonal entries but is not positive definite. Give a one-line reason this can happen.

Connects to: Inner Product, Positive Definite Matrices, Vector Length.

Distance, Projection, and Gram-Schmidt

Must-know: Distance is the length of the difference vector: d(x,y) = ||x - y||. Gram-Schmidt turns any set of independent vectors into an orthogonal (or orthonormal) set by subtracting projections step by step.

$u_{k} = v_{k} - i = 1 \sum k - 1 \frac{⟨ v _{k} , u _{i} ⟩}{⟨ u _{i} , u _{i} ⟩} u_{i}, e_{i} = \frac{u _{i}}{∥ u _{i} ∥}$

Top pitfall: Orthogonal means perpendicular; orthonormal means perpendicular AND unit length. After subtracting projections you have an orthogonal set. You must still divide by the norm to get orthonormal. Many exam answers stop one step early.

Self-check: Why must you subtract the projection onto every earlier uᵢ, not just the first one?

Connects to: Inner Product, Vector Length, Matrix Rank, PCA.

Matrix Rank

Must-know: Rank is the number of linearly independent rows (or columns) of a matrix. It equals the dimension of the output space after the transformation. Compute it by reducing to echelon form and counting non-zero rows.

$rank (A) = dim (col (A)) = dim (row (A)) \leq min (m, n)$

Top pitfall: Rank is not the number of columns. A 3×3 matrix can have rank 2 if one column is a sum of the others. Also, dimension is what goes in; rank is what comes out.

Self-check: A 3×3 matrix has columns c₃ = c₁ + c₂. What is its rank, and why?

Connects to: Linear Independence, Linear Combinations, Gram-Schmidt.

Positive Definite Matrices

Must-know: A symmetric matrix A is positive definite when xᵀAx > 0 for every non-zero x. This guarantees real lengths and a convex loss function with a unique minimum.

$x^{T} A x > 0 for all x \neq = 0$

Top pitfall: Positive definite (strict > 0) is not the same as positive semi-definite (≥ 0). A semi-definite Hessian can have many flat minima. Also, positive diagonal entries alone do not prove positive definiteness.

Self-check: How does a positive definite Hessian guarantee your optimizer will not get stuck in a local minimum?

Connects to: Inner Product, Three Axioms, Convex Optimization.

Previous LectureLecture 04 · Vector Spaces, Linear Independence, Basis and Dimension

Next LectureLecture 06 · Eigenvalues, Eigenvectors, and Determinants

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Dot Products and Inner Products

5.1 Introduction to Dot Products

5.1.1 Intuitive Motivation: Users, Movies, and Rideshares

5.1.2 Geometric Intuition: Visual Proximity as Similarity

5.2 Mathematical Formulation of Dot Product

5.2.1 Component-wise Definition

5.2.2 General Form for N-Dimensional Vectors

5.2.3 Matrix Notation

5.2.4 Geometric Definition: Angle Formulation

5.2.5 Computing Vector Length via Self-Dot-Product

5.3 Inner Products as a Generalization

5.3.1 From Identity Matrix to Arbitrary Matrix

5.3.2 Replacing I with a Positive Definite Matrix A

5.3.3 Positive Definite Matrix: Intuition and Worked Example

5.3.4 Proving Positive Definiteness: Class Exercise

5.4 Properties of Inner Products

5.4.1 Symmetry

5.4.2 Bilinearity

5.4.3 Positive Definiteness

5.5 Applications in Machine Learning

5.5.1 Recommendation Systems and User Similarity

5.5.2 Text Similarity and Word Embeddings

5.5.3 Image Recognition and Photo Clustering

5.5.4 Everyday Physical Intuition

5.6 Derived Concepts from Inner Products

5.6.1 Distance Between Vectors

5.6.2 Vector Projection Intuition

5.6.3 Gram-Schmidt Orthogonalization

5.6.4 Orthogonal and Orthonormal Vectors

5.6.5 Connection to PCA (Preview)

5.7 Matrix Rank

5.7.1 Definition via Linear Independence

5.7.2 Worked Example: Detecting Linear Dependence

5.7.3 Computing Rank via Reduced Echelon Form

5.7.4 Geometric Intuition: Rank as Dimension of Output Space

5.8 Student Q&A Log

5.8.1 Dot Product Computation and Notation

5.8.2 Applications and Real-World Usage

5.8.3 Inner Products and Positive Definite Matrices

5.8.4 Geometric Intuition and Normalization

5.8.5 Summary, Gram-Schmidt, and Rank

5.9 Exam Guidance Summary

5.9.1 Common Pitfalls

5.10 Key Industry Applications

5.10.1 Recommendation and Search

5.10.2 Embeddings, Vision, and Optimization

MFML Lecture 05 notes · Dot Products and Inner Products

Sections Breakdown

Exam Revision Notes

Dot Product

Inner Product (Weighted Dot Product)

Three Axioms of Inner Products

Distance, Projection, and Gram-Schmidt

Matrix Rank

Positive Definite Matrices

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat