A conditional probability, written $P(A\mid B)$, is the probability that event $A$ occurs given that event $B$ has already occurred. Read $P(A\mid B)$ as "probability of $A$ given $B$" — never "$A$ by $B$" or "$A$ slash $B$."

The definition:

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}$$

Here $P(A\cap B)$ is the joint probability — the probability that both $A$ and $B$ happen together. Some textbooks write this as $P(A,B)$. Both notations mean the same thing.

The symmetric form is:

$$P(B\mid A)=\frac{P(A\cap B)}{P(A)}$$

The key idea: conditioning restricts the sample space. When we condition on $B$, we throw away all outcomes where $B$ did not happen. The denominator $P(B)$ rescales what is left so the probabilities still sum to 1.

Rearrange the conditional probability definition and you get the multiplication law. The joint probability $P(A\cap B)$ can be written in two equivalent ways, depending on which event you treat as the condition:

• If $A$ happens first, then $B$: $P(A\cap B)=P(A)\cdot P(B\mid A)$
• If $B$ happens first, then $A$: $P(A\cap B)=P(B)\cdot P(A\mid B)$

Both forms give exactly the same number. Choose whichever conditional probability you already know.

Bayes theorem applies when you know that a later event $B$ has occurred and you want the probability that it was caused by one of several possible earlier events.

Setup: Let $A_1,A_2,A_3$ be three candidate causes. They must satisfy two conditions:

1. Mutually exclusive: Only one of them can happen at a time. Formally, $P(A_i\cap A_j)=0$ for $i\ne j$.
2. Exhaustive: Together they cover every possibility. Formally, $P(A_1)+P(A_2)+P(A_3)=1$.

If both conditions hold, then $B$ is an observed effect that follows after one of the $A_i$. We are given the conditional probabilities $P(B\mid A_1)$, $P(B\mid A_2)$, and $P(B\mid A_3)$.

Think of the $A_i$ as "causes" (flu, cold, allergy) and $B$ as a "symptom" (fever). The doctor sees the symptom and works backward to the cause.

The total probability of event $B$ sums over all possible paths. $B$ can happen through $A_1$ then $B$, or $A_2$ then $B$, or $A_3$ then $B$:

$$P(B)=\sum _{i=1}^{3}P(A_i)\cdot P(B\mid A_i)$$

Expanded:

$$P(B)=P(A_1)\cdot P(B\mid A_1)+P(A_2)\cdot P(B\mid A_2)+P(A_3)\cdot P(B\mid A_3)$$

Each term $P(A_i)\cdot P(B\mid A_i)$ is one path from cause to effect. Summing all paths gives the overall probability of the effect, regardless of which cause produced it.

If $B$ has already happened, the probability it was from cause $A_1$ is:

$$P(A_1\mid B)=\frac{P(A_1)\cdot P(B\mid A_1)}{P(B)}$$

The denominator $P(B)$ is the total probability from Section 5.1.4 — and it is the same denominator for all $A_i$. Only the numerator changes:

$$P(A_2\mid B)=\frac{P(A_2)\cdot P(B\mid A_2)}{P(B)},P(A_3\mid B)=\frac{P(A_3)\cdot P(B\mid A_3)}{P(B)}$$

When Bayes theorem is used for classification, the causes become classes and the effect becomes the data:

$$P(C_1\mid D)=\frac{P(D\mid C_1)\cdot P(C_1)}{P(D)},P(C_2\mid D)=\frac{P(D\mid C_2)\cdot P(C_2)}{P(D)}$$

Here $C_1$ and $C_2$ are two classes (e.g., "has disease" vs. "no disease"), and $D$ is the observed data (e.g., symptoms, test results).

Decision rule: Assign the new instance to whichever class gives the higher posterior probability $P(C_i\mid D)$. This is called the maximum a posteriori (MAP) decision rule.

The Naive Bayes classifier estimates the probability that a new data point belongs to each class. It multiplies together the conditional probabilities of its individual features. This works because it assumes those features are independent given the class.

The data setup: A collection of 24 emails, each labeled spam=yes or spam=no. The vocabulary (unique words) is: dear, friend, lunch, money.

Frequency table:

The class prior probabilities come from the row totals:

$$P(\text{Spam=Yes})=\frac{7}{24},P(\text{Spam=No})=\frac{17}{24}$$

The likelihoods (conditional probabilities of words given the class) come from each cell divided by its column total:

$$P(\text{Dear}\mid \text{Yes})=\frac{2}{7},P(\text{Dear}\mid \text{No})=\frac{8}{17},P(\text{Friend}\mid \text{Yes})=\frac{1}{7},P(\text{Friend}\mid \text{No})=\frac{5}{17}$$

With one word, Bayes theorem works directly:

$$P(\text{Spam}\mid \text{Dear})=\frac{P(\text{Dear}\mid \text{Spam})\cdot P(\text{Spam})}{P(\text{Dear})}$$

Each piece is available from the frequency table. No problem.

With two words — "dear" AND "friend" together — we need the joint probability:

$$P(\text{Spam}\mid \text{Dear}\cap \text{Friend})=\frac{P(\text{Dear}\cap \text{Friend}\mid \text{Spam})\cdot P(\text{Spam})}{P(\text{Dear}\cap \text{Friend})}$$

The problem: $P(\text{Dear}\cap \text{Friend}\mid \text{Spam})$ — the probability that both words appear together in a spam message — cannot be read from the frequency table. The table tells us how many times "dear" appears and how many times "friend" appears, but NOT how many times they appear together in the same message.

The solution: assume the words are conditionally independent given the class. This is the "naive" part of Naive Bayes. Under this assumption, the joint probability factors into a product:

$$P(\text{Dear}\cap \text{Friend}\mid \text{Spam})=P(\text{Dear}\mid \text{Spam})\cdot P(\text{Friend}\mid \text{Spam})$$

Conditional independence means: once you know which class an email belongs to, knowing whether it contains "dear" tells you nothing about whether it contains "friend." Each word's presence is independent of every other word's presence, within the same class.

What "naive" means: We assume independence without checking whether it is true. It is not our job to validate independence. We simply assume it and proceed. If we insisted on verifying independence first, we could never use Bayes theorem with multiple features — the joint probability table would be impossibly large.

For $n$ words, the factorization generalizes:

$$P(w_1,w_2,\dots ,w_n\mid C)=P(w_1\mid C)\cdot P(w_2\mid C)\cdot \dots \cdot P(w_n\mid C)$$

The Naive Bayes formula extends naturally when there are more features. For a weather classification problem with four features — outlook, temperature, humidity, windy — the factorization becomes:

$$P(\text{Class}\mid X_1,X_2,X_3,X_4)\propto P(X_1\mid \text{Class})\cdot P(X_2\mid \text{Class})\cdot P(X_3\mid \text{Class})\cdot P(X_4\mid \text{Class})\cdot P(\text{Class})$$

The only difference from the two-feature case is the number of multiplications. The logic is identical.

Consider a new message: "lunch money money money money." Using Naive Bayes, the factorization of $P(\text{Spam=Yes}\mid \text{Lunch, Money, Money, Money, Money})$ includes the term:

$$P(\text{Lunch}\mid \text{Spam=Yes})=\frac{0}{7}=0$$

Since this probability is zero, the entire product becomes zero — regardless of how strong the other evidence is. The word "money" appears four times and is strongly associated with spam ($4/7$), but the single unseen word "lunch" in the spam class nullifies everything.

This is the zero-probability problem: a single zero likelihood term makes the entire posterior probability zero. Naive Bayes cannot classify the message at all.

Where this happens: Text classification, sentiment analysis, and tagging applications are especially vulnerable. The vocabulary is huge. Training data is sparse. Many legitimate words will never appear in the training set for every class.

The fix: add the same small quantity to both the numerator and denominator of every probability estimate.

$$\frac{0}{7}\to \frac{0+k}{7+k}$$

Adding the same $k$ to numerator and denominator is like multiplying both by the same number. It does not change the meaning of a fraction when the numerator is already non-zero. For a zero numerator, it converts an impossible event into a merely unlikely one.

The most common choice is $k=1$. This is called Laplace smoothing (or the Laplace correction), named after Pierre-Simon Laplace, the French mathematician who pioneered Bayesian probability.

For the lunch example:

$$P(\text{Lunch}\mid \text{Spam=Yes})=\frac{0+1}{7+1}=\frac{1}{8}$$

The zero is replaced by a small but non-zero probability ($0.125$). The computation can now proceed, and the strong evidence from "money" (appearing four times) can properly influence the classification.

Smoothing is the general technique. The constant $k$ does not have to be 1. You could use $k=0.001$ to minimize the influence of the correction:

$$\frac{0}{7}\to \frac{0+0.001}{7+0.001}$$

In practice, add-one ($k=1$) is the most common and is built into standard libraries. If you use a non-standard constant, you need to write your own implementation.

In text classification, the denominator adjustment is often the total number of unique words (vocabulary size $|V|$) rather than 1. This is a text-classification-specific refinement.

Example — Sports vs Not-Sports tagging:

New sentence: "A very close game" — Sports or Not sports?

Word counts: Sports: 11 total words. Not sports: 9 total words.

Vocabulary (all unique words across both classes): A, great, game, The, election, was, over, Very, clean, match, but, forgettable, close, It → 14 unique words.

Raw probabilities (before smoothing):

$$P(\text{close}\mid \text{Sports})=\frac{0}{11}=0\text{— Naive Bayes fails}$$

After Laplace smoothing with vocabulary size $|V|=14$:

$$\begin{array}{rl}P(\text{A}\mid \text{Sports})& =\frac{2+1}{11+14}=\frac{3}{25}\\ P(\text{very}\mid \text{Sports})& =\frac{1+1}{11+14}=\frac{2}{25}\\ P(\text{close}\mid \text{Sports})& =\frac{0+1}{11+14}=\frac{1}{25}\\ P(\text{game}\mid \text{Sports})& =\frac{2+1}{11+14}=\frac{3}{25}\end{array}$$

For Not sports: each becomes $\frac{\text{count}+1}{9+14}=\frac{\text{count}+1}{23}$.

A random variable bridges the gap between abstract probability and real data. Given raw data (e.g., glucose levels of patients), you compute the frequency of each value, divide by the total number of observations, and obtain probabilities. Now the variable (glucose level) is associated with a probability for each of its possible values. This is the core idea: a variable whose values come with probabilities attached.

A random variable is a variable $X$ that takes on different numerical values, each with an associated probability. It is a function that maps each outcome in the sample space to a real number. Examples:

• Let $X$ be the number of heads when tossing 3 coins.
• Let $X$ be the glucose level of a randomly selected patient.
• Let $X$ be the temperature in a city on a given day.

Notation: $P(X=k)$ means "the probability that the random variable $X$ equals the value $k$." We use capital letters ($X$) for the random variable itself and lowercase letters ($x$ or $k$) for specific values it can take.

For 3 coins, listing all 8 outcomes is manageable. For 15 coins, the sample space has $2^{15}=32,768$ outcomes — listing them all would be impossible. But the random variable $X$ = number of heads simplifies everything:

• $X=0$: probability of 0 heads
• $X=1$: probability of 1 head
• …
• $X=15$: probability of 15 heads

The random variable absorbs the complexity of the full sample space and gives a compact, 16-line description of the probabilities. This is the power of random variables: they compress an enormous sample space into a manageable set of possible values with attached probabilities.

Random variables come in two fundamental types:

Discrete random variable: Takes a finite (or countably infinite) set of distinct values. You can list all possible values. The values are countable.

Examples: number of heads (0, 1, 2, 3, …), books in a library, passengers on a bus, goals in a match, steps taken.

Continuous random variable: Takes any value within an interval or range. The values are measurable — any real number in the range. You cannot list all possible values because there are infinitely many between any two values.

Examples: weight of a watermelon (2.1, 2.15, 2.153, … kg), rainfall amount, temperature, length of a beach, time to bake a cake.

Yes — absolutely. A random variable can take negative values. There is no rule that the values of a random variable must be non-negative. The only constraint is on probabilities, which must be between 0 and 1.

Example: Let $X$ be the temperature (in °C) in a cold region. $X$ could take values like $-1,-3,-2,-10,12$. Then $P(X=-3)$ is the probability that the temperature is exactly $-3$ degrees. This is a perfectly valid random variable.

Random variables are just numerical labels for outcomes. The numbers can be anything — positive, negative, zero, fractions — whatever the context demands.

Once a random variable $X$ is associated with probabilities $P(X)$, the next question is: what kind of pattern do these probabilities follow? Just as the number sequence $3,7,11,15,\dots$ follows the pattern "add 4 each time," probability values follow recognizable mathematical patterns. These patterns are called probability distributions.

Real-world motivation — why patterns matter:

• Movie theater: The pattern of customer arrivals tells you when to schedule staff.
• Petrol pump: Traffic patterns — morning vs. afternoon vs. evening — determine supply logistics.
• Stock market: Candlestick chart patterns help traders decide whether prices will rise or fall.

In every case, the goal is the same: identify the underlying pattern from the data so you can make informed decisions. Probability distributions are the mathematical language for describing these patterns.

There are standard, named probability distributions already defined in statistics. Each distribution captures a specific kind of pattern. The most common ones introduced at this level:

Each distribution has a formula (a probability mass function for discrete, a probability density function for continuous) that gives $P(X=x)$ for any value $x$. Once you identify which distribution your data follows, you can use its known properties to make predictions and quantify uncertainty.

The three-coin example from Section 5.4.3 is a Binomial distribution with $n=3$ trials and success probability $p=0.5$. Its probabilities $\frac{1}{8},\frac{3}{8},\frac{3}{8},\frac{1}{8}$ for $X=0,1,2,3$ follow the binomial formula $P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$.