Machine Learning

Naive Bayes Classifier and Ensemble Learning

📅 Published: 2026-06-29

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Naive Bayes fundamentals — introduced in Lecture 12 (Bayesian Learning: MLE, MAP, and Naive Bayes)
Occam's Razor and overfitting — covered in Lecture 9 (Decision Trees: Overfitting, Pruning, and MDL)
K-Nearest Neighbors — covered in Lectures 8 and 9 (Instance-Based Learning)
Bias-variance decomposition — referenced from Lecture 7 / earlier statistical foundations

Naive Bayes Classifier and Ensemble Learning

13.1 Review of Multinomial Naive Bayes

13.1.1 Definition and Core Assumption

**Hook:** Can you diagnose a disease by counting symptoms in a table — and still get the right answer most of the time, even when your counting ignores how symptoms interact? That is exactly what the Naive Bayes classifier does.

**Intuition + Analogy:** A *detective* walks into a crime scene. She has a prior hunch about who did it (based on past crime statistics). Then she gathers clues — fingerprints, witness reports, motive. Each clue shifts her hunch. She weighs each clue *independently*: "The fingerprint alone makes Suspect A twice as likely. The witness report alone makes Suspect A three times as likely." She never asks "do the fingerprint and the witness report correlate?" She just multiplies.

Naive Bayes is that detective. It treats every feature (clue) as an independent piece of evidence given the class (who did it). The word *naive* is the admission that this independence assumption is almost always false — fingerprints and witness reports *do* correlate. But, like our detective, the classifier works surprisingly well anyway because the ranking of suspects (class labels) is often preserved even when the exact probabilities are off.

**Where the analogy breaks:** The detective can reason about dependencies between clues; Naive Bayes cannot. It always multiplies unconditionally.

A *Naive Bayes classifier* is a probabilistic model that predicts a class label $Y$ from a set of features $X_{1}, X_{2}, \dots, X_{n}$ . The word "naive" comes from its one critical assumption: **conditional independence**. Every feature $X_{i}$ is assumed to be independent of every other feature, given the class $Y$ :

$P (X_{i}, X_{j} ∣ Y) = P (X_{i} ∣ Y) \cdot P (X_{j} ∣ Y) for all i \neq = j$

Only under this assumption does the probability factor into a simple product. If you know the class (say, whether someone will play tennis), then knowing the outlook tells you nothing extra about the humidity — they are treated as completely separate pieces of evidence.

The classifier builds on three core probability concepts from Bayesian learning:

**Prior probability** $P (Y = y)$ — How common is the class in general, before seeing any features. Before the doctor hears any symptoms, there is already some baseline probability that a person has a particular disease (maybe 1 in 1000). That is the prior.
**Likelihood** $P (X ∣ Y = y)$ — If the class is $y$ , how likely are these specific feature values? Given that someone has the disease, how likely are these particular symptoms?
**Posterior probability** $P (Y = y ∣ X)$ — After seeing the features, what is the updated probability of each class? This is what we compute and compare.

The classification rule is the **maximum a posteriori (MAP)** decision:

$y^= ar g y max P (Y = y ∣ X = x) = ar g y max P (Y = y) i = 1 \prod n P (X_{i} = x_{i} ∣ Y = y)$

The denominator $P (X = x)$ is dropped because it is the same for every class — it does not affect which class maximizes the expression.

13.1.2 Symbol Registry — Multinomial Naive Bayes

Symbol	Meaning	LaTeX	Type/Domain
$Y$	Class variable (target)	$Y$	Categorical, e.g. ${yes, no}$
$X_{i}$	The $i$ -th feature (attribute)	$X_{i}$	Categorical in multinomial case
$P (Y = y)$	Prior probability of class $y$	$P (Y = y)$	Scalar in $[0, 1]$
$P (X_{i} = x_{i} ∣ Y = y)$	Likelihood of feature $X_{i}$ taking value $x_{i}$ given class $y$	$P (X_{i} ∣ Y = y)$	Scalar in $[0, 1]$
$P (Y = y ∣ X)$	Posterior probability of class $y$ given features $X$	$P (Y = y ∣ X)$	Scalar in $[0, 1]$
$y^$	Predicted class (argmax over posteriors)	$y^$	Same set as $Y$

13.1.3 The Four Steps

The Naive Bayes prediction follows these four steps every time:

**Calculate the prior probability** — Count how many samples belong to each class, divide by total samples: $P (Y = y) = \frac{count ( Y = y )}{total samples}$ .
**Calculate the likelihood** — For each feature value, count how many times it appears with each class, divide by the total count of that class. Then take the product across all features: $\prod_{i = 1}^{n} P (X_{i} = x_{i} ∣ Y = y)$ .
**Combine prior and likelihood** — Multiply the prior probability by the product of the likelihoods to get the (unnormalized) posterior: $P (Y = y) \times \prod_{i = 1}^{n} P (X_{i} = x_{i} ∣ Y = y)$ .
**Predict the class with the highest posterior** — Take the argmax over all classes: $y^= ar g max_{y} P (Y = y ∣ X = x)$ .

13.1.4 Worked Example — Play Tennis

**Problem:** Predict whether a person will play tennis (yes/no) given:

Outlook = sunny, Temperature = cool, Humidity = high, Wind = strong

**Training data (14 records):** 9 labeled "yes", 5 labeled "no". Full counts from the standard PlayTennis dataset (Table 3.2, Mitchell):

Class	Outlook:Sun	Outlook:Over	Outlook:Rain	Temp:Hot	Temp:Mild	Temp:Cool	Hum:High	Hum:Normal	Wind:Weak	Wind:Strong	Total
Yes	2	4	3	2	4	3	3	6	6	3	9
No	3	0	2	2	2	1	4	1	2	3	5

**Step 1 — Prior probabilities:**

$P (play = yes) = \frac{9}{14} \approx 0.643, P (play = no) = \frac{5}{14} \approx 0.357$

**Step 2 — Likelihoods for the test instance (sunny, cool, high, strong):**

For the "yes" class: $P (sunny ∣ yes) P (cool ∣ yes) P (high ∣ yes) P (strong ∣ yes) = \frac{2}{9} = \frac{3}{9} = \frac{1}{3} = \frac{3}{9} = \frac{1}{3} = \frac{3}{9} = \frac{1}{3}$

For the "no" class: $P (sunny ∣ no) P (cool ∣ no) P (high ∣ no) P (strong ∣ no) = \frac{3}{5} = \frac{1}{5} = \frac{4}{5} = \frac{3}{5}$

**Step 3 — Unnormalized posterior probabilities:**

$P (yes ∣ X) \propto \frac{9}{14} \times \frac{2}{9} \times \frac{3}{9} \times \frac{3}{9} \times \frac{3}{9} = \frac{9}{14} \times \frac{2 \times 3 \times 3 \times 3}{9 \times 9 \times 9 \times 9} = \frac{9}{14} \times \frac{54}{6561} = \frac{486}{91854} \approx 0.00529$

$P (no ∣ X) \propto \frac{5}{14} \times \frac{3}{5} \times \frac{1}{5} \times \frac{4}{5} \times \frac{3}{5} = \frac{5}{14} \times \frac{3 \times 1 \times 4 \times 3}{5 \times 5 \times 5 \times 5} = \frac{5}{14} \times \frac{36}{625} = \frac{180}{8750} \approx 0.02057$

**Step 4 — Prediction:**

Since $0.02057 > 0.00529$ , the predicted class is **no** — the person will not play tennis. The normalized posterior for "no" is $\frac{0.02057}{0.00529 + 0.02057} \approx 0.795$ , confirming ~80% confidence.

**Sense-check:** Sunny outlook and high humidity are both strongly associated with "no" (3/5 and 4/5 respectively), so the prediction aligns with the data patterns.

13.1.5 Why the Denominator $P (X)$ Is Ignored

**Q:** Why don't we compute the denominator $P (X)$ in the Bayes formula?

**A:** $P (X)$ is the marginal probability of the observed feature vector — the same number regardless of which class we are considering. It is a normalizing constant:

$P (Y = y ∣ X) = \frac{P ( X ∣ Y = y ) \cdot P ( Y = y )}{P ( X )} \propto P (X ∣ Y = y) \cdot P (Y = y)$

Since we only compare which class gives the larger posterior (the argmax), the constant $P (X)$ cancels out during comparison. Computing it would require summing over all classes: $P (X) = \sum_{y} P (X ∣ Y = y) P (Y = y)$ , which adds work without changing the answer.

13.1.6 Student Questions

**Q:** Why is it called "probability without any info"?

**A:** That is the *prior probability* — what you believe before seeing any evidence (any feature values). It captures how common each class is in the training data, independent of any specific attribute. It is "without any info" about *this particular test instance*, though it does use info about the overall class distribution.

**Q (several students):** If the independence assumption is almost never true, why does Naive Bayes still work?

**A:** Three reasons: (1) Even when the exact probability values are wrong, the *ranking* of classes is often correct — and classification only needs the argmax, not the exact probabilities. (2) The bias from the independence assumption acts as a regularizer, preventing overfitting on small datasets. (3) Extensive empirical studies (Domingos & Pazzani, 1996; Friedman, 1997) have shown Naive Bayes is competitive with far more sophisticated classifiers on many real-world problems, especially text classification.

13.1.7 Assumptions and Scope

**Assumptions:**

**Conditional independence:** $P (X_{i}, X_{j} ∣ Y) = P (X_{i} ∣ Y) \cdot P (X_{j} ∣ Y)$ . This is the "naive" part.
**Features are categorical** (for multinomial NB — continuous features need Gaussian NB, Section 13.2).
**Sufficient training data** to estimate each $P (X_{i} = x ∣ Y = y)$ reliably (see Section 13.4 for the zero-probability fix).
**The training data is representative** of the true distribution — no concept drift.

**When it breaks:**

Strongly correlated features: if two features always co-occur (e.g., "New York" and "NYC" in text), the model double-counts the evidence, inflating confidence.
Zero-frequency problem: a feature-class combination unseen in training forces the entire posterior to zero (fixed by Laplace smoothing, Section 13.4).
Imbalanced classes with few samples: prior estimates become unreliable.

13.1.8 Visual Intuition

Imagine a 2D scatter plot with two features $X_{1}$ and $X_{2}$ . Each axis runs from 0 to 1. The data points are colored by class (red circle vs. blue triangle). Naive Bayes draws two sets of axis-aligned independent 1D distributions — one set per class. The decision boundary is where the product of heights from the two 1D distributions is equal for both classes. Because the model treats axes independently, the boundary is a curve that can never capture diagonal correlations — it is always factorized along the axes. The takeaway: Naive Bayes sees the world one feature at a time and multiplies, so correlated features cause it to be overconfident in the wrong direction.

13.1.9 Pitfalls

**Forgetting to apply Laplace smoothing:** In any real implementation, unseen feature-class pairs produce zeros and wipe out predictions. Always enable smoothing (Section 13.4).
**Confusing prior and posterior:** The prior $P (Y)$ is based on class counts *before* seeing features. The posterior $P (Y ∣ X)$ is the updated belief. Many beginners compute the prior and stop.
**Multiplying many small probabilities:** The product of hundreds of likelihoods can underflow to zero in floating-point arithmetic. Fix: use log-probabilities: $lo g P (Y ∣ X) \propto lo g P (Y) + \sum_{i} lo g P (X_{i} ∣ Y)$ . Since log is monotonic, the argmax is unchanged.
**Assuming Naive Bayes is always fast enough:** Training is $O (n d)$ (n samples, d features), but prediction on millions of test points with thousands of classes can still be slow — the product must be computed for every class.

13.1.10 Recap and Bridge

Naive Bayes multiplies independent feature likelihoods with a class prior to pick the most probable class — a simple counting-based classifier that works well even when its independence assumption is violated. Next: when features are continuous (e.g., hours of study, temperature), counting fails — we turn to **Gaussian Naive Bayes** (Section 13.2).

13.1.11 Real-World & Domain Connection

Naive Bayes is the workhorse of **text classification**: spam filters (Gmail, Outlook), sentiment analysis, topic categorization, and document routing all use it or its variants. Despite deep learning advances, Naive Bayes remains popular for its speed, interpretability, and small-data performance. In the broader field of machine learning, it serves as the canonical *generative classifier* — it models $P (X, Y)$ rather than $P (Y ∣ X)$ directly — and is the baseline against which more complex classifiers are measured. It also appears in recommender systems, medical diagnosis, and fraud detection.

13.2 Gaussian Naive Bayes

13.2.1 Definition and Core Assumption

**Hook:** What if your features are not "yes/no" or "sunny/rainy" but numbers on a scale — like hours of study, temperature, or blood pressure? You cannot count "how many times did the student study exactly 4.2 hours?" because continuous values almost never repeat. Gaussian Naive Bayes answers this by replacing counting with curve-fitting.

**Intuition + Analogy:** A *bell-curve tailor* measures customers to make shirts. For each size category (small, medium, large), she fits a bell curve over the chest measurements she has seen. When a new customer walks in with a 42-inch chest, she doesn't ask "have I seen exactly 42 before?" Instead, she checks: how high is the small-people bell curve at 42? How high is the medium curve? How high is the large curve? The highest curve wins.

Gaussian Naive Bayes is that tailor. It fits a Gaussian (bell) curve to each feature within each class, then reads off the height of that curve at the test value. The class with the tallest combined height (prior × product of curve heights) is the prediction.

**Where the analogy breaks:** Real bell curves have tails that extend infinitely, so even a very unusual measurement gets a non-zero height. The tailor's physical measurements are bounded.

When features are *continuous-valued*, the multinomial counting approach no longer works. **Gaussian Naive Bayes** assumes that the conditional likelihood $P (X_{i} ∣ Y = y)$ follows a *Gaussian (normal) distribution*.

This is the key difference from multinomial Naive Bayes: instead of counting frequencies, you compute the **mean** $μ$ and **variance** $σ^{2}$ of each feature for each class, and then plug those into the probability density function (PDF) of the Gaussian distribution.

The assumption is *not* that the raw attribute values themselves follow a Gaussian distribution. Rather, it is that the **likelihood** — the conditional density of the feature given the class — follows a Gaussian distribution:

$X_{i} ∣ Y = k \sim N (μ_{i, k}, σ_{i, k}^{2})$

This is an important distinction: the raw data could be skewed, but within each class, the feature values are assumed to cluster around a class-specific mean with Gaussian spread.

13.2.2 Symbol Registry — Gaussian Naive Bayes

Symbol	Meaning	LaTeX	Type/Domain
$X_{i}$	The $i$ -th feature (continuous)	$X_{i}$	Scalar in $R$
$Y$	Class variable	$Y$	Categorical, e.g. ${pass, fail}$
$μ_{i, k}$	Mean of feature $i$ for class $k$	$μ_{i, k}$	Scalar in $R$
$σ_{i, k}^{2}$	Variance of feature $i$ for class $k$	$σ_{i, k}^{2}$	Scalar $> 0$
$N (μ, σ^{2})$	Gaussian (normal) distribution with mean $μ$ , variance $σ^{2}$	$N (μ, σ^{2})$	Probability distribution
$p (X_{i} = x ∣ Y = k)$	Likelihood density under Gaussian assumption	$p (x ∣ μ, σ^{2}) = \frac{1}{2 π σ ^{2}} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})$	Density in $(0, \infty)$ — not a probability

13.2.3 The Five Steps of Gaussian Naive Bayes

**Split the data by class** — Separate training records into groups, one per class label.
**Calculate the prior probability** — Same as multinomial: $P (Y = k) = \frac{count of class k}{total samples}$ .
**Estimate the Gaussian parameters** — For every feature $X_{i}$ and every class $k$ , compute the mean $μ_{i, k}$ and variance $σ_{i, k}^{2}$ . These are the *maximum likelihood estimates* (MLE) for a Gaussian distribution:

$μ_{i, k} = \frac{1}{n _{k}} j : y_{j} = k \sum x_{ij}, σ_{i, k}^{2} = \frac{1}{n _{k}} j : y_{j} = k \sum (x_{ij} - μ_{i, k})^{2}$

These are exactly the familiar mean and variance formulas — the MLE derivation simply confirms that these natural estimators are optimal under the Gaussian assumption.

**Calculate the likelihood using the Gaussian PDF** — Plug the computed mean and variance into the Gaussian probability density function:

$p (X_{i} = x ∣ Y = k) = \frac{1}{2 π σ _{i, k}^{2}} exp (- \frac{( x - μ _{i, k} ) ^{2}}{2 σ _{i, k}^{2}})$

The formula has three pieces: a normalizing constant $\frac{1}{2 π σ ^{2}}$ (taller for smaller variance), the squared deviation $(x - μ)^{2}$ (distance from the center), and the exponential $exp (- \dots)$ that maps the deviation to a density between 0 and the constant.

**Apply Bayes rule and predict** — Multiply prior × product of densities, compare across classes, take the argmax:

$y^= ar g k max P (Y = k) i = 1 \prod d p (X_{i} = x_{i} ∣ Y = k)$

13.2.4 Worked Example — Student Pass/Fail Based on Study Hours

**Problem:** Predict whether a student will pass or fail based on hours of study. The feature $X$ (hours of study) is continuous. The new data point is $X = 4$ hours.

**Training data (6 records):**

Fail class: {1.5, 2.0, 3.0}
Pass class: {3.0, 4.5, 5.5}

*(Note: The pass-class values here are chosen to make the example concrete and numerically consistent. The essential pattern — pass students study more — is what matters.)*

**Step 1 — Split data by class:**

Fail: 3 records
Pass: 3 records

**Step 2 — Prior probabilities:**

$P (Y = fail) = \frac{3}{6} = 0.5, P (Y = pass) = \frac{3}{6} = 0.5$

**Step 3 — Compute mean and variance for each class:**

*Fail class:* $μ_{fail} = \frac{1.5 + 2.0 + 3.0}{3} = \frac{6.5}{3} = 2.167$ $σ_{fail}^{2} = \frac{( 1.5 - 2.167 ) ^{2} + ( 2.0 - 2.167 ) ^{2} + ( 3.0 - 2.167 ) ^{2}}{3} = \frac{( - 0.667 ) ^{2} + ( - 0.167 ) ^{2} + ( 0.833 ) ^{2}}{3} = \frac{0.444 + 0.028 + 0.694}{3} = \frac{1.166}{3} = 0.389$

*Pass class:* $μ_{pass} = \frac{3.0 + 4.5 + 5.5}{3} = \frac{13.0}{3} = 4.333$ $σ_{pass}^{2} = \frac{( 3.0 - 4.333 ) ^{2} + ( 4.5 - 4.333 ) ^{2} + ( 5.5 - 4.333 ) ^{2}}{3} = \frac{( - 1.333 ) ^{2} + ( 0.167 ) ^{2} + ( 1.167 ) ^{2}}{3} = \frac{1.778 + 0.028 + 1.361}{3} = \frac{3.167}{3} = 1.056$

**Step 4 — Likelihood for the test point $X = 4$ :**

For fail class: $p (X = 4 ∣ fail) = \frac{1}{2 π \times 0.389} exp (- \frac{( 4 - 2.167 ) ^{2}}{2 \times 0.389}) = \frac{1}{2.444} exp (- \frac{( 1.833 ) ^{2}}{0.778}) = \frac{1}{1.563} exp (- \frac{3.361}{0.778}) = 0.640 \times e^{- 4.322} = 0.640 \times 0.0133 = 0.0085$

For pass class: $p (X = 4 ∣ pass) = \frac{1}{2 π \times 1.056} exp (- \frac{( 4 - 4.333 ) ^{2}}{2 \times 1.056}) = \frac{1}{6.635} exp (- \frac{( - 0.333 ) ^{2}}{2.111}) = \frac{1}{2.576} exp (- \frac{0.111}{2.111}) = 0.388 \times e^{- 0.0526} = 0.388 \times 0.949 = 0.368$

**Step 5 — Posterior probabilities and prediction:**

$P (fail ∣ X = 4) P (pass ∣ X = 4) \propto 0.5 \times 0.0085 = 0.00425 \propto 0.5 \times 0.368 = 0.184$

Since $0.184 > 0.00425$ , the classifier predicts **pass** — a student who studied 4 hours is predicted to pass.

**Sense-check:** 4 hours is closer to the pass-class mean (4.33) than the fail-class mean (2.17). The pass density at 4 is much higher, so the prediction is intuitive. The normalized posterior for pass is $\frac{0.184}{0.184 + 0.00425} \approx 0.977$ — very confident.

13.2.5 Student Questions

**Q:** Where does the PDF formula come from? How is it derived?

**A:** Every probability distribution has an associated probability density function (PDF) that describes how probability mass is spread across its domain. The Gaussian PDF is derived from the definition of the normal distribution. The derivation itself is beyond scope for this course — it is an open-book exam, so you can refer to the textbook for the exact formula. What matters is that you understand the concept and can apply the formula. The formulas for mean and variance as MLE estimates for the Gaussian distribution were covered in the previous class (MLE derivation).

**Q:** Does the method change if there are more records in the training data?

**A:** No, the steps are identical regardless of dataset size. The only difference is that with more data, the mean and variance estimates become more reliable, and the Gaussian curves become better approximations of the true class-conditional distributions. The process (split → compute μ,σ² → plug into PDF → apply Bayes) stays exactly the same.

13.2.6 Assumptions and Scope

**Assumptions:**

**Conditional independence** (same as multinomial NB): $p (X_{i}, X_{j} ∣ Y) = p (X_{i} ∣ Y) \cdot p (X_{j} ∣ Y)$ .
**Gaussian likelihood:** Within each class, each continuous feature follows a normal distribution. This is stronger than the multinomial assumption and is more often violated in practice.
**Features are continuous** — categorical features should use multinomial NB (or be one-hot encoded and treated with care).

**When it breaks:**

Heavy-tailed or multi-modal feature distributions within a class: a single Gaussian cannot capture two clusters (e.g., students who study 2 hours AND students who study 8 hours both pass — the single bell curve averages them poorly).
Outliers: The Gaussian PDF has thin tails. A single extreme value can dramatically inflate the variance estimate, making the distribution too flat for the majority of data.
Small sample size per class: estimating a variance from 2-3 points is highly unreliable — the PDF height can be wildly off.

13.2.7 Visual Intuition

Picture a single horizontal axis labeled "Hours of Study" from 0 to 8. Two bell curves sit on this axis: a blue one (fail class) centered at μ = 2.17 with wide spread, and a green one (pass class) centered at μ = 4.33 with moderate spread. At $X = 4$ , draw a vertical line — it cuts through the green curve high up (density ~0.37) and through the blue curve near its right tail (density ~0.009). The heights of these intersections are the likelihoods. Multiply each by the prior 0.5, and the green tower is much taller. The takeaway: the class whose bell curve peaks closest to the test point (in standard-deviation-scaled distance) wins.

13.2.8 Pitfalls

**Using variance σ² = 0:** If all samples in a class have the identical feature value, variance is zero and the PDF denominator becomes 0 — division by zero. Real implementations add a tiny constant (e.g., 1e-9) to variance to prevent this.
**Confusing density with probability:** The PDF value at a point can be > 1 (e.g., a very narrow Gaussian). This is fine — a density is not a probability. Only the integral over an interval gives a probability.
**Forgetting that PDF outputs are multiplied:** Even if individual densities look reasonable, the product of many densities (for many features) can underflow. Use log-densities: $lo g p (X ∣ Y) = - \frac{1}{2} lo g (2 π σ^{2}) - \frac{( x - μ ) ^{2}}{2 σ ^{2}}$ .
**Assuming raw data must be Gaussian:** The assumption is about the *class-conditional* distribution, not the marginal. The overall data histogram may look nothing like a bell curve — that is fine.

13.2.9 Recap and Bridge

Gaussian Naive Bayes fits a bell curve to each feature within each class, then uses curve height (density) instead of counting for continuous features. Next: the **zero probability issue** — what happens when a feature-class pair is missing from training, and why it can cripple both multinomial and Gaussian NB (Section 13.3).

13.2.10 Real-World & Domain Connection

Gaussian Naive Bayes is used in **medical diagnosis** with continuous features (blood pressure, cholesterol level, age), in **sensor fault detection** (vibration readings, temperature), and in **financial risk scoring** (income, debt-to-income ratio). It is the default continuous-feature variant in scikit-learn's `GaussianNB`. Compared to multinomial NB, it makes a stronger distributional assumption, so it is typically used when domain knowledge suggests features are roughly bell-shaped within classes — or when you have enough data that the Gaussian approximation is reasonable.

13.3 Zero Probability Issue

13.3.1 The Problem

**Hook:** Imagine a spam filter that works perfectly — until someone sends a spam email containing the word "cryptocurrency," a word that never appeared in any of your training spam. The filter sees one zero in a long product of probabilities and concludes: "P(spam) = 0." The entire email, no matter how spammy otherwise, is declared safe. This is the zero probability problem.

**Intuition + Analogy:** A *jury* is deliberating a verdict. Ten pieces of evidence strongly point to "guilty." But one juror says: "I've never seen a guilty person wear a blue shirt. Therefore, the probability this person is guilty, wearing blue, is zero." The juror multiplies all evidence together and the single zero destroys the entire case for guilt.

Naive Bayes is that juror. Because the posterior is a **product** of probabilities, a single zero term anywhere in the chain makes the entire product zero:

$P (Y = y ∣ X) \propto P (Y = y) \times i = 1 \prod n P (X_{i} = x_{i} ∣ Y = y)$

If any $P (X_{i} = x_{i} ∣ Y = y) = 0$ , then the whole expression collapses.

**If a specific feature value never occurs with a particular class in the training data, the Naive Bayes algorithm assigns a probability of zero to that entire class prediction.** Because the posterior is a product, a single zero term in the likelihood product wipes out everything — regardless of how strongly the other features support that class.

13.3.2 Case 1 — Penalizing a Single Class

**Scenario:** Tax evasion prediction with 10 samples — 3 labeled "yes" (evade), 7 labeled "no" (not evade). Attributes: refund status (yes/no), marital status (single/married/divorced), and taxable income (continuous).

**Test record:** A person who is married with refund = yes. Predict: will they evade tax?

**Prior probabilities:** $P (evade = yes) = \frac{3}{10}, P (evade = no) = \frac{7}{10}$

**The problem:** Among the 3 evaders (yes class), none are married — one is divorced, two are single. So:

$P (marital = married ∣ evade = yes) = \frac{0}{3} = 0$

When the posterior is computed by multiplying prior × likelihood product, this single zero term makes the entire posterior for "yes" equal to zero. Even if every other feature (high income, refund = yes) strongly suggests tax evasion, the classifier will predict "no evasion" solely because no married person happened to appear in the "yes" training samples.

$P (yes ∣ X) \propto \frac{3}{10} \times P (refund = yes ∣ yes) \times 0 \times \dots = 0$

The classifier completely wipes out the chance of predicting "yes" due to a single unseen feature-class combination. This is the **zero probability issue**.

13.3.3 Case 2 — Both Classes Become Zero

**Scenario:** Same tax evasion dataset, but now delete record #7. Record #7 was the only divorced person in the "no" class. After deletion, the count of divorced people in the "no" class drops to zero.

**Test record:** A person with refund = yes, marital = divorced, taxable income = 120K.

Now compute the likelihood for both classes:

For "no": $P (divorced ∣ no) = \frac{0}{6} = 0$ (the only divorced "no" record was deleted). So $P (X ∣ no) = 0$ .
For "yes": $P (refund = yes ∣ yes) = \frac{0}{3} = 0$ (no evader in the training data had refund = yes). So $P (X ∣ yes) = 0$ .

**Result:** Both posterior probabilities are zero: $P (yes ∣ X) \propto 0, P (no ∣ X) \propto 0$

The classifier **cannot predict either class** — it is completely paralyzed. In Case 1, at least one class still had a non-zero posterior; in Case 2, neither class does. This is a total classification failure.

13.3.4 Student Questions

**Q:** Could this happen in practice?

**A:** Yes — it happens whenever a feature value for a particular class is missing from the training sample, which is very common with:

Small datasets (few training examples per class)
Rare categories (e.g., "divorced" status may only appear in a few records)
High-cardinality features (many unique values, each seen only a few times)
Text data (most words in the vocabulary never appear with most classes)

It may simply be a sampling artifact — the value does appear in the real world but not in your particular training set. The model has no way to distinguish "impossible" from "unseen in my limited sample." This is why **Laplace smoothing** (Section 13.4) is essential in practice.

13.3.5 Assumptions and Scope

**Scope:** The zero probability issue is inherent to any naive Bayes variant (multinomial, Gaussian, Bernoulli) that uses raw count-based probability estimates. It is most severe when:

The dataset is small relative to the number of feature-value combinations
Features have many unique values (high cardinality)
Classes are imbalanced (the minority class has few samples, so many feature-value pairs are unseen for that class)

13.3.6 Visual Intuition

Picture a bar chart for each class showing the estimated probability of each feature value. For the "yes" class, the bar for "married" is at height 0 — it is literally missing. When the posterior is computed, it is like multiplying many positive numbers and then hitting a zero — the entire tower collapses to ground level. The fix (Laplace smoothing) lifts every bar slightly off the floor so that no bar is ever exactly at zero.

13.3.7 Pitfalls

**Assuming "it won't happen to me":** Even with thousands of training examples, high-dimensional data (e.g., text with 10,000+ word vocabulary) guarantees many zero-frequency events. Laplace smoothing is not optional — it is standard.
**Deleting records without rechecking zeros:** Case 2 shows that removing even *one* training record can introduce new zero probabilities. If you preprocess your data, re-verify your probability tables.
**Thinking Gaussian NB is immune:** Gaussian NB uses densities, not counts, so it never produces exact zeros from missing values. But if every sample in a class has the identical feature value, variance becomes 0, and the PDF blows up (division by zero). A small variance floor is still needed.

13.3.8 Recap and Bridge

A single unseen feature-class pair makes the entire Naive Bayes posterior zero — a catastrophic failure mode. The fix: **Laplace smoothing**, which adds a small pseudo-count to every estimate so no probability is ever exactly zero (Section 13.4).

13.3.9 Real-World & Domain Connection

The zero probability problem is the primary reason that production spam filters, document classifiers, and recommender systems always use some form of smoothing. Without it, any new word or rare category would break the classifier. The problem is also a concrete illustration of the broader statistical principle: *maximum likelihood estimates can be zero for unobserved events, but zero is almost never the right answer* — it reflects sampling limitations, not reality.

13.4 Laplace Smoothing

13.4.1 Definition and Formula

**Hook:** How do you estimate the probability of an event you have never seen? If you have flipped a coin 5 times and never got heads, is $P (heads) = 0$ ? Laplace smoothing answers: "Pretend you saw it once — just to be safe."

**Intuition + Analogy:** A *new restaurant* has served 10 customers. All 10 ordered pasta — no one ordered pizza. Should the owner conclude $P (pizza) = 0$ and remove pizza from the menu? Of course not. She reasons: "I haven't seen a pizza order *yet*, but I know pizza exists. Let me act as if I have already seen every dish ordered at least once, just in small amounts."

Laplace smoothing (also called *add-one smoothing*) does exactly this: it adds one imaginary observation of every possible feature value to every class, so no probability estimate is ever exactly zero. It introduces a small, controlled bias to guard against the disaster of zero probabilities (Section 13.3).

**Standard (unsmoothed) maximum likelihood estimate:**

$P (X_{i} = a ∣ Y = c) = \frac{n _{i c}}{n _{c}}$

where $n_{i c}$ is the number of instances with attribute value $a$ in class $c$ , and $n_{c}$ is the total number of instances in class $c$ . This formula can produce zeros when $n_{i c} = 0$ .

**Laplace smoothing (add-one) formula:**

$P (X_{i} = a ∣ Y = c) = \frac{n _{i c} + 1}{n _{c} + V}$

where:

$n_{i c}$ — number of times feature $X_{i}$ takes value $a$ in class $c$
$n_{c}$ — total number of samples in class $c$
$V$ — number of *unique possible values* of feature $X_{i}$ across the entire dataset (the vocabulary size for that feature)

**Why this works:**

The "+1" in the numerator ensures the numerator is never zero — it acts as if we have already seen this feature value at least once in every class.
The "+ $V$ " in the denominator ensures the probabilities still sum to 1 across all $V$ possible values of that feature (normalization):

$a \in values (X_{i}) \sum \frac{n _{i c} + 1}{n _{c} + V} = \frac{\sum _{a} n _{i c} + V}{n _{c} + V} = \frac{n _{c} + V}{n _{c} + V} = 1$

For example, if marital status has three unique values (single, married, divorced), then $V = 3$ . If refund has two unique values (yes, no), then $V = 2$ .

13.4.2 Worked Example — Laplace Smoothing on Tax Evasion

**Before smoothing (from Section 13.3):** $P (refund = yes ∣ evade = yes) = \frac{0}{3} = 0$ $P (marital = married ∣ evade = yes) = \frac{0}{3} = 0$

**After Laplace smoothing:**

For refund (V = 2: yes, no): $P (refund = yes ∣ evade = yes) = \frac{0 + 1}{3 + 2} = \frac{1}{5} = 0.2$

Denominator: 3 (total yes-class records) + 2 (unique values of refund) = 5.

For marital status (V = 3: single, married, divorced): $P (marital = married ∣ evade = yes) = \frac{0 + 1}{3 + 3} = \frac{1}{6} \approx 0.167$

Denominator: 3 (total yes-class records) + 3 (unique values of marital status) = 6.

Now neither probability is zero, and the posterior product survives — the classifier can make a meaningful comparison between classes.

**Sense-check:** The smoothed probabilities are small (0.2 and 0.167) but non-zero, reflecting the fact that we have little evidence for these combinations. As more training data accumulates, the $n_{c}$ term dominates and the smoothed estimates converge to the true frequencies.

13.4.3 M-Estimate (Advanced Variant)

There is a more flexible version called the *m-estimate* (or *add-m smoothing*), where instead of always adding 1, you add a weighted pseudo-count:

$P (X_{i} = a ∣ Y = c) = \frac{n _{i c} + m \cdot p}{n _{c} + m}$

where:

$m$ — the *equivalent sample size*, controlling how heavily to weight the prior belief relative to observed data
$p$ — a *prior estimate* of the probability (often a uniform prior: $p = 1/ V$ )

The standard Laplace smoothing is a special case where $m = V$ and $p = 1/ V$ , giving $m \cdot p = V \cdot (1/ V) = 1$ .

The m-estimate is the form presented in the textbook (Mitchell, Chapter 6) and in standard references. When $m = 0$ , it reduces to the unsmoothed MLE. The m-estimate is not the focus of this course but is mentioned for awareness — scikit-learn's `MultinomialNB` uses a related parameter `alpha` for additive smoothing (`alpha=1` corresponds to Laplace smoothing).

13.4.4 Text Classification with Laplace Smoothing

**Problem:** Classify the document "a very close game" as sports or not-sports.

**Training documents:**

"a great game" → sports
"the election was over" → not sports
"very clean match" → sports
"a clean but forgettable game" → sports
"it was a close election" → not sports

**Bag of words:** Each document is represented by word frequencies, ignoring word order. Standard NLP preprocessing (stop-word removal, stemming, lemmatization) is typically applied, but the core idea is: count how many times each word appears per class, across all documents of that class.

**Total word counts per class (simplified):**

Sports (3 docs): 11 word tokens total
Not sports (2 docs): 8 word tokens total
Unique words across all docs (vocabulary size $V$ ): 14

**The zero probability problem:** The word "close" does not appear in any sports-tagged training document — it appears only in the not-sports document ("it was a close election"). Without smoothing:

$P ("close" ∣ sports) = \frac{0}{11} = 0$

This would make the entire posterior for sports zero, even though "a very close game" sounds like sports.

**With Laplace smoothing:**

$P ("close" ∣ sports) = \frac{0 + 1}{11 + 14} = \frac{1}{25} = 0.04$

This small non-zero probability allows the product to remain meaningful. The final prediction becomes **sports** — which is correct. Without Laplace smoothing, the document would have been misclassified as not-sports.

**Key insight:** In text classification, the vocabulary size $V$ is typically very large (thousands or millions of words), so Laplace smoothing gives very small probabilities to unseen words — but these small values still prevent the product from collapsing to zero.

13.4.5 Spam Classification Example

**Features:**

$X_{1} = 1$ if the word "offer" appears in the email, 0 otherwise
$X_{2} = 1$ if the word "money" appears in the email, 0 otherwise

**Target:** $Y = spam$ or $Y = not spam$

**Process:**

Compute prior probabilities $P (spam)$ and $P (not spam)$ from training data.
Compute all likelihoods (with Laplace smoothing): $P (X_{1} = 1 ∣ spam)$ , $P (X_{1} = 0 ∣ spam)$ , $P (X_{1} = 1 ∣ not spam)$ , $P (X_{1} = 0 ∣ not spam)$ , and the same four for $X_{2}$ .
For a test email, select the likelihood that matches each observed feature value and multiply: $P (Y) \times P (X_{1} = x_{1} ∣ Y) \times P (X_{2} = x_{2} ∣ Y)$ .
Compare — the class with the higher posterior wins.

The denominator $P (X)$ is the same for both classes and can be omitted from explicit calculation.

13.4.6 Student Questions

**Q:** Do we always apply Laplace smoothing?

**A:** By default, yes. In scikit-learn's `MultinomialNB`, the smoothing parameter `alpha` defaults to 1.0 (Laplace smoothing). In `GaussianNB`, a small `var_smoothing` parameter (default $1 0^{- 9}$ ) is added to variance to prevent division by zero. You can disable smoothing, but in practice it is almost always kept on — the small bias it introduces is far less harmful than the risk of zero probabilities.

**Q (several students):** Does Laplace smoothing distort the "true" probabilities?

**A:** Yes — it deliberately introduces bias. For frequent events (large $n_{i c}$ ), the "+1" and "+V" have negligible effect. For rare events, the smoothed estimate is a compromise between the observed frequency (possibly zero) and a uniform prior belief. The bias is the price we pay for reliability — and it is almost always worth paying.

13.4.7 Assumptions and Scope

**Assumptions:**

The feature has a known, finite set of possible values $V$ (for categorical features). For text, this is the vocabulary size.
The uniform prior (assigning equal pseudo-count to all values) is reasonable. If you have strong prior knowledge that some values are more likely than others, the m-estimate with a non-uniform $p$ may be better.

**When it breaks:**

Very large $V$ relative to $n_{c}$ : each smoothed probability becomes about $1/ V$ (nearly uniform), washing out all signal. Example: a feature with 10,000 unique values and only 100 training samples — after smoothing, all probabilities are ~1/10100.
Continuous features: Laplace smoothing is for categorical counts. For Gaussian NB, variance smoothing (adding a small constant to $σ^{2}$ ) serves a similar purpose.

13.4.8 Visual Intuition

Picture a probability bar chart for feature "refund" in the "yes" class. Before smoothing: yes-bar at height 0, no-bar at height 1. A lonely zero bar. After Laplace smoothing: the zero bar lifts off the floor to height 0.2, and the 1.0 bar drops slightly to 0.8 to make room (probabilities must sum to 1). Every bar now has a non-zero height — no product will collapse. As more data arrives, the bars drift back toward their true frequencies.

13.4.9 Pitfalls

**Using $V$ incorrectly:** $V$ is the number of *unique possible values* of that specific feature, NOT the total number of features. For refund: V=2 (yes, no). For marital status: V=3 (single, married, divorced). Mixing these up produces wrong denominators.
**Forgetting that smoothing applies per-feature:** Each feature has its own $V$ . You do not use the same $V$ across all features.
**Disabling smoothing "because my dataset is large":** Even with 1 million emails, there will be words in the test set that never appeared in training (out-of-vocabulary words). Smoothing handles these gracefully.
**Confusing Laplace smoothing with the m-estimate on exams:** The professor may ask about the m-estimate form (see 13.4.3). Know that Laplace is the special case where the additive constant is exactly 1.

13.4.10 Recap and Bridge

Laplace smoothing adds a pseudo-count of 1 to every feature-value/class combination, preventing any probability from being zero and keeping the posterior product alive. Next: we step back and ask — **when** does Naive Bayes actually do its learning? The answer distinguishes eager vs. lazy learners (Section 13.5).

13.4.11 Real-World & Domain Connection

Laplace smoothing is the default in virtually every production text classifier (spam filters, sentiment analyzers, topic models). It is one of the simplest and most reliable forms of *regularization* in machine learning — trading a small amount of bias for a large reduction in catastrophic failure. The technique is named after Pierre-Simon Laplace, who introduced it in the 18th century to estimate the probability that the sun will rise tomorrow, given that it has risen every day in recorded history. His answer: $(n + 1) / (n + 2)$ , not $n / n = 1$ .

13.5 Naive Bayes as an Eager Learner

13.5.1 Eager vs. Lazy Learning

**Hook:** When does a student actually learn — while reading the textbook (before the exam) or while answering each question (during the exam)? Machine learning algorithms face the same choice.

**Intuition + Analogy:** Two chefs prepare for a dinner service. The *eager chef* preps everything in advance — chops vegetables, measures spices, preheats ovens — so that when an order comes in, she just assembles and plates in seconds. The *lazy chef* does nothing until the order arrives, then scrambles to chop, measure, and cook from scratch.

In machine learning: **eager learners** (Naive Bayes, linear regression, neural networks) do the heavy computation during training and make fast predictions. **Lazy learners** (KNN, case-based reasoning) store the data and do all computation at prediction time.

**Eager Learner:** Builds the model during training — computes and stores all parameters (probabilities, weights, splits) as soon as it sees the training data. At prediction time, only a fast forward pass or lookup is needed.

**Lazy Learner:** Stores the training data verbatim. No model is built during training. All computation (distance calculations, comparisons) is deferred to prediction time.

**Naive Bayes is an eager learner.** During training, it calculates and stores:

All prior probabilities $P (Y = y)$ for every class
All likelihoods $P (X_{i} = a ∣ Y = y)$ for every feature-value/class combination (multinomial NB)
All means $μ_{i, k}$ and variances $σ_{i, k}^{2}$ for every feature/class combination (Gaussian NB)

When test data arrives, the only step is: look up the stored probabilities, multiply, and compare. The heavy lifting is front-loaded.

**K-Nearest Neighbors (KNN) is a lazy learner.** During training, it stores the training data as-is. When a test point arrives, it computes distances to all training points, finds the K nearest, and predicts. All computation happens at query time.

13.5.2 Comparison — Eager vs. Lazy

Dimension	Eager (Naive Bayes)	Lazy (KNN)
Training time	Higher — builds full probability tables	Near zero — just stores data
Prediction time	Low — just multiply and compare	High — compute distances to all training points
Memory	Stores compact model parameters	Stores entire training dataset
Updates with new data	Must recompute all probabilities	Just add to the stored set
Interpretability	High — probabilities are directly inspectable	Low — "because the neighbors said so"

**When to pick which:** Use eager learners when prediction speed matters (real-time systems, deployed models) and the training set is static. Use lazy learners when the data changes frequently, training time must be minimal, or local explanations ("similar past cases") are valuable.

13.5.3 Assumptions and Scope

**Scope:** The eager/lazy distinction applies to all supervised learning algorithms, not just Naive Bayes. It is a property of the algorithm's *computation schedule*, not its accuracy. Some algorithms blur the line: decision trees are eager (they build the tree during training), but k-d trees for KNN precompute a search structure (making KNN partially eager).

13.5.4 Pitfalls

**Assuming eager = better:** Eager and lazy have different strengths. Eager models are compact and fast at prediction but cannot adapt to new data without retraining. Lazy models adapt instantly but are slow at scale.
**Forgetting that eager models need retraining:** If the data distribution changes (concept drift), an eager Naive Bayes model becomes stale. A lazy KNN model naturally tracks the latest data.

13.5.5 Recap and Bridge

Naive Bayes is an eager learner — it front-loads computation into the training phase so that predictions are near-instant. This completes our deep dive into Naive Bayes. Next: we pivot to **Ensemble Learning** — combining multiple models (eager or lazy) to outperform any single one (Section 13.6).

13.5.6 Real-World & Domain Connection

The eager/lazy trade-off directly impacts system design. Google's search ranking models are eager (trained offline, served fast). Netflix's "Because you watched..." recommendations blend eager (precomputed embeddings) and lazy (real-time nearest-neighbor lookup) approaches. Understanding this distinction helps engineers choose the right architecture for latency budgets and data freshness requirements.

13.6 Introduction to Ensemble Learning

13.6.1 What "Ensemble" Means

**Hook:** Why would you trust a crowd of amateurs over a single expert? Because the crowd's mistakes cancel out while the expert's single mistake is fatal. This is the counterintuitive premise of ensemble learning.

**Intuition + Analogy:** The word *ensemble* (pronounced "on-som-ble") means "together" or "at the same time." It comes from musical ensembles — an orchestra where multiple instruments play together. A single violin is beautiful, but combine it with drums, trumpets, and cellos, and you get a richer, more powerful sound. In machine learning: instead of relying on one model, you combine many models to get a better, more reliable result.

*Ensemble learning* is the strategy of training multiple models (called *base learners* or *weak learners*) and combining their predictions to produce a final output that outperforms any individual model. The ensemble uses the principle that **errors from different models tend to cancel out** when aggregated, while correct predictions reinforce each other.

13.6.2 The Board of Doctors Analogy

Imagine you are sick. You have two choices:

Go to **one expert doctor** and take their opinion. Even an expert can make a mistake.
Go to a **board of five doctors** — a cardiologist, a neurologist, a general practitioner, a dermatologist, a nutritionist. Each examines you from their own perspective. Based on their individual analyses, they collectively diagnose your condition.

The core idea: even if individual doctors in the group are not all specialists, **the collective opinion tends to be more accurate.** One doctor may miss something that another catches. One person's blind spot is covered by someone else's expertise. In ensemble learning, the "doctors" are base models with different inductive biases.

13.6.3 The "Who Wants to Be a Millionaire" Analogy

**Phone-a-Friend** = single model philosophy. You call one friend who is an expert in that domain. If that one expert is wrong, you lose.
**Audience Poll** = ensemble model. Hundreds of people vote. Individually, many may not know the answer, but collectively the crowd often points to the right choice.

Even though some audience members know nothing about the domain, enough people have partial knowledge that the aggregate opinion gives a high probability of being correct. The ensemble's strength comes from *diversity*, not individual expertise.

13.6.4 Single Model Philosophy — Occam's Razor

*Occam's razor* states: **Do not multiply entities beyond necessity.** If you have two explanations for the same data, the simpler one is usually correct.

In machine learning, Occam's razor translates to **preventing overfitting.** If two models explain the same data equally well, always pick the simpler one:

A *simple model* captures the underlying pattern.
A *complex model* tends to memorize the noise — it overfits.

Think of fitting a curve to data points: a straight line may miss some points but captures the general trend. A squiggly line that passes through every point perfectly (100% training fit) is memorizing noise, not learning the pattern. Infinitely many models can explain any given dataset — the art is picking the simplest one that still explains the data well.

**The tension with ensembles:** Ensemble methods seem to violate Occam's razor — they combine many models, increasing complexity. But they work because diversity among simple models can capture complexity without any single model overfitting. This is a fundamental insight: aggregated simplicity can express complexity.

13.6.5 The No Free Lunch Theorem

**There is no single algorithm that is universally the best for all problems.** This is the *No Free Lunch Theorem* (Wolpert & Macready, 1997):

A neural network works best on image data.
A random forest might work best on tabular data.
A linear model may outperform both on tiny, clean datasets.

You cannot know in advance which single model is perfect for a specific problem. Relying on one type of model is risky because you do not yet know which model will accurately capture the patterns in your data. Ensemble methods sidestep this by combining multiple model types — hedging against the risk of picking the wrong one.

13.6.6 Why Every Algorithm Makes Assumptions

Every algorithm makes a set of *inductive assumptions* about the data. When those assumptions do not match reality, the model fails:

**Linear regression** assumes a linear relationship between features and target.
**Decision trees** assume the data can be split into axis-aligned rectangular regions.
**Naive Bayes** assumes conditional independence of features.

Because of these assumptions, a specific model can fail when its assumptions are violated. Rather than betting everything on one model (and one set of assumptions), ensemble methods bring together a **committee of models** — combinations that cover each other's weaknesses. If you have 10 data samples, Model A may misclassify #5 and #10, but Model B may correctly classify those two. By combining both, you may correctly predict all 10.

13.6.7 Visual Intuition

Picture a decision boundary plot with two classes (red and blue dots). A single decision tree draws a jagged, axis-aligned boundary that misclassifies a few points on each side. A single linear model draws a straight line that also misclassifies some points. The ensemble combines both boundaries: where they agree, the region is solidly colored; where they disagree, the ensemble takes a vote. The resulting boundary is smoother and correctly classifies more points than either individual model. This is the visual essence of ensembling — individual rough edges get sanded down by aggregation.

13.6.8 Pitfalls

**Confusing ensemble with "just train more models":** An ensemble only works if the base models are *diverse* (Section 13.8). Training 100 identical decision trees on the same data produces 100 identical predictions — no benefit.
**Overlooking computational cost:** Training L models and running all L at prediction time multiplies compute by L. Ensembles trade compute for accuracy — ensure the trade-off is worthwhile.
**Assuming ensemble always beats single model:** If base models are highly correlated (all make the same mistakes), the ensemble provides no benefit. The key requirement is diversity, not quantity.

13.6.9 Recap and Bridge

Ensemble learning combines multiple diverse models to produce a prediction stronger than any individual model — just as a board of doctors outperforms a single physician. Next: the formal architecture — **base learners, weights, and the combiner function** (Section 13.7).

13.6.10 Real-World & Domain Connection

Ensemble methods dominate machine learning competitions (Kaggle) and production systems. Netflix's recommendation engine uses ensembles of collaborative filtering models. Financial fraud detection systems combine rule-based models, logistic regression, and gradient-boosted trees. The random forest algorithm (an ensemble of decision trees) is one of the most widely deployed ML algorithms in industry for tabular data. Ensemble methods are the go-to approach when accuracy and reliability matter more than model simplicity or training time.

13.7 Base Learners, Weights, and the Combiner

13.7.1 The Ensemble Architecture

**Hook:** If you build a committee, three questions define it: who sits on the committee, how much does each member's vote count, and how do you tally the votes? Ensemble learning answers all three.

The architecture of an ensemble model follows a universal pattern:

$Y = F (j = 1 \sum L W_{j} \cdot D_{j} (X))$

where:

$X \in R^{d}$ — the raw input (e.g., a patient's symptoms as a feature vector)
$D_{1}, D_{2}, \dots, D_{L}$ — the $L$ **base learners** (also called *weak learners* or *committee members*). Each $D_{j}$ is an individual model that takes the same input $X$ and produces a prediction $D_{j} (X)$ .
$W_{1}, W_{2}, \dots, W_{L}$ — the **weights** assigned to each base learner, satisfying $\sum_{j = 1}^{L} W_{j} = 1$ . Not all learners are equally trusted.
$F$ — the **combiner function** (aggregator). It takes the weighted sum of individual predictions and produces the final output.

The same input $X$ is fed into every member of the committee. Each member produces its own prediction independently. The combiner aggregates them into one final output. Different ensemble algorithms (bagging, boosting, random forest) differ in *how* they create diversity among $D_{j}$ and *how* they set $W_{j}$ and $F$ .

13.7.2 What Are the Base Learners $D_{j}$ ?

The base learners can be any models. The key requirement is that they are **different from each other** in some meaningful way:

$D_{1}$ — a decision tree
$D_{2}$ — a KNN classifier
$D_{3}$ — a logistic regression
$D_{4}$ — a linear regression
$D_{5}$ — a neural network
$D_{6}$ — another decision tree with different hyperparameters (e.g., different max depth)
$D_{7}$ — another logistic regression with different weight initialization

They can be entirely different model types (*heterogeneous ensemble*) or the same model type with different configurations (*homogeneous ensemble*). Random forest uses homogeneous base learners (all decision trees). Stacking often uses heterogeneous base learners.

**Weak learners:** In boosting, base learners are deliberately simple — often *decision stumps* (trees with a single split) that are only slightly better than random guessing (e.g., 51% accuracy). The boosting algorithm iteratively improves them. The term "weak learner" refers to this simplicity, not to poor final performance — the ensemble as a whole can be arbitrarily strong.

13.7.3 The Weights $W_{j}$

**Intuition:** Not all committee members are equal. If you have chest pain, the cardiologist's opinion should count more than the dermatologist's. Ensemble weights encode this differential trust.

If $D_{1}$ is known to be 80% accurate on held-out validation data, it gets a high weight $W_{1}$ .
If $D_{2}$ is only 51% accurate (barely above chance), it gets a lower weight $W_{2}$ .

The weights must satisfy $\sum_{j = 1}^{L} W_{j} = 1$ (a convex combination). How the weights are determined depends on the ensemble algorithm:

**Bagging:** All weights are equal: $W_{j} = 1/ L$ (democratic).
**Boosting (AdaBoost):** Weights are proportional to each learner's accuracy: $α_{m} = \frac{1}{2} ln (\frac{1 - ϵ _{m}}{ϵ _{m}})$ , where $ϵ_{m}$ is the weighted error of learner $m$ .
**Weighted averaging:** Weights can be learned from validation performance or set by cross-validation.

13.7.4 The Combiner Function $F$

The combiner is the mechanism that aggregates individual predictions into one final output:

$Y = F (W_{1} \cdot D_{1} (X) + W_{2} \cdot D_{2} (X) + \dots + W_{L} \cdot D_{L} (X))$

For **classification**, the most common combiner is **majority voting** (hard voting) or **averaged probabilities** (soft voting). For **regression**, the combiner is typically a simple (weighted) average.

Different ensemble algorithms use different $F$ :

**Bagging:** Equal-weight average or majority vote.
**AdaBoost:** Weighted majority vote with weights $α_{m}$ .
**Stacking:** A meta-learner (e.g., logistic regression) is trained to learn the optimal $F$ from the base learners' outputs.

13.7.5 Visual Intuition

Picture a diagram: a single input $X$ branches into $L$ parallel boxes, each labeled $D_{1}, D_{2}, \dots, D_{L}$ . Arrows from all boxes converge into a single node labeled $F$ , with each arrow annotated by its weight $W_{j}$ . The output $Y$ emerges from $F$ . This is the universal ensemble template — the differences between bagging, boosting, and stacking are entirely in how $D_{j}$ are trained, how $W_{j}$ are set, and what $F$ does.

13.7.6 Pitfalls

**Using identical base learners:** If all $D_{j}$ are trained on the same data with the same algorithm and parameters, they produce identical predictions. The ensemble collapses to a single model — zero benefit.
**Assigning negative or unnormalized weights:** Weights must be non-negative and sum to 1 for a valid convex combination. Unnormalized weights can produce outputs outside the valid range.
**Confusing weights with model parameters:** $W_{j}$ are *ensemble-level* weights, not the internal parameters (e.g., neural network weights) of each base learner.

13.7.7 Recap and Bridge

Every ensemble = base learners $D_{j}$ + weights $W_{j}$ + combiner $F$ . The same input flows through all learners; their weighted outputs are aggregated. Next: **how to make base learners diverse**, because without diversity, an ensemble is just a single model copied $L$ times (Section 13.8).

13.7.8 Real-World & Domain Connection

The three-component architecture (learners, weights, combiner) is the blueprint behind every production ensemble system. Google's ranking ensembles combine hundreds of specialized models with learned weights. Kaggle-winning solutions routinely stack gradient-boosted trees, neural networks, and linear models with a logistic regression meta-learner as $F$ . Understanding this decomposition helps you design ensembles systematically: choose $D_{j}$ for coverage, set $W_{j}$ for trust, and pick $F$ for the right aggregation trade-off.

13.8 Diversity in Ensemble Learning

13.8.1 Why Diversity Matters

**Hook:** What is worse than one wrong doctor? Five doctors who all make the *same* wrong diagnosis. An ensemble without diversity is just a single model multiplied — and all errors are multiplied too.

**Intuition + Analogy:** If all five doctors on your board are cardiologists with identical training, and you walk in with a skin disease, the diagnosis will almost certainly be wrong — they all share the same blind spot. Replace two cardiologists with a dermatologist and a general practitioner, and suddenly someone catches what the others missed.

In machine learning: if all base learners make the *same* mistakes, the committee adds nothing. You need members with **different biases** so that their errors are *uncorrelated* and cancel out when aggregated. $D_{1}$ should be wrong where $D_{2}$ is right, and vice versa.

From the bias-variance perspective (Section 5.6.3 of the textbook): averaging reduces variance only when the individual model errors are uncorrelated. If errors are perfectly correlated ( $ρ = 1$ ), the ensemble variance equals the individual variance — no benefit.

13.8.2 Four Approaches to Create Diversity

There are four standard approaches to make base learners diverse:

**Approach 1 — Use Different Algorithms (Heterogeneous Ensemble)**

Use entirely different model types:

One decision tree
One neural network
One linear regression
One logistic regression
One KNN

*Why this works:* different models make different inductive assumptions about the data. Linear models assume linearity; decision trees assume axis-aligned rectangular splits; neural networks learn hierarchical features. By mixing model families, you cover different assumption spaces simultaneously. This is called a *heterogeneous ensemble*.

**Approach 2 — Use Different Hyperparameters (Homogeneous Ensemble)**

Use the same algorithm with different hyperparameter values:

Decision tree with depth=3, depth=5, depth=10
Neural network with 1 hidden layer, 2 hidden layers, 3 hidden layers
KNN with K=3, K=5, K=11

Same model type ( $M_{1}$ ), different configurations ( $H_{1}, H_{2}, H_{3}$ ). This is a *homogeneous ensemble* with hyperparameter diversity.

**Approach 3 — Use Different Input Representations (Multiview Ensemble)**

Feed different representations of the same input to different models:

For a video: one model analyzes audio, another analyzes visual frames
For a document: one model processes embedded images, another processes text
For tabular data: one model uses raw features, another uses polynomial feature expansions, another uses PCA-reduced features

Each base learner sees a different "view" of the same underlying data — especially powerful in multimodal problems.

**Approach 4 — Use Different Training Sets (Data Diversity)**

Draw different samples from the training data for each base learner. This is the foundation of **bagging** (Bootstrap Aggregating):

Random sampling with replacement (bootstrapping) — each learner gets a different random subset
Partitioning by input space — one model learns from young patients (age < 20), another from middle-aged (20–40), another from older (40–60), another from seniors (60+). This creates a *mixture of experts*.

Random forest combines Approaches 2 and 4: different bootstrap samples AND random feature subsets per tree.

13.8.3 Student Question

**Q:** Is the different training sets approach similar to cross-validation?

**A:** Yes — the underlying sampling concept is the same. *Bootstrapping* (random sampling with replacement) creates different data subsets. These bootstrap samples are what feed each classifier. Because each classifier trains on different data, $C_{1}$ learns different decision boundaries than $C_{2}$ , which learns different boundaries than $C_{L}$ . This is exactly the mechanism that makes bagging work. Cross-validation also partitions data, but for a different purpose: model evaluation rather than ensemble diversity.

13.8.4 Assumptions and Scope

**Key requirement for diversity to help:** Individual model errors must be at least partially *uncorrelated*. If all models make identical errors, the ensemble is useless regardless of how many learners you have.

**Mathematical justification (from bias-variance):** For regression with $L$ models having equal variance $σ^{2}$ and pairwise error correlation $ρ$ : $Var (ensemble) = ρ σ^{2} + \frac{1 - ρ}{L} σ^{2}$ When $ρ = 1$ (perfectly correlated errors): variance = $σ^{2}$ (no benefit). When $ρ = 0$ (uncorrelated errors): variance = $σ^{2} / L$ (linear reduction with $L$ ).

13.8.5 Pitfalls

**Training all learners on identical data with identical settings:** Produces zero diversity. The ensemble degenerates to a single model.
**Adding correlated learners:** If you add a 6th decision tree identical to the first 5, it adds no new information but costs extra compute. Diversity must be *effective*, not just nominal.
**One bad apple:** If one base learner has very poor accuracy (worse than random), it can drag down the ensemble performance. In boosting, learners are weighted, but in simple averaging, a bad model contributes equally.

13.8.6 Recap and Bridge

Diversity is the engine of ensemble learning — uncorrelated errors cancel out, correlated errors survive. Four approaches create diversity: different algorithms, different hyperparameters, different input views, and different training sets. Next: **how to combine diverse predictions** into a single final answer (Section 13.9).

13.8.7 Real-World & Domain Connection

Diversity engineering is a central concern in production ML systems. Netflix's recommendation ensemble combines collaborative filtering (matrix factorization), content-based models (using movie metadata), and deep learning models (using viewing sequences) — three completely different algorithmic families. Each catches patterns the others miss. In financial fraud detection, rule-based models (capturing known fraud patterns) are ensembled with gradient-boosted trees (capturing subtle statistical anomalies) — combining domain expertise with data-driven learning.

13.9 Combining Results

13.9.1 The Second Challenge

**Hook:** You have five models giving five different predictions. Do you take the average? The majority vote? The most cautious answer? The choice of combination rule can flip the final prediction — it is as consequential as the models themselves.

The second major challenge in ensemble learning: **how do you combine the predictions of individual learners?** Different ensemble algorithms differ in (a) how they create diversity and (b) how they combine results. The combination scheme determines whether the ensemble is democratic (equal votes) or meritocratic (weighted by trust).

13.9.2 Combining Schemes

**Average (Simple Average):**

$Y = \frac{1}{L} j = 1 \sum L D_{j} (X)$

All learners treated equally. Every model gets the same vote.

**When to use:** The default choice. Individual models already have reasonable accuracy, but data contains random noise. Averaging cancels out uncorrelated noise across models. From the bias-variance perspective, averaging reduces variance by a factor up to $L$ when errors are uncorrelated.

**Weighted Sum:**

$Y = j = 1 \sum L W_{j} \cdot D_{j} (X), j = 1 \sum L W_{j} = 1$

Each learner weighted by trustworthiness. A learner with 80% accuracy gets more weight than one with 51%.

**When to use:** When you have a clear accuracy ranking among base learners. Give more weight to better models. This is the scheme used by AdaBoost (weights are learned from training errors).

**Median:**

Take the median of all individual predictions: $Y = median {D_{1} (X), D_{2} (X), \dots, D_{L} (X)}$ .

**When to use:** When some models occasionally produce extreme outlier predictions. Example: $D_{1} \to 0.2$ , $D_{2} \to 0.4$ , $D_{3} \to 0.99$ (broken model). The average would be skewed to ~0.53 by the outlier. The median stays at 0.4, ignoring the crazy value. This is the same reason the median is preferred over the mean for a strong central tendency — it has a *breakdown point* of 50% (half the models can go wild without affecting the result).

**Minimum:**

$Y = min {D_{1} (X), \dots, D_{L} (X)}$ .

**Maximum:**

$Y = max {D_{1} (X), \dots, D_{L} (X)}$ .

**Product:**

$Y = j = 1 \prod L D_{j} (X)$

Multiply all predictions. Every model has equal veto power — if any model predicts near 0, the result collapses.

**When to use the product rule:** When it is critical to avoid a particular class and you want every model to have veto power. If even one model says "this cannot be Class A" (predicts near 0), that opinion is respected. The product is a *consensus* method — all must agree for high confidence. The downside is the same zero-probability issue from Naive Bayes: one zero kills everything.

13.9.3 Worked Example — Combining Three Learners

Three base learners $D_{1}, D_{2}, D_{3}$ make predictions for Class $C_{2}$ :

Learner	Prediction for $C_{2}$
$D_{1}$	0.5
$D_{2}$	0.6
$D_{3}$	0.4

Applying each combination scheme:

$Average: Median: Minimum: Maximum: Product: (0.5 + 0.6 + 0.4) /3 = 0.50 0.50 0.40 0.60 0.5 \times 0.6 \times 0.4 = 0.12$

Notice: the product rule produces a much smaller number (0.12) compared to max (0.60). The product is always ≤ the minimum, making it the most conservative aggregator. If any model had predicted 0.01, the product would be ~0.0012 — effectively a veto.

**Sense-check:** All models agree $C_{2}$ is plausible (all ≥ 0.4). Any reasonable combination would predict $C_{2}$ , but the product gives a much lower confidence, reflecting the fact that no single model was highly confident (>0.6). This illustrates how the combination rule shapes not just the decision but the confidence level.

13.9.4 Summary: When to Use Which Scheme

Scheme	Use when...
Average	Default choice. All models are reasonably good; you want to average out uncorrelated noise.
Weighted Sum	You have a clear accuracy ranking — some models consistently outperform others.
Median	Some models occasionally produce outlier/hallucinated predictions. Robust to breakdowns.
Product	Veto power is needed — every model must agree. Conservative, cautious predictions.

13.9.5 Comparison — Average vs. Weighted vs. Median

Dimension	Average	Weighted Sum	Median
Sensitivity to bad models	Moderate (one bad model pulls average)	Low if bad model has low weight	Very low (up to 50% breakdown)
Requires validation	No	Yes (to set weights)	No
Handles correlated errors	No better than single model	No better than single model	No better than single model
Computational cost	O(L)	O(L)	O(L log L) (requires sorting)

13.9.6 Pitfalls

**Using product rule without considering the zero problem:** One model predicting exactly 0 (e.g., from a hard-classifier output) makes the entire ensemble output 0.
**Confusing soft and hard voting:** If base classifiers output class labels (not probabilities), the product and average of labels are meaningless. Use majority vote instead.
**Averaging probabilities from miscalibrated models:** If one model's probabilities are poorly calibrated (e.g., always outputs 0.99 or 0.01), averaging can be skewed. Consider calibrating individual models first.

13.9.7 Recap and Bridge

How you combine predictions matters as much as the predictions themselves. Average for noise reduction, weighted sum for trust, median for reliability, product for consensus. Next: **bootstrapping and bagging** — the data-diversity approach that makes the averaging scheme work in practice (Section 13.10).

13.9.8 Real-World & Domain Connection

The combination scheme choice has real consequences. In medical diagnosis ensembles, the product rule (conservative) is preferred — you do not want to miss a cancer diagnosis because one model was uncertain. In spam filtering, the average or majority vote is fine — occasional misclassification has low cost. In autonomous driving perception systems, the median is used to reject sensor outliers (a malfunctioning camera should not override three functioning ones).

13.10 General Ensemble Approach — Bootstrapping and Bagging

13.10.1 The General Pipeline

**Hook:** How do you turn one dataset into many — each different enough to train a different model, yet each still representative of the original? The answer is as simple as drawing names from a hat, putting each name back after you draw it.

**Purpose:** The standard ensemble workflow transforms a single training dataset into multiple diverse training sets, trains a model on each, and aggregates their predictions.

**Procedure:**

**Start with original training data** — $N$ records, e.g., 1000.
**Create $L$ bootstrap samples** — For each learner $j = 1, \dots, L$ , draw $N$ samples *with replacement* from the original data. Each bootstrap sample $D_{j}$ contains $N$ records, but with duplicates and omissions.
**Train a separate model $C_{j}$ ** on each bootstrap sample $D_{j}$ . Because each model sees different data (different duplicates, different omissions), $C_{1}$ learns different patterns than $C_{2}$ , which differs from $C_{L}$ .
**Combine the classifiers** into the final ensemble $C^{*}$ using a chosen combination scheme (Section 13.9).

This is like giving different, overlapping chapters of a textbook to different students — Student 1 gets chapters 1-5 (with some chapters repeated, some missing), Student 2 gets a different mix. Each student specializes in their assigned material, and together they cover the whole book.

13.10.2 Bootstrapping

*Bootstrapping* is random sampling **with replacement**. From a dataset of $N$ records, you draw $N$ samples with replacement:

Some records appear multiple times (duplicates).
Some records never appear (omissions).

The probability that a specific record is selected at least once in a bootstrap sample of size $N$ :

$P (selected) = 1 - (1 - \frac{1}{N})^{N} \approx 1 - e^{- 1} \approx 0.632$

So each bootstrap sample contains about **63.2%** of the original records (with repetitions making up the remaining 36.8%). The ~37% of records left out are called the *out-of-bag* (OOB) samples — they provide a free validation set for estimating the ensemble's generalization error.

Bootstrapping is the same concept used in cross-validation but applied here for diversity creation rather than evaluation. Each bootstrap sample feeds one base learner, creating the data diversity that makes bagging work.

13.10.3 Bagging

*Bagging* = **Bootstrap Aggregating** (Breiman, 1996).

It combines:

**Bootstrapping** — Create $L$ diverse training sets by sampling with replacement.
**Aggregating** — Combine the $L$ trained models' predictions (typically by averaging for regression or majority voting for classification).

**Bagging algorithm (formal):**

For $m = 1$ to $L$ :

Draw a bootstrap sample $D_{m}$ of size $N$ from the original data.
Train a base learner $C_{m}$ on $D_{m}$ .

Prediction: $C^{*} (x) = ar g max_{y} \sum_{m = 1}^{L} 1 [C_{m} (x) = y]$ (classification, majority vote).

**Why bagging works:** From the bias-variance decomposition, bagging primarily **reduces variance** — the sensitivity of the model to the specific training sample. The bias remains similar to that of a single model trained on the full dataset. Bagging is most effective with **unstable learners** — models whose predictions change substantially with small changes in the training data (e.g., deep decision trees, neural networks). For stable learners (e.g., linear regression, KNN with large K), bagging provides little benefit because bootstrap samples produce nearly identical models.

**Random Forest** is bagging applied to decision trees, with an extra twist: at each split, only a random subset of features is considered (Approach 2 diversity). This decorrelates the trees further, reducing variance beyond what bagging alone achieves.

13.10.4 What Comes Next

The concepts covered — base learners, weights, diversity approaches, combination schemes, bootstrapping, and bagging — are the foundation for the specific ensemble algorithms in the next session:

**Bagging** (Bootstrap Aggregating) — today's focus
**Random Forest** — bagging + random feature subsets on decision trees
**Boosting** — sequential training where each model focuses on previous models' mistakes
**AdaBoost** — adaptive boosting with weighted samples and weighted voting
**XGBoost** — gradient boosting, one of the most successful ML algorithms in practice

Each differs in (a) how it creates diversity and (b) how it combines predictions. The underlying ensemble architecture $(D_{j}, W_{j}, F)$ remains the same.

13.10.5 When to Use / Alternatives

**Use bagging when:**

Your base learner is unstable (decision trees, neural networks without strong regularization).
You want to reduce variance without increasing bias.
You have a single, static dataset and need to create diversity from it.

**Alternatives:**

**Boosting** (AdaBoost, XGBoost): Reduces both bias and variance by sequentially focusing on hard examples. Generally outperforms bagging on clean data but is more prone to overfitting on noisy data.
**Random Forest:** Bagging + random feature subsets. Almost always preferred over plain bagging with decision trees — it achieves lower variance through extra decorrelation.
**Single model with regularization:** If your model is already low-variance (e.g., heavily regularized linear model), bagging adds unnecessary complexity.

13.10.6 Visual Intuition

Picture a scatter plot with the original data in black. Three bootstrap samples are overlaid in different colors — each one is roughly 63% of the black points, with some points appearing 2-3 times (larger markers) and some missing entirely. A decision boundary is drawn through each colored set. The three boundaries are similar but differ in the details — one bends left at a region where a particular point was duplicated, another bends right where that point was omitted. The bagged boundary (the average) sits in the middle, smoother than any individual boundary. This is variance reduction in action.

13.10.7 Pitfalls

**Bagging a stable learner:** If your base model barely changes with different training data (e.g., 1-nearest-neighbor with large N, or a heavily regularized linear model), bootstrap samples produce nearly identical models. Bagging adds compute with zero accuracy gain.
**Using too few bootstrap samples:** With $L$ too small (e.g., 5), the ensemble does not benefit from the law of large numbers. Typical values: $L = 50$ to $500$ for bagging.
**Ignoring out-of-bag (OOB) error:** The ~37% of data omitted from each bootstrap sample provides a free validation metric. Use OOB error instead of a separate validation set to tune $L$ .

13.10.8 Recap and Bridge

Bagging = bootstrap sampling for diversity + averaging for variance reduction. It is the simplest ensemble method and the foundation for random forest. Each bootstrap sample contains ~63% of the data; the omitted 37% provides free validation (OOB error). Next session: boosting, random forest, AdaBoost, and XGBoost.

13.10.9 Real-World & Domain Connection

Bagging and its random forest variant are go-to techniques for tabular data problems in industry: credit scoring, customer churn prediction, fraud detection, and medical diagnosis all frequently use bagged ensembles. The technique is particularly valuable in high-stakes domains where a single decision tree's high variance is unacceptable — bagging smooths out the jagged boundaries into stable, reliable predictions. The 63.2% bootstrap coverage is a number worth remembering: it appears in interview questions, and the OOB error it enables is a practical alternative to cross-validation when training many models.

Exam Guidance Summary

**Exam note:** Naive Bayes problems are described as **"clear cut"** with **"fixed marks"** — you can reliably score well on them by practicing the step-by-step procedure.

The exam is **open book** — you do not need to memorize the PDF formulas for probability distributions (Gaussian PDF, etc.). You can refer to the textbook. What matters is understanding the concept and being able to apply the formula.
Expect **numerical problems** on both Multinomial Naive Bayes and Gaussian Naive Bayes. Practice computing:
Prior probabilities $P (Y = y)$ from class counts
Likelihoods $P (X_{i} = a ∣ Y = y)$ from frequency tables (multinomial)
Means $μ_{i, k}$ and variances $σ_{i, k}^{2}$ from continuous data (Gaussian)
Gaussian PDF evaluation: plugging numbers into $\frac{1}{2 π σ ^{2}} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})$
Final posterior computation and argmax prediction
**Laplace smoothing** is applied by default. Know the formula $\frac{n _{i c} + 1}{n _{c} + V}$ and when to use it. Be prepared to compute $V$ correctly per feature.
**Zero probability issue:** Understand both cases (single class zeroed vs. both classes zeroed). Know why smoothing is essential.
For **ensemble learning**: the current session covers concepts (base learners, weights, diversity, combination schemes, bootstrapping, bagging). The specific algorithms (random forest, boosting, AdaBoost, XGBoost) are covered in the next session. Focus on conceptual understanding for now.
The **m-estimate** variant $\frac{n _{i c} + m \cdot p}{n _{c} + m}$ is mentioned for awareness but is not the primary exam focus — Laplace smoothing (the $m = V, p = 1/ V$ special case) is the exam focus.
Additional practice problems are available — practice them. Numerical Naive Bayes is described as a reliable source of marks.

Key Industry Applications

**Naive Bayes in text classification:** Spam detection (Gmail, Outlook classify emails as spam/not-spam based on word presence), document categorization (news articles → sports/politics/tech), sentiment analysis (product reviews → positive/negative). Naive Bayes remains popular because it trains fast, requires little data, and its probability outputs are interpretable.
**Bag of words:** A foundational NLP technique where documents are represented by word frequencies, ignoring word order. Combined with Naive Bayes, it powers most baseline text classifiers. Modern extensions include TF-IDF weighting (down-weighting common words like "the") and n-gram features (capturing short phrases).
**Laplace smoothing in practice:** Enabled by default in scikit-learn's `MultinomialNB` (`alpha=1.0`) and `BernoulliNB`. In `GaussianNB`, a variance smoothing parameter (`var_smoothing=1e-9`) prevents division by zero. Smoothing is almost never disabled in production — the small bias is negligible compared to the risk of zero probabilities.
**Gaussian Naive Bayes:** Used in medical diagnosis (continuous features: blood pressure, age, lab values), sensor fault detection (vibration/temperature readings), and financial risk models (income, debt ratios). Available as `GaussianNB` in scikit-learn.
**Ensemble methods in practice:** Random Forest and XGBoost are among the most widely deployed ML algorithms for tabular data. Random Forest (scikit-learn's `RandomForestClassifier`) is a strong default for classification. XGBoost, LightGBM, and CatBoost dominate Kaggle competitions and production systems where accuracy is key.
**Bagging:** The bagging meta-estimator in scikit-learn (`BaggingClassifier`) wraps any base estimator. Useful for reducing variance of unstable models.
**Python implementation:** All discussed algorithms are available in scikit-learn. Code demonstrations for individual ensemble methods are provided in the course materials.

ML Lecture 13 notes · Naive Bayes Classifier and Ensemble Learning

Machine Learning· postgraduate· 2026-06-29

Sections Breakdown

113.1 Review of Multinomial Naive Bayes

Core assumption, symbol registry, four-step procedure, worked example with Play Tennis dataset, denominator explanation, student questions, pitfalls, and domain connections.

213.2 Gaussian Naive Bayes

Continuous features, bell curve fitting, Gaussian PDF likelihood, five-step procedure, worked example with student pass/fail prediction, student questions, and pitfalls.

313.3 Zero Probability Issue

Single unseen feature-class pair zeroing the posterior, single class penalized vs both classes zeroed, why it happens, and why Laplace smoothing is essential.

413.4 Laplace Smoothing

Add-one smoothing formula, worked tax evasion example, m-estimate variant, text classification, spam example, student Q&A, and pitfalls.

513.5 Naive Bayes as an Eager Learner

Eager vs lazy learning paradigms, comparison table, when to use each, pitfalls, and real-world trade-offs.

613.6 Introduction to Ensemble Learning

Board of doctors analogy, Millionaire analogy, Occam's razor, No Free Lunch Theorem, inductive assumptions, and diversity motivation.

713.7 Base Learners, Weights, and the Combiner

Universal ensemble architecture, heterogeneous vs homogeneous base learners, weights (equal, AdaBoost, learned), combiner functions (majority vote, average, stacking).

813.8 Diversity in Ensemble Learning

Why uncorrelated errors cancel, four approaches to create diversity (different algorithms, hyperparameters, input views, training sets), bias-variance justification.

913.9 Combining Results

Average, weighted sum, median, min/max, product combination schemes with worked three-learner example and comparison table.

1013.10 Bootstrapping and Bagging

General ensemble pipeline, bootstrapping with replacement (63.2% coverage), out-of-bag error, bagging algorithm, bias-variance connection, random forest preview.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Multinomial Naive Bayes

Must-know: Naive Bayes multiplies independent feature likelihoods with a class prior. The classification rule is the MAP decision: y-hat = argmax P(Y) * product of P(X_i | Y). Memorize the four-step procedure.

$y^= ar g y max P (Y = y) i = 1 \prod n P (X_{i} = x_{i} ∣ Y = y)$

Top pitfall: Forgetting that the denominator P(X) cancels out during argmax comparison — computing it wastes time.

Self-check: Why does the denominator P(X) not affect which class is predicted?

Connects to: Gaussian Naive Bayes (Section 13.2), Laplace Smoothing (Section 13.4), Maximum A Posteriori Estimation

Gaussian Naive Bayes

Must-know: For continuous features, replace counting with Gaussian PDF. Compute mean and variance per feature per class, then plug into the PDF: 1/sqrt(2*pi*sigma^2) * exp(-(x-mu)^2/(2*sigma^2)).

$p (X_{i} = x ∣ Y = k) = \frac{1}{2 π σ _{i, k}^{2}} exp (- \frac{( x - μ _{i, k} ) ^{2}}{2 σ _{i, k}^{2}})$

Top pitfall: Confusing density with probability. PDF values can exceed 1 — only integrals over intervals give probabilities.

Self-check: When variance is zero for a class (all samples identical), what happens to the Gaussian PDF? How should it be fixed?

Connects to: Multinomial Naive Bayes (Section 13.1), Maximum Likelihood Estimation

Zero Probability Issue

Must-know: A single unseen feature-class pair makes the entire posterior zero because the posterior is a product. This causes either one class to be zeroed (Case 1) or all classes to be zeroed (Case 2 — complete failure).

$P (Y = y ∣ X) \propto P (Y = y) \times i = 1 \prod n P (X_{i} = x_{i} ∣ Y = y)$

Top pitfall: Deleting training records without rechecking probability tables can introduce new zeros.

Self-check: Explain why Case 2 (both classes zeroed) is worse than Case 1 (one class zeroed).

Connects to: Laplace Smoothing (Section 13.4), Product Rule in Combining Results (Section 13.9)

Laplace Smoothing

Must-know: Add-one smoothing formula: P = (n_ic + 1) / (n_c + V), where V is the number of unique values of that specific feature. Apply per feature with correct V. Prevents zero probabilities.

$P (X_{i} = a ∣ Y = c) = \frac{n _{i c} + 1}{n _{c} + V}$

Top pitfall: Using total number of features as V instead of the number of unique values of that specific feature.

Self-check: For a feature with 3 unique values and a class with 10 samples, what is the smoothed probability of an unseen value?

Connects to: Zero Probability Issue (Section 13.3), M-Estimate (Section 13.4.3)

Eager vs Lazy Learning

Must-know: Eager learners (Naive Bayes, neural networks) build the model during training. Lazy learners (KNN) defer all computation to prediction time. Eager = fast prediction, slow training; Lazy = instant training, slow prediction.

N/A — conceptual distinction

Top pitfall: Assuming eager models are always better. They cannot adapt to new data without retraining, unlike lazy models.

Self-check: Why would you choose a lazy learner over an eager learner for a system where training data changes hourly?

Connects to: Comparison Table (Section 13.5.2), KNN (covered in earlier lectures)

Ensemble Learning — Core Concept

Must-know: Ensemble learning combines multiple diverse models to outperform any individual model. Errors from different models tend to cancel out while correct predictions reinforce each other. Diversity is the key requirement.

$Var (ensemble) = ρ σ^{2} + \frac{1 - ρ}{L} σ^{2}$

Top pitfall: Training identical models on identical data produces zero benefit — the ensemble degenerates to a single model.

Self-check: Why does Occam's Razor seem to conflict with ensemble methods, and how is this resolved?

Connects to: No Free Lunch Theorem (Section 13.6.5), Diversity (Section 13.8), Bagging (Section 13.10)

Ensemble Architecture — Learners, Weights, Combiner

Must-know: Every ensemble = base learners D_j + weights W_j + combiner F. Y = F(sum of W_j * D_j(X)). Base learners can be heterogeneous or homogeneous. Weights sum to 1. Combiner can be majority vote, average, or meta-learner.

$Y = F (j = 1 \sum L W_{j} \cdot D_{j} (X))$

Top pitfall: Confusing ensemble weights W_j with internal model parameters. W_j are ensemble-level trust weights, not the weights inside a neural network.

Self-check: In AdaBoost, how are the ensemble weights determined differently from bagging?

Connects to: Combining Results (Section 13.9), Bagging (Section 13.10), AdaBoost (next lecture)

Diversity in Ensemble Learning

Must-know: Four approaches to diversity: (1) different algorithms, (2) different hyperparameters, (3) different input representations, (4) different training sets (bootstrapping). Uncorrelated errors cancel; correlated errors survive.

$Var (ensemble) = ρ σ^{2} + \frac{1 - ρ}{L} σ^{2}$

Top pitfall: Adding more correlated learners adds compute cost with no accuracy benefit — diversity must be effective, not just nominal.

Self-check: Which of the four diversity approaches does Random Forest use, and how?

Connects to: Bias-Variance Decomposition, Bagging (Section 13.10), Random Forest (next lecture)

Combining Results

Must-know: Five combination schemes: average (default, reduces noise), weighted sum (trust-based), median (strong against outliers), min/max (bounding), product (consensus with veto power). Choice is as consequential as model selection.

$Average: Y = \frac{1}{L} j = 1 \sum L D_{j} (X); Product: Y = j = 1 \prod L D_{j} (X)$

Top pitfall: Using product rule without considering the zero problem — one zero prediction kills the entire ensemble output.

Self-check: When three models predict [0.2, 0.4, 0.99] for a class, which combination scheme is strongest and why?

Connects to: Zero Probability Issue (Section 13.3), Ensemble Architecture (Section 13.7), Bagging (Section 13.10)

Bootstrapping and Bagging

Must-know: Bootstrapping = random sampling with replacement, each sample covers ~63.2% of data. Bagging = bootstrap + aggregation. Primarily reduces variance. Works best with unstable learners (deep decision trees). OOB data provides free validation.

$P (selected) = 1 - (1 - \frac{1}{N})^{N} \approx 1 - e^{- 1} \approx 0.632$

Top pitfall: Bagging a stable learner (e.g., regularized linear model) adds compute with zero accuracy gain.

Self-check: What portion of training data is typically omitted from each bootstrap sample, and what is this omitted data called?

Connects to: Diversity (Section 13.8), Combining Results (Section 13.9), Random Forest (next lecture)

Previous LectureLecture 12 · Bayesian Learning — Maximum Likelihood, MAP, and Naive Bayes

Next LectureLecture 14 · Ensemble Methods — Bagging, Random Forest, and Boosting

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Naive Bayes Classifier and Ensemble Learning

13.1 Review of Multinomial Naive Bayes

13.1.1 Definition and Core Assumption

13.1.2 Symbol Registry — Multinomial Naive Bayes

13.1.3 The Four Steps

13.1.4 Worked Example — Play Tennis

13.1.5 Why the Denominator P(X) Is Ignored

13.1.6 Student Questions

13.1.7 Assumptions and Scope

13.1.8 Visual Intuition

13.1.9 Pitfalls

13.1.10 Recap and Bridge

13.1.11 Real-World & Domain Connection

13.2 Gaussian Naive Bayes

13.2.1 Definition and Core Assumption

13.2.2 Symbol Registry — Gaussian Naive Bayes

13.2.3 The Five Steps of Gaussian Naive Bayes

13.2.4 Worked Example — Student Pass/Fail Based on Study Hours

13.2.5 Student Questions

13.2.6 Assumptions and Scope

13.2.7 Visual Intuition

13.2.8 Pitfalls

13.2.9 Recap and Bridge

13.2.10 Real-World & Domain Connection

13.3 Zero Probability Issue

13.3.1 The Problem

13.3.2 Case 1 — Penalizing a Single Class

13.3.3 Case 2 — Both Classes Become Zero

13.3.4 Student Questions

13.3.5 Assumptions and Scope

13.3.6 Visual Intuition

13.3.7 Pitfalls

13.3.8 Recap and Bridge

13.3.9 Real-World & Domain Connection

13.4 Laplace Smoothing

13.4.1 Definition and Formula

13.4.2 Worked Example — Laplace Smoothing on Tax Evasion

13.4.3 M-Estimate (Advanced Variant)

13.4.4 Text Classification with Laplace Smoothing

13.4.5 Spam Classification Example

13.4.6 Student Questions

13.4.7 Assumptions and Scope

13.4.8 Visual Intuition

13.4.9 Pitfalls

13.4.10 Recap and Bridge

13.4.11 Real-World & Domain Connection

13.5 Naive Bayes as an Eager Learner

13.5.1 Eager vs. Lazy Learning

13.5.2 Comparison — Eager vs. Lazy

13.5.3 Assumptions and Scope

13.5.4 Pitfalls

13.5.5 Recap and Bridge

13.5.6 Real-World & Domain Connection

13.6 Introduction to Ensemble Learning

13.6.1 What "Ensemble" Means

13.6.2 The Board of Doctors Analogy

13.6.3 The "Who Wants to Be a Millionaire" Analogy

13.6.4 Single Model Philosophy — Occam's Razor

13.6.5 The No Free Lunch Theorem

13.6.6 Why Every Algorithm Makes Assumptions

13.6.7 Visual Intuition

13.6.8 Pitfalls

13.6.9 Recap and Bridge

13.6.10 Real-World & Domain Connection

13.7 Base Learners, Weights, and the Combiner

13.7.1 The Ensemble Architecture

13.7.2 What Are the Base Learners Dj​?

13.7.3 The Weights Wj​

13.7.4 The Combiner Function F

13.7.5 Visual Intuition

13.7.6 Pitfalls

13.7.7 Recap and Bridge

13.7.8 Real-World & Domain Connection

13.8 Diversity in Ensemble Learning

13.8.1 Why Diversity Matters

13.8.2 Four Approaches to Create Diversity

13.8.3 Student Question

13.8.4 Assumptions and Scope

13.1.5 Why the Denominator $P (X)$ Is Ignored

13.7.2 What Are the Base Learners $D_{j}$ ?

13.7.3 The Weights $W_{j}$

13.7.4 The Combiner Function $F$