Machine Learning

Bayesian Learning — Maximum Likelihood, MAP, and Naive Bayes

📅 Published: 2026-06-29

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

IID Data — Independent and Identically Distributed — covered in Lecture 1
Probability Foundations — covered in Lecture 11
Bayesian Learning — Introduction — covered in Lecture 11
Probability Distributions — covered in Lecture 11
Parameter Estimation — covered in Lecture 11

Bayesian Learning — Maximum Likelihood, MAP, and Naive Bayes

How do we pick the best model parameters from data? And what if we already have a hunch before seeing any data? This lecture answers both questions. We start with the difference between probability and likelihood — two words people use as synonyms but that mean opposite things in statistics. From there we build MLE, MAP, and the Naive Bayes classifier, ending with a clear picture of when to use each.

12.1 Probability vs Likelihood — Core Distinction

Hook. You flip a coin 10 times and get 8 heads. Is the coin fair? That question has two directions. You can ask: "If the coin is fair, how weird is 8 heads?" — that is probability. Or you can ask: "Given 8 heads, what's the most likely bias of the coin?" — that is likelihood. Same coin, same data, two opposite arrows of reasoning. Most beginners confuse them. This section draws the line firmly.

12.1.1 Intuition — The Direction of the Arrow

The analogy: a factory and its products. Imagine a factory with a known defect rate of 5%. Probability asks: in a batch of 100 items, how many defects do we expect? The factory (the model) is known; we predict the output. Likelihood turns this around: you open a box and find 12 defective items out of 100. What defect rate at the factory would make this observation most plausible? The data is fixed; you infer the hidden factory setting. Where the analogy breaks: the factory's defect rate is a physical fact you could measure directly. In statistics, the "true" parameter is forever hidden — we only ever estimate it. But the arrow direction is exactly right.

Probability is forward-looking: the model parameters are fixed, and we ask what data is likely. In notation:

P (data ∣ model)

. You know the world; you ask about what you will see. A bowl has 90 red balls and 10 blue balls. You know this. You close your eyes and pick one. What is the chance it is red? That is probability — the model is known (90 red, 10 blue), the outcome is unknown. Likelihood is backward-looking: the data is fixed, and we ask what parameter values would make that data most probable. In notation:

L (θ ∣ X)

— the likelihood of parameters

θ

given observed data

X

. You saw the data; you ask about the hidden world that produced it.

Formal definition. Let

θ

denote the parameters of a statistical model (mean, variance, slope, etc.). Let

X

denote observed data. - Probability:

P (X ∣ θ)

— "If the world is

θ

, how likely are we to see data

X

?" The function is over

X

;

θ

is fixed. - Likelihood:

L (θ ∣ X)

— "Given we saw

X

, how plausible is each candidate

θ

?" The function is over

θ

;

X

is fixed. The two functions are numerically equal —

L (θ ∣ X) = P (X ∣ θ)

as numbers — but they are functions of different arguments. That is the whole distinction.

12.1.2 Worked Contrast — Heights Example

Setup. Suppose human heights follow a normal distribution with known mean

μ = 170

cm and standard deviation

σ = 10

cm. Probability question: What is the chance a randomly selected person's height falls between 170 and 180 cm? Compute the area under the normal curve from 170 to 180. The z-score for 180 is

\frac{180 - 170}{10} = 1.0

. The area from mean to

+ 1 σ

is approximately 0.341. So:

P (170 \leq height \leq 180 ∣ μ = 170, σ = 10) \approx 0.341 (34.1%)

Likelihood question: We measure one person at 180 cm. Which mean is more plausible —

μ = 170

μ = 175

(with

σ = 10

fixed)? Plug into the normal PDF:

L (μ = 170 ∣ X = 180) = \frac{1}{2 π \cdot 100} exp (- \frac{( 180 - 170 ) ^{2}}{2 \cdot 100}) \approx 0.0242

L (μ = 175 ∣ X = 180) = \frac{1}{2 π \cdot 100} exp (- \frac{( 180 - 175 ) ^{2}}{2 \cdot 100}) \approx 0.0352

The observed height of 180 cm is more likely under

μ = 175

than under

μ = 170

. Sense-check: 180 is closer to 175 than to 170, so this makes sense.

12.1.3 Key Contrast Table

	Probability	Likelihood
What is fixed?	Parameters $θ$	Data $X$
What varies?	Data $X$	Parameters $θ$
Notation	$P (X ∣ θ)$	$L (θ ∣ X)$
Question	"What will I see?"	"What world produced what I saw?"
Sum/Integral	Sums to 1 over all $X$	Does NOT sum to 1 over all $θ$
Example	Fair coin: P(3 heads in a row)	8 heads in 10 flips: what bias?

12.1.4 Symbol Registry

Symbol	Meaning	Type	Domain
$θ$	model parameters (generic)	vector	depends on model
$X$	observed data	vector/scalar	depends on data
$P (X ∣ θ)$	probability of data given parameters	scalar	$[0, 1]$
$L (θ ∣ X)$	likelihood of parameters given data	scalar	$\geq 0$ , not normalized
$μ$	mean of normal distribution	scalar	$R$
$σ$	standard deviation	scalar	$σ > 0$
$σ^{2}$	variance	scalar	$σ^{2} > 0$

12.1.5 Visual Intuition

Imagine a bell curve drawn on graph paper. The x-axis is height (cm), the y-axis is probability density. For probability, the curve is fixed at

μ = 170

σ = 10

. You shade the area between 170 and 180 — that's your 34.1%. For likelihood, you have a single vertical line at

X = 180

(your data point). You then slide different bell curves left and right (varying

μ

) and read off the height where each curve crosses the

X = 180

line. The curve whose peak is closest to 180 gives the highest reading. Takeaway: probability integrates horizontally; likelihood reads vertically.

12.1.6 Assumptions & Scope

Scope. The probability/likelihood distinction applies whenever you have a parametric model — a family of distributions indexed by parameters. It does not require the data to be IID, nor does it require the model to be "correct." It is a purely conceptual distinction about which argument is held fixed. Key limitation: Likelihood is not a probability distribution over parameters. You cannot integrate

L (θ ∣ X)

over

θ

and get 1. This is why we need a prior to convert likelihood into a proper posterior distribution (see Section 12.8, MAP).

12.1.7 Pitfalls

1. Calling likelihood a probability. "The likelihood that

μ = 170

is 0.0242" — this is sloppy. Likelihood is the probability of the data under that parameter, not the probability of the parameter. Parameters don't have probabilities in the MLE framework. 2. Thinking $L (θ ∣ X) = P (θ ∣ X)$ . The vertical bar means different things in these two expressions.

P (A ∣ B)

is a conditional probability;

L (θ ∣ X)

is just notation meaning "likelihood of

θ

given

X

." They are not the same function. 3. Forgetting the model assumption. Likelihood only makes sense after you assume a distribution family. Saying "the likelihood of the data" without specifying "under what model" is meaningless. 4. Confusing "most likely parameter" with "true parameter." The parameter that maximizes likelihood is just the best fit given the data and model. With small samples, it can be far from the truth.

12.1.8 Professor Intuition

Think of likelihood as: "Given this observed data, what are the most likely values of the model parameters?" You assume the data comes from a normal distribution — that is an assumption you must make based on domain knowledge. Once you assume the shape (the model family), you try to find the parameters that make that shape most perfectly fit your data. You do not know the true distribution; you only have a sample. So you assume a distribution family and find the best-fitting member of that family.

12.1.9 Recap & Bridge

Probability looks forward (model → data); likelihood looks backward (data → model). They are the same formula evaluated two different ways. Now that we know which direction we are walking, we need the machinery to combine many data points into one likelihood — that requires the IID assumption, next.

12.1.10 Real-World & Domain Connection

The probability/likelihood distinction is the silent engine inside every statistical model. Consider a pharmaceutical company running a clinical trial. When they report "the drug is effective with p < 0.05," they used likelihood to estimate the treatment effect. They used probability to quantify uncertainty. When Netflix recommends a movie, it used likelihood to fit your preference model from your watch history. In machine learning specifically, every time you call `.fit()` on a model, you are climbing a likelihood surface — even if the library hides that from you. The arrow distinction prevents a common reasoning error: treating an estimate as if it were the truth.

12.2 IID Assumption and Joint Likelihood

Hook. You have 1000 height measurements. How do you combine them into one number that tells you how well a candidate mean

μ

explains all the data? You cannot just look at one point. The answer depends on a deceptively simple assumption — one that every ML pipeline makes but rarely checks.

12.2.1 Intuition — Why Independence Lets Us Multiply

The analogy: independent dice rolls. You roll a fair die twice. The chance of rolling a 6 on the first roll is 1/6. The chance on the second roll is also 1/6 — regardless of what happened on the first. Because the rolls are independent, the joint probability of two sixes is

(1/6) \times (1/6) = 1/36

. If the dice remembered the first roll (dependence), you could not simply multiply — the math would be far more complex. The IID assumption says every data point is like an independent die roll from the same die. This lets us multiply individual probabilities into one joint likelihood. Without it, maximum likelihood estimation becomes dramatically harder.

12.2.2 Formal Definition

Independent and Identically Distributed (IID) is a two-part assumption about your data: - Independent: knowing the value of one data point tells you nothing about any other. One person's height does not affect another's. Each observation carries no information about any other. - Identically Distributed: every data point comes from the same underlying probability distribution with the same parameters. All heights are drawn from the same normal distribution with the same

μ

and

σ

Joint likelihood under IID. Because the data points are independent, the probability of seeing the entire dataset is the product of the probabilities of seeing each point individually:

L (θ ∣ X_{1}, X_{2}, \dots, X_{n}) = i = 1 \prod n f (X_{i} ∣ θ)

Here: -

\prod_{i = 1}^{n}

(capital pi) denotes the product over

i = 1, 2, \dots, n

, just as

\sum

(capital sigma) denotes summation. -

f (X_{i} ∣ θ)

is the Probability Density Function (PDF) if

X

is continuous, or the Probability Mass Function (PMF) if

X

is discrete. In words: the likelihood of all the data = (probability of point 1) × (probability of point 2) × … × (probability of point n).

12.2.3 Worked Example — Two Heights

Suppose heights follow a normal distribution with unknown mean

μ

and known

σ = 10

. You observe two independent heights:

X_{1} = 170

cm and

X_{2} = 180

cm. The joint likelihood for a candidate

μ

is:

L (μ) = f (170 ∣ μ, 10) \cdot f (180 ∣ μ, 10)

For

μ = 175

f (170) = \frac{1}{2 π \cdot 100} e^{- (170 - 175)^{2} /200} \approx 0.0352

f (180) = \frac{1}{2 π \cdot 100} e^{- (180 - 175)^{2} /200} \approx 0.0352

L (175) = 0.0352 \times 0.0352 \approx 0.00124

For

μ = 170

f (170) = \frac{1}{2 π \cdot 100} e^{0} \approx 0.0399

f (180) = \frac{1}{2 π \cdot 100} e^{- (180 - 170)^{2} /200} \approx 0.0242

L (170) = 0.0399 \times 0.0242 \approx 0.00097

L (175) > L (170)

, so

μ = 175

explains both points better. Sense-check: 175 sits exactly between 170 and 180, so this makes sense — the best mean should be in the middle of the data.

12.2.4 Visual Intuition

Picture two bell curves side by side on the same axes. The x-axis is height. For the first data point (

X_{1} = 170

), you read the height of the curve at 170. For the second (

X_{2} = 180

), you read the height at 180. The joint likelihood is the product of those two vertical readings. As you slide the bell curve left and right (varying

μ

), both readings change. You are looking for the

μ

that makes their product as large as possible. Takeaway: with one point, the best

μ

is just that point. With two points, the best

μ

balances both.

12.2.5 Assumptions & Scope

Assumptions. IID requires: 1. Independence — no correlation between data points. Violated in time series (today's stock price depends on yesterday's), spatial data (nearby houses have similar prices), and clustered samples. 2. Identical distribution — no distribution shift. Violated when training data comes from one population and test data from another (e.g., different hospitals, different time periods). What breaks when IID fails: - If data are dependent (e.g., duplicate records), the product formula overcounts evidence — you think you have more independent information than you really do. Confidence intervals become too narrow. - If data are not identically distributed (e.g., mixture of two populations), a single set of parameters cannot describe the whole dataset. The MLE will be a meaningless average.

12.2.6 Pitfalls

1. Assuming IID without checking. In the real world, data is often correlated. Always ask: could one data point influence another? If yes, IID is violated. 2. Confusing IID with "random sample." A random sample can still be dependent (e.g., sampling families — siblings' heights are correlated). Random sampling helps, but it does not guarantee independence. 3. Forgetting to mention IID in exam answers. If you write a likelihood product without stating the IID assumption, your derivation is incomplete.

12.2.7 Exam Note

Exam note: When writing MLE answers, always begin with: "Consider a sample of

n

IID random variables drawn from a distribution…" This single sentence signals to the examiner that you understand why the likelihood factorizes into a product. Omitting it loses marks.

12.2.8 Recap & Bridge

IID lets us multiply individual probabilities into one joint likelihood:

L (θ) = \prod_{i} f (X_{i} ∣ θ)

. Without this assumption, MLE would not have its simple product form. Now we are ready to maximize that product — which brings us to MLE.

12.2.9 Real-World & Domain Connection

The IID assumption is the silent contract behind every ML training loop. When you shuffle your training data before each epoch, you are trying to make batches approximately IID. When a self-driving car company discovers their training data was collected only on sunny California roads, they have violated the "identically distributed" assumption for deployment in rainy Seattle. Distribution shift — the failure of identical distribution between train and test — is one of the hardest problems in production ML, and it all traces back to this foundational assumption.

12.3 Maximum Likelihood Estimate (MLE)

Hook. You are given 100 numbers and told they come from a normal distribution — but nobody told you the mean or the variance. How do you find them? The answer is so natural that you have been doing it since school: take the average. But why is the average the right answer? MLE gives the mathematical reason.

12.3.1 Intuition — Fitting a Bell Curve to Data

The analogy: adjusting a stencil. Imagine you have a transparent plastic stencil shaped like a bell curve. You place it over a scatter of data points drawn on paper. You can slide the stencil left and right (changing

μ

) and stretch or squeeze it horizontally (changing

σ

). Your goal: position and shape the stencil so that the data points "light up" the curve as brightly as possible. The curve should be high where points are dense and low where points are sparse. The

μ

and

σ

of that best-positioned stencil are your MLE. Where the analogy breaks: the "brightness" is really a product of heights (not a sum), and the stencil can only be a bell curve. If your data is not bell-shaped, the stencil is the wrong tool entirely — but MLE will still give you the best-fitting bell curve, however poor the fit.

The professor's own analogy: "These are our data points. The goal of maximum likelihood estimate is to find the optimal way to fit a distribution to the data. If you assume it comes from a normal distribution, we need to figure out where to center that shape. Did it come from this normal distribution with mean this and standard deviation this? Or did it come from the second one? Or the third one? Or the fourth one? The distribution that corresponds to the maximum value of likelihood — that will be the one."

12.3.2 Formal Definition

Maximum Likelihood Estimate (MLE). Given observed data

X = (X_{1}, X_{2}, \dots, X_{n})

assumed to be IID draws from a distribution with unknown parameters

θ

, the MLE is the parameter value

θ^_{MLE}

that maximizes the likelihood function:

θ^_{MLE} = ar g θ max L (θ ∣ X) = ar g θ max i = 1 \prod n f (X_{i} ∣ θ)

In words: among all possible parameter values, pick the one that makes the observed data most probable.

12.3.3 Why Log-Likelihood?

Likelihood values for individual data points are typically very small (e.g., 0.0004, 0.0054). Multiply many such numbers, and the result becomes so tiny that computers cannot represent it — this is numerical underflow. The solution: take the logarithm. The logarithm converts multiplication into addition:

lo g L (θ) = lo g (i = 1 \prod n f (X_{i} ∣ θ)) = i = 1 \sum n lo g f (X_{i} ∣ θ)

The symbol

ℓ

(script lowercase L) denotes log-likelihood:

ℓ (θ) = lo g L (θ)

Why maximizing log-likelihood is equivalent. The logarithm is a monotonically increasing function — if

a > b

, then

lo g a > lo g b

. So the

θ

that maximizes

lo g L (θ)

is exactly the

θ

that maximizes

L (θ)

. The parameter values do not change; only the computational pathway does. Professor note: "If I'm using log, won't the value change? No. Log-likelihood will give you the exact same parameter values as the raw likelihood function. Taking the logarithm will not change the values."

12.3.4 The MLE Optimization Problem

θ^_{MLE} = ar g θ max ℓ (θ) = ar g θ max i = 1 \sum n lo g f (X_{i} ∣ θ)

To find the maximum, we take the derivative of the log-likelihood with respect to

θ

, set it to zero, and solve:

\frac{\partial ℓ ( θ )}{\partial θ} = 0

This works because at the peak of a curve, the slope is zero. The point where the derivative vanishes is a candidate for the maximum. (We must also check the second derivative to confirm it is a maximum, not a minimum.)

12.3.5 The Four-Step MLE Recipe

1. Assume the distribution — based on domain knowledge, choose which distribution family the data follows (normal, binomial, Bernoulli, Poisson, etc.). This is the most important step and the one most open to error. 2. Write the likelihood function —

L (θ) = \prod_{i = 1}^{n} f (X_{i} ∣ θ)

, substituting the actual PDF or PMF of the chosen distribution for

f

. 3. Take the logarithm —

ℓ (θ) = \sum_{i = 1}^{n} lo g f (X_{i} ∣ θ)

to convert the product to a sum. 4. Maximize — take the derivative with respect to each parameter, set to zero, and solve. Verify the second derivative is negative (confirming a maximum).

12.3.6 Worked Mini-Example — Bernoulli MLE

Problem. You flip a coin 5 times and observe: H, T, H, H, T (3 heads, 2 tails). Assume each flip is an independent Bernoulli trial with unknown probability

p

of heads. Find

p^_{MLE}

. Step 1 — Distribution. Bernoulli:

P (X_{i} = 1) = p

P (X_{i} = 0) = 1 - p

. The PMF is

f (x ∣ p) = p^{x} (1 - p)^{1 - x}

. Step 2 — Likelihood. For 3 heads (

x = 1

) and 2 tails (

x = 0

L (p) = p^{3} \cdot (1 - p)^{2}

Step 3 — Log-likelihood.

ℓ (p) = 3 lo g p + 2 lo g (1 - p)

Step 4 — Maximize.

\frac{d ℓ}{d p} = \frac{3}{p} - \frac{2}{1 - p} = 0

\frac{3}{p} = \frac{2}{1 - p} \Rightarrow 3 (1 - p) = 2 p \Rightarrow 3 - 3 p = 2 p \Rightarrow 5 p = 3

p^_{MLE} = \frac{3}{5} = 0.6

Sense-check: 3 heads out of 5 flips — the MLE for the probability of heads is simply the observed proportion. This is exactly what intuition would say.

12.3.7 Visual Intuition

Draw a graph with

θ

on the x-axis and

ℓ (θ)

on the y-axis. The log-likelihood curve typically rises to a single peak and then falls — it is often concave (shaped like an upside-down bowl). The x-coordinate of the peak is

θ^_{MLE}

. The derivative is the slope of this curve. At the peak, the slope is exactly zero. Takeaway: MLE is hill-climbing on the log-likelihood landscape — you find the highest point by solving for where the slope vanishes.

12.3.8 Assumptions & Scope

Assumptions. MLE requires: 1. IID data — the likelihood factorizes as a product only under independence. 2. Correct model family — you must assume the distribution family (normal, Bernoulli, etc.). If the true data-generating process is not in that family, MLE still gives the best fit within that family, but the estimate may be meaningless. 3. Sufficient sample size — MLE is asymptotically unbiased and efficient (best possible variance as

n \to \infty

), but can be biased for small samples. 4. Regularity conditions — the log-likelihood must be differentiable and the parameter space must be open (no boundary solutions). When the maximum lies on a boundary, the derivative test fails.

12.3.9 Pitfalls

1. Overfitting with small $n$ . MLE can overfit when data is scarce. For example, if you observe 0 successes in 3 trials, the Bernoulli MLE is

p^= 0

— a claim that success is impossible, which is rarely justified. MAP (Section 12.8) fixes this with a prior. 2. Forgetting the second derivative check. Setting

\partial ℓ / \partial θ = 0

gives a stationary point, but it could be a minimum or a saddle point. Always verify

\partial^{2} ℓ / \partial θ^{2} < 0

(negative second derivative) to confirm a maximum. 3. Confusing log-likelihood with negative log-likelihood. Many ML frameworks minimize the negative log-likelihood (NLL). Maximizing

ℓ (θ)

is the same as minimizing

- ℓ (θ)

. Know which direction your optimizer walks. 4. Assuming MLE always has a closed form. For normal and Bernoulli distributions, yes — the math works out cleanly. For logistic regression and most neural networks, no — there is no formula, and we use iterative optimization (gradient ascent).

12.3.10 Recap & Bridge

MLE = assume a distribution family, write the likelihood product, take the log, differentiate, set to zero, solve. The four-step recipe is your universal tool. Next we apply it to the most important distribution in statistics: the normal distribution.

12.3.11 Real-World & Domain Connection

MLE is the estimation workhorse across all of statistics and ML. When scikit-learn fits a `LinearRegression`, it computes the MLE under Gaussian noise. When you train a neural network with mean squared error loss, you are computing the MLE under a Gaussian output distribution. When you use binary cross-entropy loss, you are computing the MLE under a Bernoulli output distribution. The loss functions you choose are not arbitrary — each one corresponds to an MLE problem with a specific noise assumption. Understanding MLE means understanding why your loss function exists.

12.4 MLE for the Normal Distribution

Hook. Why do we summarize data with the average? Why not the median, or the mode, or the midrange? The answer is not tradition — it is mathematics. The sample mean is the MLE under a normal distribution. This section proves that connection.

12.4.1 Intuition — The Average as the Best Fit

The analogy: balancing a ruler on a fulcrum. Place a ruler on your finger and hang equal weights at different positions along it. Where should you place your finger so the ruler balances? At the center of mass — the average position. The sample mean is the "balance point" of your data. MLE shows this is also the point that maximizes the likelihood under a bell curve. For variance, think of how "wiggly" the weights are around that balance point. The more spread out they are, the harder it is to balance — that spread is the variance.

12.4.2 Derivation — Complete Step-by-Step

Assume data

X_{1}, X_{2}, \dots, X_{n}

are IID draws from a normal distribution with unknown mean

μ

and unknown variance

σ^{2}

. The PDF of a normal distribution is:

f (x ∣ μ, σ^{2}) = \frac{1}{2 π σ ^{2}} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

Step 1 — Likelihood function (product form):

L (μ, σ^{2}) = i = 1 \prod n \frac{1}{2 π σ ^{2}} exp (- \frac{( X _{i} - μ ) ^{2}}{2 σ ^{2}})

Step 2 — Log-likelihood (convert product to sum):

ℓ (μ, σ^{2}) = lo g L (μ, σ^{2}) = i = 1 \sum n lo g [\frac{1}{2 π σ ^{2}} exp (- \frac{( X _{i} - μ ) ^{2}}{2 σ ^{2}})] = i = 1 \sum n [lo g (\frac{1}{2 π σ ^{2}}) + lo g (exp (- \frac{( X _{i} - μ ) ^{2}}{2 σ ^{2}}))] = i = 1 \sum n [- \frac{1}{2} lo g (2 π σ^{2}) - \frac{( X _{i} - μ ) ^{2}}{2 σ ^{2}}] = - \frac{n}{2} lo g (2 π) - \frac{n}{2} lo g (σ^{2}) - \frac{1}{2 σ ^{2}} i = 1 \sum n (X_{i} - μ)^{2}

Step 3 — Maximize with respect to $μ$ (keep $σ^{2}$ fixed):

\frac{\partial ℓ}{\partial μ} = \frac{\partial}{\partial μ} [- \frac{1}{2 σ ^{2}} i = 1 \sum n (X_{i} - μ)^{2}] = \frac{1}{σ ^{2}} i = 1 \sum n (X_{i} - μ)

Set to zero:

\frac{1}{σ ^{2}} i = 1 \sum n (X_{i} - μ) = 0 \Rightarrow i = 1 \sum n X_{i} - n μ = 0 \Rightarrow μ^_{MLE} = \frac{1}{n} i = 1 \sum n X_{i}

μ^_{MLE}

is the sample mean. Step 4 — Maximize with respect to $σ^{2}$ (substitute $μ^$ ): Take the derivative of

ℓ

with respect to

σ^{2}

(treating

σ^{2}

as a single variable):

\frac{\partial ℓ}{\partial σ ^{2}} = - \frac{n}{2} \cdot \frac{1}{σ ^{2}} + \frac{1}{2 ( σ ^{2} ) ^{2}} i = 1 \sum n (X_{i} - μ^)^{2}

Set to zero:

- \frac{n}{2 σ ^{2}} + \frac{1}{2 ( σ ^{2} ) ^{2}} i = 1 \sum n (X_{i} - μ^)^{2} = 0

Multiply through by

2 (σ^{2})^{2}

- n σ^{2} + i = 1 \sum n (X_{i} - μ^)^{2} = 0 \Rightarrow σ^_{MLE}^{2} = \frac{1}{n} i = 1 \sum n (X_{i} - μ^)^{2}

σ^_{MLE}^{2}

is the sample variance (dividing by

n

, not

n - 1

12.4.3 The Beautiful Result

The MLE for the mean of a normal distribution is the sample mean. The MLE for the variance is the sample variance (dividing by

n

). This is not a coincidence — it is the mathematical reason we use these formulas. Professor insight: "Surprisingly and to our happiness, we will find that the MLE for mean is just the sample mean and MLE for variance is just the sample variance. Now you understand this is how that formula had come. This is the reason behind that formula. Mathematically, this proves why we use average to summarize data — it is the MLE. The sample average is the maximum likelihood estimate of the normal distribution. That is why we are able to summarize data by using the mean."

12.4.4 Worked Example — Five Heights

Data: Heights (cm): 168, 172, 170, 175, 165. Assume a normal distribution. Find

μ^_{MLE}

and

σ^_{MLE}^{2}

. Step 1 — Sample mean:

μ^_{MLE} = \frac{168 + 172 + 170 + 175 + 165}{5} = \frac{850}{5} = 170 cm

Step 2 — Sample variance:

σ^_{MLE}^{2} = \frac{1}{5} [(168 - 170)^{2} + (172 - 170)^{2} + (170 - 170)^{2} + (175 - 170)^{2} + (165 - 170)^{2}] = \frac{1}{5} [4 + 4 + 0 + 25 + 25] = \frac{58}{5} = 11.6 cm^{2}

σ^_{MLE} = 11.6 \approx 3.41 cm

Sense-check: The data ranges from 165 to 175 (span of 10 cm). A standard deviation of ~3.4 cm means about 95% of data within ±6.8 cm of the mean — which matches the observed range well.

12.4.5 Visual Intuition

Draw a normal curve centered at

μ = 170

with

σ \approx 3.4

. Mark the five data points (168, 172, 170, 175, 165) on the x-axis. At each data point, draw a vertical line up to the curve. The height of each intersection is the likelihood contribution of that point. The product of those five heights is the joint likelihood. If you shift the curve to

μ = 160

, the points at 172 and 175 would fall in the tails — their heights would be tiny, and the product would shrink dramatically. Takeaway: the MLE is the curve that makes every data point simultaneously as "tall" as possible.

12.4.6 Assumptions & Scope

Assumptions. MLE for the normal distribution requires: 1. IID data — the product form depends on independence. 2. Normality — the derivation assumes the data-generating process is normal. If the true distribution is heavy-tailed (e.g., Cauchy), the sample mean can be a terrible estimate, and MLE may not even have a unique solution. 3. Sufficient sample size — with small

n

σ^_{MLE}^{2}

(dividing by

n

) is a biased estimator of the true variance. The unbiased version divides by

n - 1

. For large

n

, the bias is negligible. Note on biased vs. unbiased variance:

σ^_{MLE}^{2} = \frac{1}{n} \sum (X_{i} - X ˉ)^{2}

is the MLE but is biased downward. The standard "sample variance"

s^{2} = \frac{1}{n - 1} \sum (X_{i} - X ˉ)^{2}

is unbiased but is not the MLE. Know which one you are computing.

12.4.7 Pitfalls

1. Using the MLE variance formula for small samples without correction. If

n = 3

, dividing by

n

systematically underestimates the true variance. Use

n - 1

for unbiased estimates in small samples. 2. Forgetting the derivative with respect to $σ^{2}$ . Students often stop after finding

μ^

. MLE means finding ALL parameters. 3. Confusing $σ$ with $σ^{2}$ in the derivative. When differentiating with respect to

σ^{2}

, treat

σ^{2}

as the variable — do not differentiate with respect to

σ

unless you use the chain rule correctly.

12.4.8 Exam Note

Exam note: Whenever asked to find MLE for a distribution, follow the four steps. Keep PDF/PMF formulas for common distributions handy. For the normal distribution, the derivation is a standard exam question — know every algebraic line.

12.4.9 Recap & Bridge

MLE for the normal mean = sample mean. MLE for the normal variance = sample variance (divide by

n

). This proves why the average is the "right" summary — it is the MLE. Next, we apply the same MLE logic to linear regression and discover why we minimize squared errors.

12.4.10 Real-World & Domain Connection

The normal distribution's MLE is the theoretical foundation for the Central Limit Theorem's practical use. When pollsters report "the margin of error is ±3%," they are using the normal MLE: the sample mean estimates the population mean, and the sample variance quantifies uncertainty. In manufacturing quality control, the mean and variance of product dimensions are estimated via MLE under a normality assumption to set tolerance limits. The formulas you derived here are run billions of times daily in every statistics package on Earth.

12.5 MLE for Simple Linear Regression

Hook. You have used LinearRegression().fit(X, y) a hundred times. Under the hood, it solves an optimization problem. But WHICH optimization problem, and WHY that one? MLE answers: minimizing squared errors is not an arbitrary choice — it falls out of assuming Gaussian noise. The loss function has a probabilistic justification.

12.5.1 Intuition — The Noisy Line

The analogy: shooting arrows at a moving target. Imagine a target that moves along a straight line as you pull a lever (X). You shoot arrows at it. Your arrows do not hit perfectly — they scatter around the target with a bell-shaped pattern (normal noise). After many shots at different lever positions, you want to recover the line the target was moving along. Each arrow's distance from the true line is the error

ϵ

. MLE says: find the line that makes the observed scatter pattern most probable under the assumption that errors are normally distributed. Where the analogy breaks: the target's position changes with X, so each shot has a different "center." But the spread of errors (

σ

) is the same at every lever position — this is the homoscedasticity assumption.

12.5.2 Setup and Assumptions

In linear regression, we fit a line

Y = β_{0} + β_{1} X

with slope

β_{1}

and intercept

β_{0}

. Real data never lies perfectly on the line. The error (residual)

ϵ

is the gap between each data point and the line:

Y_{i} = β_{0} + β_{1} X_{i} + ϵ_{i}

The critical assumption: errors are IID draws from a normal distribution with mean 0 and variance $σ^{2}$ :

ϵ_{i} \sim N (0, σ^{2})

Because the errors are normally distributed,

Y

is also normally distributed, with its mean being the line itself:

Y_{i} ∣ X_{i} \sim N (β_{0} + β_{1} X_{i}, σ^{2})

Notice: the mean of

Y

depends on

X

— it IS the regression line. The variance

σ^{2}

is constant across all

X

(homoscedasticity).

12.5.3 Derivation — Complete Step-by-Step

Step 1 — Single-observation PDF:

f (Y_{i} ∣ X_{i}, β_{0}, β_{1}, σ^{2}) = \frac{1}{2 π σ ^{2}} exp (- \frac{( Y _{i} - β _{0} - β _{1} X _{i} ) ^{2}}{2 σ ^{2}})

Step 2 — Likelihood function (product over all $n$ observations):

L (β_{0}, β_{1}, σ^{2}) = i = 1 \prod n \frac{1}{2 π σ ^{2}} exp (- \frac{( Y _{i} - β _{0} - β _{1} X _{i} ) ^{2}}{2 σ ^{2}})

Step 3 — Log-likelihood:

ℓ (β_{0}, β_{1}, σ^{2}) = i = 1 \sum n [- \frac{1}{2} lo g (2 π σ^{2}) - \frac{( Y _{i} - β _{0} - β _{1} X _{i} ) ^{2}}{2 σ ^{2}}] = - \frac{n}{2} lo g (2 π) - \frac{n}{2} lo g (σ^{2}) - \frac{1}{2 σ ^{2}} i = 1 \sum n (Y_{i} - β_{0} - β_{1} X_{i})^{2}

Three terms: 1.

- \frac{n}{2} lo g (2 π)

— constant (does not depend on any parameter) 2.

- \frac{n}{2} lo g (σ^{2})

— depends only on

σ^{2}

- \frac{1}{2 σ ^{2}} \sum (Y_{i} - β_{0} - β_{1} X_{i})^{2}

— depends on

β_{0}, β_{1}

and

σ^{2}

12.5.4 Maximization → Ordinary Least Squares

To find

β^_{0}

and

β^_{1}

, we maximize

ℓ

with respect to

β_{0}

and

β_{1}

. Terms 1 and 2 are constant with respect to

β_{0}, β_{1}

— their derivatives are zero. Only term 3 matters. Term 3 has a negative sign in front. Maximizing a negative quantity means making the positive part inside as small as possible:

ar g β_{0}, β_{1} max ℓ (β_{0}, β_{1}, σ^{2}) = ar g β_{0}, β_{1} max (- \frac{1}{2 σ ^{2}} i = 1 \sum n (Y_{i} - β_{0} - β_{1} X_{i})^{2}) = ar g β_{0}, β_{1} min i = 1 \sum n (Y_{i} - β_{0} - β_{1} X_{i})^{2}

This is the Residual Sum of Squares (RSS) — exactly the Ordinary Least Squares (OLS) objective. Maximizing the log-likelihood under Gaussian noise is mathematically equivalent to minimizing the sum of squared errors. The MLE for the regression coefficients is the OLS solution.

Professor insight: "Maximizing the log-likelihood is equivalent to minimizing the residual sum of squares. Maximum likelihood estimation gives you the exact same result as the ordinary least squares method used in linear regression. The solution to the MLE problem for linear regression is the same as the solution to the ordinary least squares problem. This is the explanation of why we were using those formulas in linear regression — this is the reason behind all of that."

12.5.5 MLE for the Variance

Differentiate

ℓ

with respect to

σ^{2}

and set to zero. The result is the Mean Squared Error (MSE):

σ^_{MLE}^{2} = \frac{1}{n} i = 1 \sum n (Y_{i} - β^_{0} - β^_{1} X_{i})^{2} = MSE

12.5.6 Worked Example — Three Points

Data:

(X, Y)

(1, 2), (2, 4), (3, 5)

. Fit a line

Y = β_{0} + β_{1} X

using MLE. Step 1 — Compute OLS estimates:

X ˉ = 2, Y ˉ = \frac{11}{3} \approx 3.667

β^_{1} = \frac{\sum ( X _{i} - X ˉ ) ( Y _{i} - Y ˉ )}{\sum ( X _{i} - X ˉ ) ^{2}} = \frac{( 1 - 2 ) ( 2 - 3.667 ) + 0 + ( 3 - 2 ) ( 5 - 3.667 )}{( 1 - 2 ) ^{2} + 0 + ( 3 - 2 ) ^{2}}

= \frac{( - 1 ) ( - 1.667 ) + ( 1 ) ( 1.333 )}{1 + 1} = \frac{1.667 + 1.333}{2} = \frac{3}{2} = 1.5

β^_{0} = Y ˉ - β^_{1} X ˉ = 3.667 - 1.5 \times 2 = 0.667

Fitted line:

Y = 0.667 + 1.5 X

Step 2 — Compute MLE for $σ^{2}$ :

σ^^{2} = \frac{1}{3} [(2 - 0.667 - 1.5 \cdot 1)^{2} + (4 - 0.667 - 1.5 \cdot 2)^{2} + (5 - 0.667 - 1.5 \cdot 3)^{2}]

= \frac{1}{3} [(- 0.167)^{2} + (0.333)^{2} + (- 0.167)^{2}] = \frac{1}{3} (0.028 + 0.111 + 0.028) = 0.056

Sense-check: The residuals are small (

\approx \pm 0.33

), so

σ^^{2} \approx 0.056

(RMSE

\approx 0.24

) is reasonable for data ranging from 2 to 5.

12.5.7 Visual Intuition

Plot three points: (1,2), (2,4), (3,5). Draw the fitted line

Y = 0.667 + 1.5 X

. At each

X_{i}

, draw a vertical dashed line from the point to the line — these are the residuals. The MLE line is the one that minimizes the sum of the squared lengths of those dashed lines. If you tilt the line to be steeper, the residual at

X = 1

grows; if flatter, the residual at

X = 3

grows. The OLS line balances all three. Takeaway: MLE under Gaussian noise = the line closest to all points in squared distance.

12.5.8 Assumptions & Scope

Assumptions. The MLE → OLS equivalence requires: 1. IID errors — residuals must be independent with constant variance (homoscedasticity). 2. Normal errors — if errors are not normal, OLS still gives the best linear unbiased estimator (Gauss-Markov theorem), but it is no longer the MLE. 3. Zero-mean errors — the line is unbiased; systematic bias violates the model. 4. No noise in $X$ — only

Y

has measurement error. If

X

is also noisy, the problem becomes "errors-in-variables" and OLS is inconsistent. What breaks: If errors are heavy-tailed (e.g., Cauchy), OLS is unstable — one outlier can pull the line arbitrarily far. If errors are heteroscedastic (variance changes with

X

), OLS is still consistent but inefficient; weighted least squares becomes the MLE.

12.5.9 Pitfalls

1. Forgetting OLS = MLE only under normality. If your data has outliers, the Gaussian assumption is wrong, and OLS is no longer optimal. Consider robust regression (Huber loss) or a heavier-tailed likelihood. 2. Ignoring $σ^{2}$ . Finding

β^_{0}

and

β^_{1}

is only part of MLE. You must also estimate

σ^{2}

— it quantifies how noisy the relationship is. 3. Confusing MSE with the unbiased variance estimator.

σ^_{MLE}^{2} = MSE = \frac{1}{n} \sum (residuals)^{2}

divides by

n

. The unbiased estimator divides by

n - p

where

p

is the number of parameters.

12.5.10 Symbol Registry

Symbol	Meaning	Type
$β_{0}$	intercept parameter	scalar
$β_{1}$	slope parameter	scalar
$σ^{2}$	error variance	scalar, $> 0$
$ϵ_{i}$	error/residual for observation $i$	scalar
$Y_{i}$	observed dependent variable	scalar
$X_{i}$	independent variable (feature)	scalar

12.5.11 Recap & Bridge

MLE for linear regression under Gaussian noise = Ordinary Least Squares. Minimizing RSS is not an arbitrary choice — it is the MLE. Next, we ask: what happens when the output is not continuous but binary? That leads to logistic regression.

12.5.12 Real-World & Domain Connection

The MLE → OLS connection is why linear regression is the first model taught in every statistics course. It is not just simple — it is principled. In econometrics, OLS is used to estimate causal effects (e.g., "does an extra year of education increase income?"), and the MLE underpinning gives confidence intervals and hypothesis tests their validity. In engineering, OLS calibrates sensors by fitting a linear response curve to calibration data. Every `lm()` in R and every `LinearRegression` in scikit-learn is silently executing the MLE derivation you just worked through.

12.6 MLE for Logistic Regression

Hook. Linear regression gave us a clean equivalence: MLE = OLS = minimize squared errors. Logistic regression is messier. There is no closed-form formula for the best parameters. Yet MLE still gives us an answer — it just requires climbing a hill one step at a time. And the loss function it produces? Binary cross-entropy — the same one you have been using in every classification model.

12.6.1 Intuition — Squashing a Line into a Probability

The analogy: a dimmer switch vs. a light switch. Linear regression outputs any real number — like a dimmer that can go from 0% to 100% brightness, but also to −50% or 150%, which makes no physical sense. Logistic regression adds a "squashing" mechanism — the sigmoid function. This forces the output into

[0, 1]

. It works like a light switch that can only be OFF (0) or ON (1), with a smooth dimmer range in between representing uncertainty. The linear combination

z = β_{0} + β_{1} X

is the raw voltage; the sigmoid

σ (z)

converts it to a probability. Where the analogy breaks: a real dimmer is deterministic. The sigmoid output is a probability, which means even at 0.8, the light might still be off — just less likely.

12.6.2 Setup — The Sigmoid Model

Logistic regression predicts a binary outcome (0 or 1). It starts with a linear combination of features:

z = β_{0} + β_{1} X_{1} + β_{2} X_{2} + \dots + β_{n} X_{n}

This

z

is passed through the sigmoid (logistic) function:

σ (z) = \frac{1}{1 + e ^{- z}}

In compact vector notation:

z = θ^{⊤} X

, and

h_{θ} (X) = σ (θ^{⊤} X) = \frac{1}{1 + e ^{- θ^{⊤} X}}

The output

h_{θ} (X)

is a number between 0 and 1 — interpreted as

P (Y = 1 ∣ X)

12.6.3 Underlying Probability Model — Bernoulli

Since the outcome has only two possibilities (0 or 1), each observation follows a Bernoulli distribution. The probability mass function (PMF) for a single data point is:

P (Y ∣ X, θ) = h_{θ} (X)^{Y} \cdot (1 - h_{θ} (X))^{1 - Y}

This compact form works because: - If

Y = 1

h_{θ} (X)^{1} \cdot (1 - h_{θ} (X))^{0} = h_{θ} (X)

✓ - If

Y = 0

h_{θ} (X)^{0} \cdot (1 - h_{θ} (X))^{1} = 1 - h_{θ} (X)

✓

12.6.4 Likelihood and Log-Likelihood

Likelihood function (product over all

n

observations):

L (θ) = i = 1 \prod n h_{θ} (X_{i})^{Y_{i}} \cdot (1 - h_{θ} (X_{i}))^{1 - Y_{i}}

Log-likelihood (convert product to sum):

ℓ (θ) = lo g L (θ) = i = 1 \sum n lo g [h_{θ} (X_{i})^{Y_{i}} \cdot (1 - h_{θ} (X_{i}))^{1 - Y_{i}}] = i = 1 \sum n [Y_{i} lo g h_{θ} (X_{i}) + (1 - Y_{i}) lo g (1 - h_{θ} (X_{i}))]

12.6.5 Connection to Binary Cross-Entropy

Multiply the log-likelihood by

- 1

- ℓ (θ) = - i = 1 \sum n [Y_{i} lo g h_{θ} (X_{i}) + (1 - Y_{i}) lo g (1 - h_{θ} (X_{i}))]

This is exactly the binary cross-entropy loss function. So: - Maximizing the log-likelihood = Minimizing the binary cross-entropy loss Professor insight: "If you multiply this by −1, this formula is exactly the binary cross-entropy loss function which we studied in logistic regression. Maximizing the likelihood is exactly like minimizing the cross-entropy. Just like in linear regression maximizing likelihood was like minimizing the sum of squared errors, in logistic regression maximizing the likelihood is the same as minimizing the cross-entropy."

12.6.6 Why No Closed-Form Solution?

Unlike linear regression, setting

\frac{\partial ℓ}{\partial θ} = 0

for logistic regression does not yield a direct formula for

θ

. The sigmoid function

σ (z) = 1/ (1 + e^{- z})

is nonlinear, so the derivative equation is transcendental — it cannot be rearranged to isolate

θ

. Instead, we use gradient ascent — an iterative hill-climbing algorithm:

θ^{new} = θ^{old} + α \cdot \frac{\partial ℓ ( θ )}{\partial θ}

where: -

α

is the learning rate — how big a step we take each iteration -

\frac{\partial ℓ}{\partial θ}

is the gradient — the direction of steepest ascent on the log-likelihood surface Starting from an initial guess (usually zeros), we repeatedly take steps uphill until the gradient is near zero — meaning we have reached the peak.

12.6.7 Worked Example — Gradient Ascent Trace

Setup: Two data points:

(X = 1, Y = 1)

and

(X = 2, Y = 0)

. Model:

h_{θ} (X) = σ (θ X)

with

θ

a single parameter. Initial

θ = 0

. Learning rate

α = 0.1

. Iteration 1: -

h_{0} (1) = σ (0) = 0.5

h_{0} (2) = 0.5

- Gradient (simplified for this case):

\frac{\partial ℓ}{\partial θ} = \sum (Y_{i} - h_{θ} (X_{i})) X_{i}

\frac{\partial ℓ}{\partial θ} = (1 - 0.5) \cdot 1 + (0 - 0.5) \cdot 2 = 0.5 - 1.0 = - 0.5

θ \leftarrow 0 + 0.1 \cdot (- 0.5) = - 0.05

Iteration 2: -

h_{- 0.05} (1) = σ (- 0.05) \approx 0.4875

h_{- 0.05} (2) = σ (- 0.10) \approx 0.4750

- Gradient:

(1 - 0.4875) \cdot 1 + (0 - 0.4750) \cdot 2 = 0.5125 - 0.95 = - 0.4375

θ \leftarrow - 0.05 + 0.1 \cdot (- 0.4375) = - 0.0938

The parameter

θ

is moving negative, which makes sense — the negative example (

Y = 0

) has a larger

X

value, so

θ

should be negative to push

h_{θ} (2)

toward 0. After many iterations,

θ

converges to the MLE. Sense-check: The positive example at

X = 1

and negative at

X = 2

suggests the decision boundary should be between 1 and 2 — a negative

θ

achieves this.

12.6.8 Visual Intuition

Picture a 3D landscape where the x-axis and y-axis are

β_{0}

and

β_{1}

, and the z-axis (height) is the log-likelihood

ℓ (θ)

. The surface is smooth and concave — it has a single global maximum (no local traps). Gradient ascent starts at some point (say, the origin) and takes small steps in the steepest uphill direction. Each step is a small nudge of

β_{0}

and

β_{1}

. The path spirals or zig-zags toward the summit. Takeaway: gradient ascent is a hiker climbing a foggy hill who can only feel the slope underfoot — no map, no GPS, just step uphill.

12.6.9 Comparison — Linear vs. Logistic Regression MLE

	Linear Regression MLE	Logistic Regression MLE
Output type	Continuous ( $R$ )	Binary ( ${0, 1}$ )
Noise distribution	Normal	Bernoulli
MLE objective	Minimize RSS (OLS)	Minimize cross-entropy
Closed-form solution?	Yes	No
Optimization	Direct formula	Gradient ascent
Loss name	Mean Squared Error	Binary Cross-Entropy

When to pick which: Use linear regression when the output is a real number (price, temperature, height). Use logistic regression when the output is a category (spam/not-spam, sick/healthy, click/no-click).

12.6.10 Assumptions & Scope

Assumptions. MLE for logistic regression requires: 1. Binary outcomes — the Bernoulli model only handles 0/1. For multi-class, use multinomial logistic regression (softmax). 2. Independent observations — the likelihood factorizes under independence. 3. Linearity in the log-odds — the log-odds

lo g \frac{p}{1 - p} = θ^{⊤} X

is linear in

X

. If the true decision boundary is a circle, logistic regression needs polynomial features. 4. No complete separation — if the classes are perfectly separable by a hyperplane, the MLE for

θ

diverges to infinity (the sigmoid becomes a step function). Regularization fixes this. What breaks: With perfectly separable data, gradient ascent never converges —

∥ θ ∥ \to \infty

. Always add L2 regularization (ridge penalty) to keep

θ

finite.

12.6.11 Pitfalls

1. Thinking logistic regression has a closed form like linear regression. It does not. Setting the derivative to zero gives an equation you cannot solve algebraically. Gradient ascent (or Newton's method) is required. 2. Confusing the sigmoid derivative with the log-likelihood derivative. The gradient for logistic regression is

\sum (Y_{i} - h_{θ} (X_{i})) X_{i}

— beautifully simple. Do not confuse this with the derivative of the sigmoid itself. 3. Forgetting that MLE maximizes, while ML libraries minimize. PyTorch and TensorFlow minimize the negative log-likelihood (cross-entropy). If you are deriving gradient ascent for MLE, that is the opposite direction. 4. Using MSE loss for binary classification. MSE assumes Gaussian noise. Binary classification needs Bernoulli noise → cross-entropy loss. Using MSE for classification gives poorly calibrated probabilities and slower convergence.

12.6.12 Recap & Bridge

MLE for logistic regression = maximize log-likelihood = minimize binary cross-entropy. No closed form exists; use gradient ascent. The loss function you have been using is not arbitrary — it is the MLE under a Bernoulli model. Next, we work through a full numerical MLE example with a custom distribution.

12.6.13 Real-World & Domain Connection

Logistic regression's MLE is the engine behind credit scoring, medical diagnosis, and click-through-rate prediction. When a bank decides whether to approve a loan, a logistic regression model estimates

P (default ∣ applicant features)

via MLE. When Facebook predicts whether you will click an ad, it is running gradient ascent on a log-likelihood surface — often with millions of parameters. The binary cross-entropy loss you derived here is the most widely used classification loss in deep learning, from image recognition to natural language processing.

12.7 MLE — Worked Numerical Example (Custom Distribution)

Hook. Most MLE problems use familiar distributions — Normal, Bernoulli, Poisson. But what if someone hands you a completely custom probability table and says "find the best parameter"? The four-step recipe still works. This example proves it.

12.7.1 Intuition — MLE with a Custom Table

Think of each value (0, 1, 2, 3) as a slot machine with a different payout probability. The parameter

θ

is a hidden dial that controls all four probabilities simultaneously. You pull the lever 10 times and record the outcomes. Your job: infer where the dial was set by looking only at the tally.

12.7.2 Problem Statement

We observe a variable

X

that can take values 0, 1, 2, or 3. The probability of each value depends on an unknown parameter

θ

where

0 < θ < 1

$X$	$P (X)$
0	$2 θ /3$
1	$θ /3$
2	$2 (1 - θ) /3$
3	$(1 - θ) /3$

Verify this is a valid distribution:

2 θ /3 + θ /3 + 2 (1 - θ) /3 + (1 - θ) /3 = (3 θ + 3 (1 - θ)) /3 = 1

✓ We observe 10 independent draws:

3, 0, 2, 1, 2, 2, 0, 1, 1, 3

Find

θ^_{MLE}

12.7.3 Step-by-Step Solution

Step 1 — Count occurrences:

Value	Count
0	2
1	3
2	3
3	2

Total:

2 + 3 + 3 + 2 = 10

✓ Step 2 — Write the likelihood function: The likelihood is the product of the probabilities for each observed value, raised to their counts:

L (θ) = P (X = 0)^{2} \cdot P (X = 1)^{3} \cdot P (X = 2)^{3} \cdot P (X = 3)^{2} = (\frac{2 θ}{3})^{2} \cdot (\frac{θ}{3})^{3} \cdot (\frac{2 ( 1 - θ )}{3})^{3} \cdot (\frac{1 - θ}{3})^{2}

Step 3 — Take the log-likelihood:

ℓ (θ) = 2 lo g (\frac{2 θ}{3}) + 3 lo g (\frac{θ}{3}) + 3 lo g (\frac{2 ( 1 - θ )}{3}) + 2 lo g (\frac{1 - θ}{3}) = 2 (lo g 2 + lo g θ - lo g 3) + 3 (lo g θ - lo g 3) + 3 (lo g 2 + lo g (1 - θ) - lo g 3) + 2 (lo g (1 - θ) - lo g 3)

Step 3b — Simplify using the shortcut (ignore constants): Group terms containing

θ

and terms containing

(1 - θ)

. Constants (

lo g 2

lo g 3

) will vanish during differentiation. -

lo g θ

appears in the first term (coefficient 2) and the second term (coefficient 3) → total:

5 lo g θ

lo g (1 - θ)

appears in the third term (coefficient 3) and the fourth term (coefficient 2) → total:

5 lo g (1 - θ)

So (ignoring constants):

ℓ (θ) = 5 lo g θ + 5 lo g (1 - θ) + constants

Step 4 — Take the derivative:

\frac{d ℓ}{d θ} = \frac{5}{θ} - \frac{5}{1 - θ}

(The derivative of

lo g (1 - θ)

\frac{- 1}{1 - θ}

; multiplying by 5 gives

\frac{- 5}{1 - θ}

.) Step 5 — Set to zero and solve:

\frac{5}{θ} - \frac{5}{1 - θ} \frac{5}{θ} \frac{1}{θ} 1 - θ 2 θ θ^_{MLE} = 0 = \frac{5}{1 - θ} = \frac{1}{1 - θ} = θ = 1 = 0.5

Step 6 — Verify second derivative (confirm it is a maximum):

\frac{d ^{2} ℓ}{d θ ^{2}} = - \frac{5}{θ ^{2}} - \frac{5}{( 1 - θ ) ^{2}}

θ = 0.5

\frac{d ^{2} ℓ}{d θ ^{2}} = - 20 - 20 = - 40 < 0

→ maximum confirmed. ✓

12.7.4 Interpretation

θ^_{MLE} = 0.5

is the parameter value that makes this observed sequence of 10 draws most probable. With

θ = 0.5

, the distribution becomes symmetric:

P (0) = 1/3

P (1) = 1/6

P (2) = 1/3

P (3) = 1/6

. The data's counts (2,3,3,2) closely match these proportions. Professor note: "This was a custom distribution — not normal or anything standard. Since they gave the probability function in terms of

θ

, we could use it. The idea is: for this specific dataset, the value of

θ

that makes the observed data most likely is 0.5."

12.7.5 Visual Intuition

Plot

ℓ (θ)

against

θ

for

θ \in (0, 1)

. The curve starts at

- \infty

θ \to 0

(because

lo g θ \to - \infty

), rises to a smooth peak at

θ = 0.5

where

ℓ (0.5) = 5 lo g (0.5) + 5 lo g (0.5) \approx - 6.93

, then falls back to

- \infty

θ \to 1

(because

lo g (1 - θ) \to - \infty

). The curve is symmetric and concave. Takeaway: the log-likelihood is a hill with a single peak — easy to find by setting the slope to zero.

12.7.6 Assumptions & Scope

Assumptions. This worked example relies on: 1. IID observations — each of the 10 draws is independent and from the same custom distribution. If draws were dependent, the likelihood would not factor as a simple product of powers. 2. Correct probability table — the given

P (X)

formulas are assumed to be the true data-generating process. In practice, specifying the right distribution family is the hardest step. 3. Interior solution —

θ = 0.5

lies strictly inside

(0, 1)

. If the derivative test gave

θ \leq 0

θ \geq 1

, the MLE would be on the boundary and the derivative test would fail.

12.7.7 Pitfalls

1. Forgetting to count correctly. Double-check your tally. A single counting error changes all the exponents and gives a wrong MLE. 2. Including constants in the derivative. Terms like

2 lo g 2

and

- 10 lo g 3

are constants — their derivatives are zero. Include them if you want, but they cancel out anyway. 3. Sign error on $lo g (1 - θ)$ derivative.

\frac{d}{d θ} lo g (1 - θ) = \frac{- 1}{1 - θ}

, NOT

\frac{1}{1 - θ}

. The negative sign is critical.

12.7.8 Student Q&A

Q: Can we use the shortcut method of grouping

θ

terms and

(1 - θ)

terms for other problems? A: Yes — constants like

lo g 2

lo g 3

all become zero during differentiation. You can group terms containing the parameter of interest and ignore constants. This simplifies the algebra significantly. Q: What types of numerical problems can we expect? A: Numerical problems on MLE, possibly with Bernoulli or other distributions. You may also be asked to take derivatives. The distribution's PDF/PMF may or may not be given — keep formulas for common distributions handy.

12.7.9 Recap & Bridge

MLE works for custom distributions too — follow the four-step recipe, group by parameter, differentiate, solve. The result

θ^= 0.5

is the best fit for this data. Next we add prior knowledge to the estimation problem — introducing MAP.

12.7.10 Real-World & Domain Connection

Custom distributions appear in specialized domains: queuing theory (service time distributions), reliability engineering (lifetime distributions for components), and ecology (species abundance models). When no standard distribution fits, domain experts design custom probability models. MLE then provides the principled way to estimate parameters from data — the same four steps work regardless of the distribution's complexity.

12.8 Maximum A Posteriori (MAP) Estimate

Hook. MLE asks: "What parameters best explain the data?" MAP asks a richer question: "What parameters best explain the data, given what I already believe?" That one addition — a prior belief — changes everything. It prevents wild estimates from small samples and lets you inject domain expertise into your model.

12.8.1 Intuition — The Detective with a Hunch

The analogy: a detective updating a suspect list. A detective starts with a list of suspects, each with an initial probability of guilt (the prior). Then a new piece of evidence arrives — a fingerprint (the data). The detective updates each suspect's probability using Bayes' rule: how likely is the fingerprint if this suspect is guilty (the likelihood)? The new probabilities (the posterior) combine the prior hunch with the new evidence. MLE is like a detective who ignores the initial suspect list and only looks at the fingerprint. MAP is the detective who weighs both. A strong prior (e.g., the suspect was in another country) can override even seemingly damning evidence.

12.8.2 Formal Definition — MLE vs MAP

- MLE:

θ^_{MLE} = ar g max_{θ} P (D ∣ θ)

— maximize the likelihood only. - MAP:

θ^_{MAP} = ar g max_{θ} P (θ ∣ D)

— maximize the posterior probability. MAP includes a prior distribution

P (θ)

that encodes what we believe about

θ

before seeing any data. When the prior is uniform (all

θ

equally likely), MAP reduces to MLE. Professor contrast: "MLE ignores the prior distribution. MAP includes the prior knowledge. If you have strong prior knowledge, MAP will give high weight to the prior. If you have less prior knowledge, MAP will rely more on the data."

12.8.3 Bayes Theorem — The Foundation

Bayes theorem relates the posterior to the likelihood and prior:

P (θ ∣ D) = \frac{P ( D ∣ θ ) \cdot P ( θ )}{P ( D )}

Each term: -

P (θ ∣ D)

— Posterior: probability of

θ

after seeing data

D

P (D ∣ θ)

— Likelihood: how probable is

D

θ

is true? -

P (θ)

— Prior: how probable was

θ

before seeing any data? -

P (D)

— Evidence (marginal likelihood):

P (D) = \int P (D ∣ θ) P (θ) d θ

, a normalizing constant Since

P (D)

does not depend on

θ

, we can drop it for maximization:

P (θ ∣ D) \propto P (D ∣ θ) \cdot P (θ)

Posterior $\propto$ Likelihood $\times$ Prior

12.8.4 MAP Estimation Formula

θ^_{MAP} = ar g θ max P (D ∣ θ) \cdot P (θ)

Or equivalently, working with logarithms:

θ^_{MAP} = ar g θ max [lo g P (D ∣ θ) + lo g P (θ)]

The first term is the log-likelihood (same as MLE). The second term

lo g P (θ)

is the log-prior — this is the new piece that MAP adds to MLE.

12.8.5 Bayes Theorem — Disease Testing Example

Suppose you are tested for a rare disease: - Prior: The disease affects 1% of the population:

P (disease) = 0.01

- Likelihood (sensitivity): If you have the disease, the test is positive 95% of the time:

P (+ ∣ disease) = 0.95

- Likelihood (false positive): If you are healthy, the test is positive 5% of the time:

P (+ ∣ healthy) = 0.05

- Evidence:

P (+) = P (+ ∣ disease) P (disease) + P (+ ∣ healthy) P (healthy)

= 0.95 \times 0.01 + 0.05 \times 0.99 = 0.0095 + 0.0495 = 0.059

Question: Given a positive test, what is the probability you actually have the disease?

P (disease ∣ +) = \frac{P ( + ∣ disease ) \cdot P ( disease )}{P ( + )} = \frac{0.95 \times 0.01}{0.059} \approx 0.161

Intuition check: Only 16.1% — not 95%! The test is accurate, but the disease is so rare (1% prior) that most positive results are false positives. This is why screening tests for rare conditions always need follow-up confirmation.

12.8.6 MAP Steps

1. Define the prior distribution

P (θ)

— express initial beliefs about the parameter before seeing data. Choose a distribution family and hyperparameters. 2. Write the likelihood function

P (D ∣ θ)

— same as MLE. 3. Form the (unnormalized) posterior — multiply:

P (θ ∣ D) \propto P (D ∣ θ) \cdot P (θ)

. 4. Maximize the posterior — take the derivative of

lo g P (D ∣ θ) + lo g P (θ)

with respect to

θ

, set to zero, and solve.

12.8.7 Visual Intuition

Picture two curves on the same axes (

θ

on x-axis, density on y-axis). The likelihood curve is tall and peaked — it represents what the data alone says. The prior curve is broader — it represents your pre-data belief. The posterior curve (their product, normalized) is a compromise: it is shifted from the likelihood toward the prior, and it is narrower than either (more certainty). The MAP estimate is the x-coordinate of the posterior's peak. Takeaway: the prior pulls the estimate away from the MLE toward what you already believed.

12.8.8 Comparison — MLE vs MAP

	MLE	MAP
Maximizes	$P (D ∣ θ)$	$P (θ ∣ D)$
Uses prior?	No	Yes
Formula	$ar g max_{θ} P (D ∣ θ)$	$ar g max_{θ} P (D ∣ θ) P (θ)$
Small-sample behavior	Can overfit (e.g., $p^= 0$ after 0/3 heads)	Prior regularizes (pulls toward prior mean)
As $n \to \infty$	Converges to true value (if model correct)	Converges to MLE (prior washes out)
When to use	Large $n$ , no domain knowledge	Small $n$ , strong domain knowledge

12.8.9 Assumptions & Scope

Assumptions. MAP requires: 1. A proper prior —

P (θ)

must be a valid probability distribution (integrates to 1). Improper priors (e.g., uniform over

R

) can still produce valid MAP estimates but require care. 2. Prior-likelihood compatibility — the prior should have support where the likelihood is nonzero. A prior that is zero everywhere the likelihood is positive gives a zero posterior everywhere. 3. Unimodal posterior — the derivative test finds a stationary point, but the posterior may have multiple peaks. MAP finds a mode, not necessarily the global maximum. When the prior dominates: With very small

n

, the MAP estimate is pulled strongly toward the prior mode. This is regularization — it prevents extreme estimates but can introduce bias if the prior is wrong.

12.8.10 Pitfalls

1. Choosing a prior that is too strong. A highly concentrated prior can overwhelm the data, even with large

n

. Always ask: would my conclusion change if I doubled the prior's variance? 2. Forgetting that MAP is a point estimate. MAP gives you a single "best" parameter value — it does not give you the full posterior distribution. If you need uncertainty quantification (credible intervals), use full Bayesian inference, not MAP. 3. Confusing $P (θ ∣ D)$ with $P (D ∣ θ)$ . MAP maximizes the posterior; MLE maximizes the likelihood. The formulas look similar but the quantity being maximized is different. 4. Using MAP when the posterior is multimodal. If the posterior has two peaks of similar height, reporting only the taller one (MAP) discards a plausible alternative explanation.

12.8.11 Recap & Bridge

MAP = MLE + prior. Posterior

\propto

Likelihood

\times

Prior. The prior acts as a regularizer, pulling estimates toward what you already believe. Next we work a concrete MAP example with coin tosses and the Beta-Binomial conjugate pair.

12.8.12 Real-World & Domain Connection

MAP estimation is the backbone of Bayesian statistics. In A/B testing, MAP estimates with informative priors prevent "peeking" problems — you can update estimates continuously without inflating false positives. In natural language processing, MAP with Dirichlet priors (Laplace smoothing) prevents zero probability estimates for unseen words. In robotics, MAP is used for simultaneous localization and mapping (SLAM), where the prior (the robot's previous position estimate) is combined with new sensor readings to update its belief about where it is.

12.9 MAP — Worked Example (Beta-Binomial Coin Toss)

Hook. You flip a coin 10 times and get 7 heads. The MLE says the probability of heads is exactly 0.7. But would you really bet your life savings that the 11th flip will be heads 70% of the time? Probably not — 10 flips is not much evidence. MAP lets you temper this estimate with the reasonable belief that most coins are near fair.

12.9.1 Intuition — Conjugate Priors as "Virtual Data"

The analogy: adding imaginary flips. Imagine that before you started flipping, you imagined 2 flips: 1 head and 1 tail. These are not real data — they represent your prior belief that the coin is probably fair. Now you combine: 7 real heads + 1 imaginary head = 8 heads; 3 real tails + 1 imaginary tail = 4 tails. Your MAP estimate becomes 8/(8+4) = 0.667 — slightly pulled toward 0.5 from the MLE's 0.7. This "imaginary flips" interpretation is exact for the Beta-Binomial conjugate pair. The Beta prior's parameters

α

and

β

can be thought of as "pseudo-counts" of prior heads and tails.

12.9.2 Problem Statement

Estimate the probability of heads

p

for a biased coin. You toss it 10 times and observe 7 heads, 3 tails. Use MAP with a Beta prior. Assume no prior information (uniform prior).

12.9.3 Prior Distribution — Beta(1, 1)

For a probability parameter

p \in [0, 1]

, the Beta distribution is the natural prior:

Beta (p ∣ α, β) = \frac{p ^{α - 1} ( 1 - p ) ^{β - 1}}{B ( α , β )}

where

B (α, β) = \frac{Γ ( α ) Γ ( β )}{Γ ( α + β )}

is the Beta function (a normalizing constant). Setting

α = 1

β = 1

gives

Beta (p ∣ 1, 1) = 1

for all

p \in [0, 1]

— a uniform distribution. Every

p

between 0 and 1 is equally likely. This encodes "I have no idea what the bias is."

12.9.4 Likelihood — Binomial Distribution

The likelihood of observing 7 heads in 10 independent tosses, given bias

p

P (data ∣ p) = (7 10) \cdot p^{7} \cdot (1 - p)^{3}

Where: -

(7 10) = \frac{10 !}{7 ! \cdot 3 !} = 120

counts the number of ways to arrange 7 heads in 10 flips -

p^{7}

= probability of 7 heads -

(1 - p)^{3}

= probability of 3 tails

12.9.5 Beta-Binomial Conjugacy

When the prior is

Beta (α, β)

and the likelihood is

Binomial (n, p)

, the posterior is also a Beta distribution:

Beta (α + k, β + n - k)

where

k

is the number of successes (heads) and

n

is the total number of trials. This is called a conjugate pair — the prior and posterior belong to the same distribution family. The update rule is beautifully simple:

α_{posterior} = k + α_{prior} = 7 + 1 = 8

β_{posterior} = (n - k) + β_{prior} = 3 + 1 = 4

Posterior:

Beta (8, 4)

Professor note: "Whenever there are coin toss kind of experiments, the prior can be modeled using Beta and the likelihood can be modeled using Binomial. If prior is Beta and likelihood is Binomial, the posterior will be Beta. It is a conjugate pair. This is a concept to learn — so that when you see a similar problem you'll be able to solve it."

12.9.6 MAP Estimate — Mode of the Posterior

The MAP estimate is the mode (peak) of the posterior Beta distribution. For

Beta (α, β)

with

α > 1

and

β > 1

, the mode is:

mode = \frac{α - 1}{α + β - 2}

Derivation (for completeness): The log-posterior is

(α - 1) lo g p + (β - 1) lo g (1 - p) + constant

. Differentiating:

\frac{d}{d p} = \frac{α - 1}{p} - \frac{β - 1}{1 - p} = 0 \Rightarrow p = \frac{α - 1}{α + β - 2}

Substituting

α = 8

β = 4

p^_{MAP} = \frac{8 - 1}{8 + 4 - 2} = \frac{7}{10} = 0.7

12.9.7 Interpretation

With a uniform prior

Beta (1, 1)

, the MAP estimate is 0.7 — identical to the MLE (7/10). This makes sense: a uniform prior provides no preference for any

p

, so the data alone determines the estimate. What if the prior were informative? Suppose instead we used

Beta (5, 5)

— representing a prior belief that the coin is likely fair (pseudo-count of 5 heads, 5 tails from prior experience):

α_{posterior} = 7 + 5 = 12, β_{posterior} = 3 + 5 = 8

p^_{MAP} = \frac{12 - 1}{12 + 8 - 2} = \frac{11}{18} \approx 0.611

The informative prior pulls the estimate from 0.7 toward 0.5 — a more conservative answer given only 10 flips of evidence.

12.9.8 Visual Intuition

Plot three curves on the same axes (

p

from 0 to 1 on x-axis, density on y-axis): - Prior:

Beta (1, 1)

— a flat line at height 1 - Likelihood: Binomial with 7/10 — a bell-shaped curve peaking at

p = 0.7

- Posterior:

Beta (8, 4)

— a bell-shaped curve peaking at

p = 0.7

, but slightly narrower than the likelihood because the prior (even uniform) adds information With an informative

Beta (5, 5)

prior, the posterior peak shifts left to 0.611 — the prior pulls it toward 0.5. Takeaway: the posterior is a compromise between prior and data.

12.9.9 Assumptions & Scope

Assumptions. The Beta-Binomial MAP example requires: 1. Independent coin flips — each toss outcome is independent of others. 2. Constant bias — the probability

p

does not change across flips (no drift, no learning). 3. Conjugate prior — Beta is the conjugate prior for the Binomial likelihood. For other likelihoods, you need the appropriate conjugate prior (Normal for Normal mean, Gamma for Poisson rate, Dirichlet for Multinomial, etc.). When MAP diverges from MLE: The prior matters most when data is scarce. With

n \to \infty

, the likelihood dominates and MAP → MLE regardless of the prior (as long as the prior is not zero anywhere the true parameter could be).

12.9.10 Pitfalls

1. Using the mode formula when $α \leq 1$ or $β \leq 1$ . The formula

\frac{α - 1}{α + β - 2}

only works when

α > 1

and

β > 1

. For

Beta (0.5, 0.5)

, the mode is at the boundaries (0 and 1) — the derivative test fails. 2. Confusing the mode with the mean. The Beta mean is

\frac{α}{α + β}

, not

\frac{α - 1}{α + β - 2}

. For

Beta (8, 4)

, the mean is

8/12 \approx 0.667

while the mode is

7/10 = 0.7

. They are different. 3. Thinking MAP = full Bayesian inference. MAP gives a point estimate. The full posterior

Beta (8, 4)

gives you an entire distribution — you can compute credible intervals, variance, and the probability that

p > 0.5

. MAP throws away all that information.

12.9.11 Exam Note

Exam note: If the problem involves a different scenario (not coin toss), the distribution to use for prior and likelihood will be specified. However, know the Beta-Binomial pair — it is the most common conjugate family and a standard exam question. Keep PDF/PMF formulas handy for open-book exams.

12.9.12 Recap & Bridge

Beta-Binomial is the canonical MAP example:

α_{post} = α_{prior} + heads

β_{post} = β_{prior} + tails

. The mode formula gives the MAP point estimate. Next, we move from estimating parameters to making classifications — the Bayes Optimal Classifier.

12.9.13 Real-World & Domain Connection

The Beta-Binomial model is the workhorse of online A/B testing. When a website tests a new button color, it starts with a Beta(1,1) prior (or an informative prior from previous tests). As users click or do not click, the posterior updates in real time:

α \leftarrow α + clicks

β \leftarrow β + non-clicks

. The MAP estimate gives the current best guess of the click-through rate. This same framework powers Thompson sampling for multi-armed bandits, adaptive clinical trials, and recommendation system exploration.

12.10 Bayesian Optimal Classifier

Hook. If MAP picks the single best hypothesis and uses it to classify, is that the best we can do? Surprisingly, no. You can do better by asking ALL plausible hypotheses to vote, each weighted by how probable it is. The Bayesian Optimal Classifier never makes a mistake it could have avoided — it is theoretically unbeatable.

12.10.1 Intuition — A Committee of Experts

The analogy: getting a second (and third) opinion. You face a medical decision. Doctor A (the MAP hypothesis) is the most trusted expert and says "surgery." But you also consult Doctor B, who is slightly less trusted, and Doctor C, who is equally trusted as B. Doctor B says "medication." Doctor C says "medication." The weighted vote is 0.4 for surgery vs. 0.6 for medication. You go with medication — even though the single most trusted doctor disagreed. The Bayesian Optimal Classifier does exactly this: it polls every hypothesis in the space, weights each by its posterior probability, and picks the majority verdict. MAP picks one doctor; the optimal classifier convenes the whole panel.

12.10.2 Formal Definition

For a new query point

X

, the Bayesian Optimal Classifier computes the posterior probability of each possible class

C_{k}

P (C_{k} ∣ X) = \frac{P ( X ∣ C _{k} ) \cdot P ( C _{k} )}{P ( X )}

Then it picks the class with the maximum posterior:

y^= ar g k max P (C_{k} ∣ X)

More generally, when the hypothesis space contains multiple models

h \in H

, the probability of class

v_{j}

is the weighted sum over all hypotheses:

P (v_{j} ∣ D) = h_{i} \in H \sum P (v_{j} ∣ h_{i}) \cdot P (h_{i} ∣ D)

Each hypothesis votes for its predicted class, weighted by how probable that hypothesis is given the training data.

12.10.3 Worked Example — Weather Prediction

Setup. Three weather models have been developed. Given that it is cloudy:

Hypothesis	Prediction (rain if cloudy)	Posterior $P (H ∣ D)$
H₁ (Rainy model)	100% chance of rain	0.4
H₂ (Unsure model)	50% chance of rain	0.3
H₃ (Sunny model)	0% chance of rain	0.3

Question: It is cloudy. Predict rain or no rain. Bayesian Optimal Classifier — weighted vote across ALL hypotheses: Total probability of rain:

P (rain ∣ cloudy) = P (rain ∣ H_{1}) \cdot P (H_{1}) + P (rain ∣ H_{2}) \cdot P (H_{2}) + P (rain ∣ H_{3}) \cdot P (H_{3}) = 1.0 \times 0.4 + 0.5 \times 0.3 + 0.0 \times 0.3 = 0.40 + 0.15 + 0.00 = 0.55

Total probability of no rain:

P (no rain ∣ cloudy) = 0.0 \times 0.4 + 0.5 \times 0.3 + 1.0 \times 0.3 = 0.00 + 0.15 + 0.30 = 0.45

Decision:

0.55 > 0.45

→ predict rain. Contrast with MAP: The MAP hypothesis is H₁ (posterior 0.4), which also predicts rain. But the optimal classifier gives 55% confidence vs. H₁'s 40% — by incorporating H₂'s uncertain 50% prediction. What if the hypotheses were H₁(rain, 0.4), H₂(no rain, 0.35), H₃(no rain, 0.25)? MAP would predict rain (H₁ wins). But the optimal classifier: rain = 0.4, no rain = 0.6 → predicts no rain. MAP can be wrong even when the best hypothesis is right — because the other hypotheses collectively disagree.

12.10.4 Gibbs Classifier — A Cheaper Alternative

The Gibbs classifier avoids summing over all hypotheses. Instead, it randomly samples ONE hypothesis from the posterior distribution and uses that hypothesis's prediction. In the weather example, Gibbs would: - Pick H₁ with 40% probability → predict rain - Pick H₂ with 30% probability → predict rain (50% of the time), no rain (50%) - Pick H₃ with 30% probability → predict no rain Gibbs is less accurate but much faster. Its expected error is at most twice the Bayes optimal error.

Exam note: Gibbs classifier is not in the syllabus but was mentioned as context. Focus on the Bayesian Optimal Classifier and Naive Bayes.

12.10.5 Visual Intuition

Imagine three experts standing at a whiteboard. Each writes their prediction and confidence level. The Bayesian Optimal Classifier is the moderator who collects all three predictions, weights each by the expert's credibility (posterior probability), and computes the weighted average. If two less-credible experts agree against the most credible one, the moderator sides with the majority. Takeaway: the optimal classifier never trusts any single expert absolutely — it always listens to the full panel.

12.10.6 Assumptions & Scope

Assumptions. 1. Known posterior over hypotheses — you must be able to compute

P (h_{i} ∣ D)

for every hypothesis. This requires a prior and a likelihood model. 2. Complete hypothesis space — the true data-generating model must be in

H

. If not, even the Bayes optimal classifier can be wrong. 3. Computational feasibility — summing over all hypotheses is exponential in the size of the hypothesis space. The Bayes optimal classifier is a theoretical gold standard, not always a practical algorithm. Why it is "optimal": No other classifier using the same hypothesis space and prior knowledge can achieve a lower expected misclassification rate. It achieves the Bayes error rate — the irreducible minimum.

12.10.7 Pitfalls

1. Confusing the Bayes Optimal Classifier with MAP classification. MAP uses the single best hypothesis. The optimal classifier uses ALL hypotheses weighted by posterior. They can disagree — and when they do, the optimal classifier is correct more often on average. 2. Thinking it is practical for large hypothesis spaces. With 100 binary features, the hypothesis space has

2^{100}

candidates. Weighted voting across all of them is impossible. This is why simpler methods like Naive Bayes and Gibbs exist. 3. Forgetting that optimality is relative to the prior. If your prior is wrong, the Bayes optimal classifier can be consistently wrong. "Optimal" means "best given your assumptions," not "objectively correct."

12.10.8 Recap & Bridge

The Bayesian Optimal Classifier polls ALL hypotheses, weights by posterior, and picks the majority class. It is theoretically optimal but computationally prohibitive. Naive Bayes (next) simplifies this dramatically — at the cost of a "naive" independence assumption.

12.10.9 Real-World & Domain Connection

The Bayes Optimal Classifier is the theoretical ceiling for any classification system. In medical diagnosis, researchers compare their practical models against the Bayes error rate to measure how much room for improvement remains. In ensemble methods like random forests and gradient boosting, the idea of "weighted voting across multiple models" directly echoes the Bayesian Optimal Classifier — each tree is a hypothesis, and the ensemble averages their predictions. The optimal classifier provides the theoretical justification for why ensembles work.

12.11 Generative vs Discriminative Models

Hook. Two ways to classify an email as spam. Way 1: learn what separates spam from ham — look for a boundary (discriminative). Way 2: learn what spam emails look like and what ham emails look like — model each class's distribution, then compare (generative). Same goal, opposite strategies. Understanding the difference tells you which model to reach for.

12.11.1 Intuition — Drawing a Line vs. Modeling a Population

The analogy: border control vs. census. A discriminative model is like a border guard who only cares about which side of a line you are on. The guard does not need to know everything about your country — just whether you belong on this side or that side. A generative model is like a census bureau that builds a detailed profile of each country's population: average income, age distribution, common languages. When a new person arrives, the bureau compares them to both profiles and assigns them to the better-matching country. Where the analogy breaks: generative models can also generate new data (hence the name) — they can produce realistic synthetic emails, images, or text. Discriminative models cannot.

12.11.2 Formal Definition

- Discriminative models learn the decision boundary directly. They model

P (Y ∣ X)

— the probability of the class given the features. Examples: logistic regression, SVM, neural networks. - Generative models learn the joint distribution

P (X, Y)

or the class-conditional distribution

P (X ∣ Y)

. They then use Bayes theorem to compute

P (Y ∣ X)

. Examples: Naive Bayes, Gaussian Mixture Models, Hidden Markov Models. The generative process: learn

P (X ∣ Y)

and

P (Y)

→ apply Bayes theorem → classify using

P (Y ∣ X)

12.11.3 Comparison Table

	Discriminative	Generative
What is modeled?	$P (Y ∣ X)$ directly	$P (X ∣ Y)$ and $P (Y)$ , then Bayes
Goal	Find decision boundary	Learn data distribution per class
Can generate new data?	No	Yes
Sample efficiency	Needs more data	Can work with less data
Handling missing features	Harder	Natural (marginalize out)
Outlier detection	No built-in mechanism	Yes (low $P (X)$ = outlier)
Asymptotic accuracy	Generally higher	May be lower if model is misspecified
Examples	Logistic regression, SVM, NN	Naive Bayes, LDA, GMM, HMM

When to pick which: Use discriminative models when you have plenty of labeled data and only care about prediction accuracy. Use generative models when data is scarce, when you need to handle missing values, when you need to detect outliers, or when you want to generate new samples.

12.11.4 Visual Intuition

Picture a scatter plot with two classes: red circles and blue triangles. A discriminative model draws a single line (or curve) separating red from blue — it only cares about the boundary. A generative model draws two contour maps — one showing where red circles tend to cluster, another showing where blue triangles cluster. To classify a new point, the generative model checks which contour map gives the point a higher density. Takeaway: discriminative = one dividing line; generative = two density maps.

12.11.5 Assumptions & Scope

Scope. The discriminative/generative distinction applies to classification problems. For regression, most models are discriminative (model

Y ∣ X

directly). Generative models require modeling the full joint distribution

P (X, Y)

, which is harder — you must specify distributions for

X

, not just for

Y ∣ X

. When generative beats discriminative: With small training sets, generative models can outperform because they make stronger assumptions about the data structure (e.g., features are conditionally independent). These assumptions act as a regularizer. With infinite data, a well-specified discriminative model will match or beat a generative model.

12.11.6 Pitfalls

1. Using a generative model when you only need predictions. If your sole goal is classification accuracy and you have plenty of data, a discriminative model is usually simpler and more accurate. 2. Assuming generative models are always "Bayesian." Naive Bayes is generative and uses Bayes theorem, but not all generative models are Bayesian in the sense of using priors over parameters. GMM trained with EM is generative but frequentist. 3. Ignoring the generative capability. If you need to simulate data (e.g., for data augmentation, anomaly detection, or understanding the data-generating process), you need a generative model.

12.11.7 Recap & Bridge

Discriminative = model the boundary (

P (Y ∣ X)

). Generative = model each class (

P (X ∣ Y)

) and apply Bayes. Naive Bayes, the subject of the next section, is the canonical generative classifier — it models

P (X ∣ Y)

under a strong independence assumption.

12.11.8 Real-World & Domain Connection

The generative/discriminative divide shapes modern AI. GANs (Generative Adversarial Networks) and diffusion models (Stable Diffusion, DALL-E) are generative — they model

P (X)

P (X ∣ Y)

to create new images. Most production classifiers (spam filters, fraud detection) are discriminative — they model

P (Y ∣ X)

for speed and accuracy. The choice between them is one of the first architectural decisions in any ML project.

12.12 Naive Bayes Classifier

Hook. You want to classify emails as spam or not-spam. There are thousands of words. Modeling their joint distribution exactly would require estimating the probability of every possible word combination — a number larger than the atoms in the universe. Naive Bayes sidesteps this with one "naive" trick: pretend all words are independent given the class. It is mathematically wrong. It is practically brilliant.

12.12.1 Intuition — The "Naive" Independence Bet

The analogy: diagnosing a patient by treating symptoms as unrelated. A doctor sees fever, cough, and fatigue. In reality, these symptoms are correlated — the flu often causes all three together. Naive Bayes says: "For simplicity, let me pretend fever, cough, and fatigue are independent given the disease. I will just count how often each symptom appears with each disease and multiply." Shockingly, this simplification often gives the correct diagnosis — because the relative ranking of disease probabilities is usually preserved, even if the absolute probabilities are wrong. Why it works: Naive Bayes does not need accurate probability estimates — it only needs the CORRECT CLASS to have the highest score. The independence assumption biases all classes similarly, so the ranking often survives.

12.12.2 The Conditional Independence Assumption

The defining assumption of Naive Bayes: all features are conditionally independent given the class label.

P (X_{1}, X_{2}, \dots, X_{n} ∣ Y = y) = P (X_{1} ∣ Y = y) \cdot P (X_{2} ∣ Y = y) \cdot \dots \cdot P (X_{n} ∣ Y = y)

In words: once you know the class, knowing one feature tells you nothing about any other feature. This assumption is almost always false. Fever and cough are correlated given the flu. The words "machine" and "learning" co-occur in documents about AI. That is why the classifier is called "naive" — it naively ignores these dependencies. Yet it often works. The estimates of

P (X_{i} ∣ Y)

are individually accurate (estimated from data), and the product, while not a valid joint probability, still orders classes correctly.

12.12.3 The Naive Bayes Formula

Under the naive independence assumption, the posterior probability for class

y

is:

P (Y = y ∣ X_{1}, \dots, X_{n}) \propto P (Y = y) \cdot i = 1 \prod n P (X_{i} ∣ Y = y)

P (Y = y)

— Class prior: how often class

y

occurs in the training data -

\prod_{i = 1}^{n} P (X_{i} ∣ Y = y)

— Likelihood product: multiply the probabilities of seeing each feature value, given class

y

P (X)

— the denominator, same for all classes being compared → can be ignored (the

\propto

) The prediction rule:

y^= ar g y max P (Y = y) \cdot i = 1 \prod n P (X_{i} ∣ Y = y)

12.12.4 The Four Steps of Naive Bayes

1. Calculate prior probabilities — for each class, count its samples and divide by total:

P (Y = y) = \frac{Number of samples with class y}{Total number of samples}

2. Calculate likelihoods — for each feature value in the query point, count matching samples per class:

P (X_{i} = x_{i} ∣ Y = y) = \frac{Number of samples with X _{i} = x _{i} AND class y}{Number of samples with class y}

Only compute likelihoods for the specific feature values in your query — not the entire table. 3. Compute the (unnormalized) posterior — multiply prior × likelihood product for each class. 4. Predict — pick the class with the highest score.

12.12.5 The Zero-Probability Problem and Laplace Smoothing

The trap: If a feature value never appears with a certain class in the training data, its likelihood is

0/5 = 0

. Multiplying by zero zeros out the entire posterior for that class — even if all other features strongly support it. This is catastrophic. The fix: Laplace (add-one) smoothing. Add 1 to every count:

P (X_{i} = x_{i} ∣ Y = y) = \frac{Count ( X _{i} = x _{i} , Y = y ) + 1}{Count ( Y = y ) + k}

where

k

is the number of distinct values feature

X_{i}

can take. This ensures no probability is ever zero. The "+1" acts like a tiny uniform prior over feature values. For the Play Tennis example, if we had never seen Wind=Strong with PlayTennis=No, the unsmoothed estimate would be 0/5 = 0. With Laplace smoothing (Wind has 2 values):

(0 + 1) / (5 + 2) = 1/7 \approx 0.143

— a small but nonzero probability.

12.12.6 Visual Intuition

Picture a 2D grid of data points colored by class (red = Yes, blue = No). A discriminative model draws a single line separating red from blue. A Naive Bayes model instead computes two independent histograms — one for feature X₁ and one for feature X₂ — separately for each class. To classify a new point, it checks: how often does this X₁ value appear in the red histogram? How often in blue? Same for X₂. It multiplies these per-feature scores (with the class prior) and picks the winner. Takeaway: Naive Bayes treats each feature as an independent voter; it multiplies their votes, ignoring any coordination between them.

12.12.7 Assumptions & Scope

Assumptions. 1. Conditional independence — the "naive" assumption. Violated in almost all real datasets. The classifier may still work if the independence violations are symmetric across classes. 2. Categorical features — the basic Naive Bayes works with discrete features. For continuous features, use Gaussian Naive Bayes (model each

P (X_{i} ∣ Y)

as a normal distribution) or discretize the features. 3. Sufficient data per class-feature combination — with many features and small data, many combinations have zero counts. Use Laplace smoothing. When Naive Bayes works well: text classification (spam filtering, sentiment analysis, document categorization), medical diagnosis with discrete symptoms, recommendation systems. The independence assumption is least harmful when features are numerous and individually weak (each word contributes a small signal). When it fails: when features are strongly correlated and the correlations differ by class. For example, if feature A and feature B are positively correlated in class 1 but negatively correlated in class 2, Naive Bayes will double-count the evidence.

12.12.8 Pitfalls

1. Zero probabilities without smoothing. A single unseen feature-value combination can zero out a class's posterior. Always use Laplace smoothing or another smoothing method. 2. Underflow from multiplying many small probabilities. Multiply 1000 numbers each around 0.01, and the result underflows to zero. Solution: work in log-space — sum log-probabilities instead of multiplying probabilities. 3. Thinking Naive Bayes is always a bad model because the assumption is false. The assumption is wrong, but the classifier is often right. In text classification, Naive Bayes routinely competes with sophisticated deep learning models. 4. Applying Naive Bayes to regression problems. Naive Bayes is a classifier (predicts discrete classes). For continuous outputs, use Bayesian regression models instead.

12.12.9 Recap & Bridge

Naive Bayes = class prior × product of per-feature likelihoods. The independence assumption is false but the classifier works. The four-step recipe (priors → likelihoods → product → argmax) is your exam template. Next we run through the canonical Play Tennis example.

12.12.10 Real-World & Domain Connection

Naive Bayes is the engine behind most spam filters. When Gmail classifies an email, it computes

P (spam ∣ words)

using exactly the formula above, with each word as a feature. It is fast enough to run on every incoming email, requires no GPU, and updates incrementally as you mark messages as spam. The same algorithm powers sentiment analysis (positive/negative reviews), news categorization, and even medical diagnosis systems in low-resource settings where deep learning is infeasible.

12.13 Naive Bayes — Worked Example (Play Tennis)

Hook. Here is the classic Naive Bayes example — predicting whether someone will play tennis based on the weather. Four features, 14 training examples, one query. The math is all counting and multiplying. Master this example and you can solve any Naive Bayes problem.

12.13.1 Dataset

Day	Outlook	Temperature	Humidity	Wind	PlayTennis
1	Sunny	Hot	High	Weak	No
2	Sunny	Hot	High	Strong	No
3	Overcast	Hot	High	Weak	Yes
4	Rain	Mild	High	Weak	Yes
5	Rain	Cool	Normal	Weak	Yes
6	Rain	Cool	Normal	Strong	No
7	Overcast	Cool	Normal	Strong	Yes
8	Sunny	Mild	High	Weak	No
9	Sunny	Cool	Normal	Weak	Yes
10	Rain	Mild	Normal	Weak	Yes
11	Sunny	Mild	Normal	Strong	Yes
12	Overcast	Mild	High	Strong	Yes
13	Overcast	Hot	Normal	Weak	Yes
14	Rain	Mild	High	Strong	No

Query point: Outlook = Sunny, Temperature = Cool, Humidity = High, Wind = Strong. Predict: PlayTennis = Yes or No?

12.13.2 Step 1 — Prior Probabilities

Total samples: 14. Yes: 9, No: 5.

P (Yes) = \frac{9}{14} \approx 0.643

P (No) = \frac{5}{14} \approx 0.357

12.13.3 Step 2 — Likelihoods

For each feature value in the query, count how many rows match BOTH the feature value AND the class. Outlook = Sunny:

Class	Matching rows	Total in class	Likelihood
Yes	Days 9, 11 → 2	9	$2/9 \approx 0.222$
No	Days 1, 2, 8 → 3	5	$3/5 = 0.600$

Temperature = Cool:

Class	Matching rows	Total in class	Likelihood
Yes	Days 5, 7, 9 → 3	9	$3/9 \approx 0.333$
No	Day 6 → 1	5	$1/5 = 0.200$

Humidity = High:

Class	Matching rows	Total in class	Likelihood
Yes	Days 3, 4, 12 → 3	9	$3/9 \approx 0.333$
No	Days 1, 2, 8, 14 → 4	5	$4/5 = 0.800$

Wind = Strong:

Class	Matching rows	Total in class	Likelihood
Yes	Days 7, 11, 12 → 3	9	$3/9 \approx 0.333$
No	Days 2, 6, 14 → 3	5	$3/5 = 0.600$

12.13.4 Step 3 — Compute Posterior (Unnormalized)

For Yes:

P (Yes ∣ X) \propto P (Yes) \times P (Sunny ∣ Yes) \times P (Cool ∣ Yes) \times P (High ∣ Yes) \times P (Strong ∣ Yes) = \frac{9}{14} \times \frac{2}{9} \times \frac{3}{9} \times \frac{3}{9} \times \frac{3}{9} = \frac{9 \times 2 \times 3 \times 3 \times 3}{14 \times 9 \times 9 \times 9 \times 9} = \frac{162}{91854} \approx 0.00176

For No:

P (No ∣ X) \propto P (No) \times P (Sunny ∣ No) \times P (Cool ∣ No) \times P (High ∣ No) \times P (Strong ∣ No) = \frac{5}{14} \times \frac{3}{5} \times \frac{1}{5} \times \frac{4}{5} \times \frac{3}{5} = \frac{5 \times 3 \times 1 \times 4 \times 3}{14 \times 5 \times 5 \times 5 \times 5} = \frac{180}{8750} \approx 0.0206

12.13.5 Step 4 — Predict

Since

0.0206 > 0.00176

y^= ar g y \in {Yes, No} max P (y ∣ X) = No

The Naive Bayes classifier predicts: Do NOT play tennis.

12.13.6 Normalized Probabilities (For Completeness)

To get actual posterior probabilities, divide each unnormalized score by their sum:

P (Yes ∣ X) = \frac{0.00176}{0.00176 + 0.0206} \approx 0.079

P (No ∣ X) = \frac{0.0206}{0.00176 + 0.0206} \approx 0.921

There is a 92.1% probability of "No" — a confident prediction.

12.13.7 Why No Won — Feature-by-Feature Analysis

Let us see which features drove the decision:

Feature	Favors	Why
Outlook=Sunny	No	3/5 No vs. 2/9 Yes — Sunny days are mostly No
Temperature=Cool	Yes	3/9 Yes vs. 1/5 No — Cool days are mostly Yes (but weak signal)
Humidity=High	No	4/5 No vs. 3/9 Yes — High humidity strongly favors No
Wind=Strong	No	3/5 No vs. 3/9 Yes — Slightly favors No

Three features favor No; only one (Cool temperature) favors Yes. The cumulative evidence correctly points to No.

12.13.8 Practical Tip — Only Compute What You Need

12.13.9 Visual Intuition

Imagine four dials (Outlook, Temperature, Humidity, Wind), each with settings. For each class (Yes/No), you have a histogram showing how often each setting appeared. The query point sets all four dials to specific positions. For Yes: look up how often Sunny appears in the Yes-Outlook histogram → 2/9. Repeat for each dial. Multiply all four fractions with the Yes prior. Do the same for No. The class with the larger product wins. Takeaway: Naive Bayes is just counting and multiplying — no optimization, no backpropagation, no matrix inversion.

12.13.10 Assumptions & Scope

Assumptions in this example: 1. Conditional independence — Outlook, Temperature, Humidity, and Wind are treated as independent given PlayTennis. In reality, Hot temperature and High humidity often co-occur (violating independence), but the classifier still works. 2. Discrete features — all features are categorical. For continuous features (e.g., temperature in degrees), use Gaussian Naive Bayes or discretize. 3. Sufficient data — with only 14 examples, some combinations (Temperature=Cool | No) have only 1 observation. The estimate 1/5 is noisy. Laplace smoothing would help.

12.13.11 Recap & Bridge

The Play Tennis example is the canonical Naive Bayes template: count priors, count per-feature likelihoods for the query, multiply, argmax. This four-step process works for any discrete-feature classification problem. Next, we summarize the key comparisons across MLE, MAP, and Naive Bayes.

12.13.12 Real-World & Domain Connection

The Play Tennis dataset is a miniature version of how real recommendation systems work. Replace "Outlook" with "User's age group," "Temperature" with "Time of day," "Humidity" with "Device type," and "Wind" with "Previous purchase" — and you have a Naive Bayes product recommender. The same counting-and-multiplying logic scales to millions of users and items, running in milliseconds per prediction. When interpretability matters (e.g., explaining why a loan was denied), Naive Bayes shines — each feature's contribution is a simple multiplicative factor that anyone can understand.

12.14 Summary — Key Comparisons

Here is the entire lecture in one table. MLE finds the parameters that best explain the data. MAP adds a prior belief. Naive Bayes uses Bayes theorem with a "naive" independence assumption to make class predictions.

Aspect	MLE	MAP	Naive Bayes
What it maximizes	Likelihood $P (D ∣ θ)$	Posterior $P (θ ∣ D)$	Posterior $P (y ∣ X)$
Uses prior?	No	Yes (parameter prior)	Yes (class prior)
Key formula	$ar g max_{θ} P (D ∣ θ)$	$ar g max_{θ} P (D ∣ θ) P (θ)$	$ar g max_{y} P (y) \prod P (x_{i} ∣ y)$
Output	Parameter values	Parameter values	Class prediction
Small-sample behavior	Can overfit (extreme estimates)	Prior regularizes	Works surprisingly well
Computational cost	Low (if closed-form)	Low (if conjugate prior)	Very low (just counting)
When to use	Large $n$ , no prior knowledge	Small $n$ , strong priors	Text classification, categorical features

12.14.1 The Big Picture Arc

1. Probability vs Likelihood — the arrow direction: probability goes model→data, likelihood goes data→model. 2. IID — the assumption that lets us multiply individual probabilities into a joint likelihood. 3. MLE — four-step recipe: assume distribution, write likelihood, take log, differentiate, solve. 4. MLE for Normal — the sample mean is the MLE. This is WHY we use averages. 5. MLE for Linear Regression — maximizing likelihood = minimizing RSS = OLS. 6. MLE for Logistic Regression — maximizing likelihood = minimizing cross-entropy. No closed form. 7. MAP — MLE + prior. Posterior ∝ Likelihood × Prior. 8. Beta-Binomial MAP — the canonical conjugate pair. Mode =

(α - 1) / (α + β - 2)

. 9. Bayes Optimal Classifier — polls all hypotheses, weighted by posterior. Theoretically unbeatable. 10. Naive Bayes — assumes conditional independence. False assumption, useful classifier. 11. Play Tennis — the canonical Naive Bayes worked example. Count, multiply, argmax.

12.14.2 The Three Key Equivalences

1. MLE (Normal) = Sample mean → it proves why we use averages. 2. MLE (Linear Regression) = Ordinary Least Squares → it proves why we minimize RSS. 3. MLE (Logistic Regression) = Binary Cross-Entropy minimization → it proves why we use that loss.

12.15 Exam Guidance Summary

Exam note: The professor gave specific guidance for each major topic. Follow these tips to maximize marks.

1. MLE problems — Expect numerical problems where you are given data and a distribution (possibly custom, like the worked example in Section 12.7). Follow the four-step recipe. Always begin your answer with: "Consider a sample of $n$ IID random variables drawn from a distribution…" — omitting this loses marks. 2. MAP problems — Coin toss / Beta-Binomial problems are the standard exam type. Know that Beta is the conjugate prior for the Binomial likelihood. The mode formula for the Beta distribution is

\frac{α - 1}{α + β - 2}

(only valid when

α > 1

and

β > 1

). 3. Naive Bayes problems — Expect a dataset like Play Tennis. Calculate prior probabilities, then likelihoods only for the query point's feature values, multiply, and predict the class with the highest posterior. Do not waste time computing the full conditional probability table. 4. Distribution formulas — Keep the PDF and PMF of common distributions (Normal, Bernoulli, Binomial, Beta, Poisson, Exponential) handy for open-book examinations. You will need to substitute them into the likelihood function. 5. Gradient descent in logistic regression — You may be asked for the gradient update rule:

θ^{new} = θ^{old} + α \sum (Y_{i} - h_{θ} (X_{i})) X_{i}

. This IS the MLE result for logistic regression, since no closed-form solution exists. 6. Conceptual questions — Expect questions comparing: - MLE vs MAP (prior vs. no prior) - Probability vs likelihood (arrow direction) - Discriminative vs generative models (

P (Y ∣ X)

vs.

P (X ∣ Y)

) - Why we use averages (because they are MLE under normality) 7. Study advice — First refresh probability concepts (conditional probability, PDF, PMF, Bayes theorem). Then study MLE for linear regression side-by-side with the linear regression slides. Study MLE for logistic regression side-by-side with the logistic regression slides. This parallel study builds complete understanding of why the loss functions exist. 8. Key connections to memorize: - Maximizing log-likelihood in linear regression = minimizing sum of squared errors (RSS) - Maximizing log-likelihood in logistic regression = minimizing binary cross-entropy loss - MLE for normal mean = sample mean (average) - MLE for normal variance = sample variance (divide by

n

) - MAP

\propto

Likelihood

\times

Prior

12.16 Key Industry Applications

- Bayesian inference powers A/B testing platforms (Optimizely, Google Optimize), medical diagnosis systems, spam filters (Gmail, Outlook), and recommendation engines. Any system that updates beliefs as new evidence arrives is running Bayes theorem under the hood. - Naive Bayes is the workhorse of text classification: spam detection (filtering billions of emails daily), sentiment analysis (brand monitoring on social media), document categorization (legal discovery, news routing), and intent classification in chatbots. Its speed and interpretability make it the first model tried in any text classification task. - MLE underpins parameter estimation in virtually every statistical ML model. Every time you call `.fit()` in scikit-learn, statsmodels, or R, MLE (or a variant) is running. Linear regression, logistic regression, Poisson regression, survival analysis, and structural equation models all use MLE. In deep learning, the loss function is the negative log-likelihood — MLE is training. - MAP estimation is essential when domain expertise or historical data provides informative priors. It is used in Bayesian optimization (hyperparameter tuning), hierarchical models (multi-level marketing mix modeling), small-data regimes (rare disease clinical trials), and any setting where overfitting from MLE is a risk. MAP with L2 regularization is equivalent to ridge regression — a connection that unites Bayesian and frequentist thinking. - Beta-Binomial conjugacy specifically drives online controlled experiments: Thompson sampling for multi-armed bandits, adaptive clinical trial design, and real-time personalization systems where the prior is continuously updated as user interactions stream in.

ML Lecture 12 notes · Bayesian Learning — Maximum Likelihood, MAP, and Naive Bayes

Machine Learning· postgraduate· 2026-06-29

Sections Breakdown

112.1 Probability vs Likelihood â€” Core Distinction

12.1 Probability vs Likelihood â€” Core Distinction

212.2 IID Assumption and Joint Likelihood

12.2 IID Assumption and Joint Likelihood

312.3 Maximum Likelihood Estimate (MLE)

12.3 Maximum Likelihood Estimate (MLE)

412.4 MLE for the Normal Distribution

12.4 MLE for the Normal Distribution

512.5 MLE for Simple Linear Regression

12.5 MLE for Simple Linear Regression

612.6 MLE for Logistic Regression

12.6 MLE for Logistic Regression

712.7 MLE â€” Worked Numerical Example (Custom Distribution)

12.7 MLE â€” Worked Numerical Example (Custom Distribution)

812.8 Maximum A Posteriori (MAP) Estimate

12.8 Maximum A Posteriori (MAP) Estimate

912.9 MAP â€” Worked Example (Beta-Binomial Coin Toss)

12.9 MAP â€” Worked Example (Beta-Binomial Coin Toss)

1012.10 Bayesian Optimal Classifier

12.10 Bayesian Optimal Classifier

1112.11 Generative vs Discriminative Models

12.11 Generative vs Discriminative Models

1212.12 Naive Bayes Classifier

12.12 Naive Bayes Classifier

1312.13 Naive Bayes â€” Worked Example (Play Tennis)

12.13 Naive Bayes â€” Worked Example (Play Tennis)

1412.14 Summary â€” Key Comparisons

12.14 Summary â€” Key Comparisons

15Exam Guidance Summary

Exam Guidance Summary

16Key Industry Applications

Key Industry Applications

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Probability vs Likelihood

Must-know: Probability is forward-looking (model to data). Likelihood is backward-looking (data to model). They are numerically equal but functions of different arguments.

$L (θ ∣ X) = P (X ∣ θ)$

⚠️ Top pitfall: Treating likelihood as a probability over parameters. Likelihood does not sum to 1 over \theta.

Self-check: If you fix the data and vary the parameters, are you computing probability or likelihood?

Connects to: IID Assumption, Maximum Likelihood Estimate

IID Assumption

Must-know: IID (Independent and Identically Distributed) lets the joint likelihood factorize into a product of individual probabilities.

$L (θ ∣ X_{1}, \dots, X_{n}) = i = 1 \prod n f (X_{i} ∣ θ)$

⚠️ Top pitfall: Assuming IID without checking for dependencies in the data.

Self-check: Why does the likelihood function become a product under IID?

Connects to: Maximum Likelihood Estimate

Maximum Likelihood Estimate (MLE)

Must-know: MLE finds the parameter values that make the observed data most probable. The four-step recipe: assume distribution, write likelihood, take log, maximize.

$θ^_{MLE} = ar g θ max i = 1 \prod n f (X_{i} ∣ θ)$

⚠️ Top pitfall: Forgetting to check the second derivative to confirm you found a maximum, not a minimum.

Self-check: What is the MLE for the probability of heads in a coin that lands heads 3 times out of 5?

Connects to: MLE for Normal Distribution, MLE for Linear Regression, MLE for Logistic Regression

MLE for the Normal Distribution

Must-know: The MLE for the mean is the sample mean. The MLE for the variance is the sample variance (dividing by n, not n-1).

$μ^_{MLE} = \frac{1}{n} i = 1 \sum n X_{i}, σ^_{MLE}^{2} = \frac{1}{n} i = 1 \sum n (X_{i} - μ^)^{2}$

⚠️ Top pitfall: Using the MLE variance formula for small samples without correction (it is biased downward).

Self-check: Why does the sample mean maximize the likelihood under a normal distribution?

Connects to: MLE for Linear Regression

MLE for Linear Regression

Must-know: Under Gaussian noise, maximizing the log-likelihood is equivalent to minimizing the sum of squared errors (Ordinary Least Squares).

$Y_{i} = β_{0} + β_{1} X_{i} + ϵ_{i}, ϵ_{i} \sim N (0, σ^{2})$

⚠️ Top pitfall: Forgetting that OLS = MLE only under normally distributed errors.

Self-check: What loss function does MLE produce for linear regression under Gaussian noise?

Connects to: MLE for Logistic Regression

MLE for Logistic Regression

Must-know: Logistic regression uses the sigmoid function to map a linear combination to a probability. MLE produces the binary cross-entropy loss.

$h_{θ} (X) = σ (θ^{⊤} X) = \frac{1}{1 + e ^{- θ^{⊤} X}}, ℓ (θ) = \sum [Y lo g h + (1 - Y) lo g (1 - h)]$

⚠️ Top pitfall: No closed-form solution exists — optimization requires gradient ascent.

Self-check: Why is there no closed-form solution for logistic regression MLE?

Connects to: MLE for Linear Regression

Maximum A Posteriori (MAP) Estimate

Must-know: MAP combines the likelihood with a prior distribution. MAP = argmax of posterior = argmax of (likelihood × prior).

$θ^_{MAP} = ar g θ max P (θ ∣ X) = ar g θ max P (X ∣ θ) P (θ)$

⚠️ Top pitfall: When the prior is uniform, MAP reduces to MLE. When data is scarce, the prior dominates.

Self-check: What happens to MAP when the prior is uniform?

Connects to: MLE, Bayesian Optimal Classifier

Bayesian Optimal Classifier

Must-know: The Bayesian Optimal Classifier chooses the class with the highest posterior probability. It is provably optimal — no other classifier can have lower error.

$y^= ar g c max P (Y = c ∣ X)$

⚠️ Top pitfall: The Bayesian classifier is optimal only when the true probabilities are known. In practice, we must estimate them.

Self-check: Why is the Bayesian classifier considered optimal?

Connects to: Naive Bayes Classifier

Generative vs Discriminative Models

Must-know: Discriminative models estimate P(Y|X) directly. Generative models estimate P(X|Y) and P(Y), then use Bayes theorem to get P(Y|X).

$Discriminative: P (Y ∣ X), Generative: P (X ∣ Y) \cdot P (Y)$

⚠️ Top pitfall: Generative models require more data but can handle missing values naturally.

Self-check: Is Naive Bayes a generative or discriminative model?

Connects to: Naive Bayes Classifier

Naive Bayes Classifier

Must-know: Naive Bayes assumes conditional independence of features given the class. Despite this strong assumption, it performs well in many real-world tasks.

$P (Y = c ∣ X) \propto P (Y = c) i = 1 \prod n P (X_{i} ∣ Y = c)$

⚠️ Top pitfall: If a feature value never appears in training for a given class, the entire probability becomes zero. Laplace smoothing fixes this.

Self-check: What is the 'naive' assumption in Naive Bayes?

Connects to: Bayesian Optimal Classifier, Generative vs Discriminative Models

Previous LectureLecture 11 · Instance-Based Learning — Distance-Weighted KNN, Locally Weighted Regression, and Bayesian Learning Foundations

Next LectureLecture 13 · Naive Bayes Classifier and Ensemble Learning

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.