Introduction to Statistical Methods

Gaussian Mixture Models, Maximum Likelihood Estimation, and Hypothesis Testing

📅 Published: 2026-07-07

🎓 Level: postgraduate

👥 Audience: Postgraduate students in statistics and data science

Gaussian Mixture Models, Maximum Likelihood Estimation, and Hypothesis Testing Review

16.1 Gaussian Mixture Models — Recap and Mathematical Representation

Hook. What if a data point sits right in the overlap between two clusters — neither fully in one nor the other? Hard clustering forces you to pick a side and lie about the uncertainty. Gaussian mixture models don't. They give you honest probabilities: "this point is 60% cluster A, 40% cluster B."

16.1.1 What a Gaussian Mixture Model Is

Intuition. Imagine two radio stations broadcasting on nearby frequencies. When you tune your dial, you hear a blend — sometimes one station louder, sometimes the other. A Gaussian mixture model works the same way: it blends bell curves instead of radio signals. Each bell curve is a component, and the blend knob — how loud each component plays — is called the mixing coefficient.

The analogy breaks when components overlap completely: radio signals interfere destructively, but Gaussians just add their probability densities together. Still, the blend idea is what matters.

A Gaussian mixture model (GMM) is a weighted sum of several Gaussian distributions. Each Gaussian has its own parameters — its own mean and its own variance (or covariance matrix for multivariate data). The need for GMMs arises when data clusters overlap — when a point could reasonably belong to more than one cluster.

The core idea: take a set of Gaussians, assign each one a weight — called a mixing coefficient (how "loud" that component is in the overall blend). Sum them to get a single overall distribution. That overall distribution is your GMM.

16.1.2 Hard Clustering vs Soft Clustering

In hard clustering — like k-means — every point belongs to exactly one cluster. You compute Euclidean distance. Whichever centroid is closer, the point goes there. No ambiguity, no probability. That works fine when clusters are well separated.

In soft clustering — which GMM gives you — a point can belong to multiple clusters with different probabilities. This matters when clusters overlap. If a point falls in the overlapping region, hard clustering forces a binary decision. Soft clustering says: this point is 60% likely from cluster A and 40% likely from cluster B. That is a much more honest answer.

The professor gave this real-world example: suppose two companies' share prices both follow normal distributions, but with different means and standard deviations. If you mix the two sets of daily returns and plot them, you get a GMM. Some returns unmistakably belong to company A. Some unmistakably belong to company B. And some could be from either — the soft assignment captures that ambiguity.

Formalize — the GMM as a weighted sum. A GMM with $K$ components builds the overall density $p (x)$ as a convex combination of $K$ Gaussian densities. Each component $k$ gets a weight $π_{k}$ (the mixing coefficient). These weights must be non-negative and sum to 1 — they are probabilities over components.

The model has three families of parameters per component:

Mixing coefficient $π_{k}$ : how much of the overall mixture comes from component $k$ . $π_{k} \in [0, 1]$ , $\sum_{k = 1}^{K} π_{k} = 1$ .
Mean $μ_{k}$ : the center of component $k$ . For $d$ -dimensional data, $μ_{k} \in R^{d}$ .
Covariance matrix $Σ_{k}$ : how component $k$ spreads and rotates in $d$ -dimensional space. $Σ_{k}$ is $d \times d$ , symmetric, positive definite. In one dimension ( $d = 1$ ), it collapses to the scalar variance $σ_{k}^{2}$ .

The full density is:

p (x) = k = 1 \sum K π_{k} N (x ∣ μ_{k}, Σ_{k})

where each component is a multivariate Gaussian:

N (x ∣ μ_{k}, Σ_{k}) = \frac{1}{( 2 π ) ^{d /2} ∣ Σ _{k} ∣ ^{1/2}} exp (- \frac{1}{2} (x - μ_{k})^{⊤} Σ_{k}^{- 1} (x - μ_{k}))

The professor put it simply: "Pi one into the first Gaussian plus Pi two into the second gives P of X." In the 1D case ( $d = 1$ ), the multivariate Gaussian collapses to the familiar bell curve:

f (x) = \frac{1}{σ 2 π} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

As the professor noted: "If you take d equal to 1, then we go back to this simple one random variable which follows Gaussian distribution." The multivariate form is the general case; the 1D form is what you use in exam numerical problems.

Symbol	Meaning	Domain
$K$	Number of Gaussian components	$K \geq 1$
$π_{k}$	Mixing coefficient for component $k$	$π_{k} \in [0, 1]$ , $\sum π_{k} = 1$
$μ_{k}$	Mean vector of component $k$	$μ_{k} \in R^{d}$
$Σ_{k}$	Covariance matrix of component $k$	$d \times d$ , symmetric positive definite
$σ_{k}^{2}$	Variance (1D case only)	$σ_{k}^{2} > 0$
$x$	A data point (observation)	$x \in R^{d}$

16.1.3 The Covariance Matrix — Why It Appears

When you work with a single Gaussian random variable, you have a mean $μ$ and a variance $σ^{2}$ . But with multiple random variables, variance alone is not enough. You need a variance-covariance matrix.

The professor explained: "Instead of Sigma one, it is variance covariance matrix or simply covariance matrix." The $2 \times 2$ case tells the whole story:

Σ = [σ_{11} σ_{21} σ_{12} σ_{22}]

$σ_{11}$ is the variance of $X_{1}$ — how much $X_{1}$ spreads on its own.
$σ_{22}$ is the variance of $X_{2}$ .
$σ_{12} = σ_{21}$ is the covariance between $X_{1}$ and $X_{2}$ — it captures how they move together. Positive covariance means when $X_{1}$ is large, $X_{2}$ tends to be large too.

For $K$ random variables, you get a $K \times K$ matrix. In the 1D case, the matrix collapses to the scalar $σ^{2}$ . In exam problems with one variable, you use $σ^{2}$ ; in real implementations with multiple variables, you use the full covariance matrix.

Assumptions & Scope. The GMM assumes your data actually arises from a mixture of Gaussians. If the true distribution is skewed, heavy-tailed, or has non-elliptical clusters, the model is fundamentally misspecified and will give poor density estimates. The professor flagged this explicitly: "We can't expect that all data are drawn from Gaussian distributions." GMMs also assume you know $K$ — the number of components — in advance. In practice you either pick $K$ by domain knowledge or use model selection criteria like BIC. The model works best when clusters are roughly elliptical and you need soft assignments rather than hard boundaries.

Visual Intuition. Picture a plot with your data variable on the x-axis and probability density on the y-axis. Draw two bell curves centered at different means with different spreads. The taller, narrower curve has a bigger mixing coefficient $π$ . Now sum them point by point along the x-axis — the result is a single wavy curve that might have two humps (bimodal) or one broad hump (if the Gaussians overlap heavily). The GMM density is that summed curve. Where the two component curves cross is where soft clustering matters most — data in that zone gets split probabilities.

Pitfalls.

Confusing hard and soft clustering. K-means gives you one cluster per point; GMM gives you probabilities. Don't treat GMM responsibilities as hard assignments unless you threshold them.
Forgetting the mixing coefficient constraint. The $π_{k}$ must sum to 1. If you initialize them without normalizing, your GMM is not a valid probability distribution.
Thinking $K$ is always obvious. The two-companies example has $K = 2$ by construction, but real data rarely gives you that luxury. Choosing $K$ is a genuine challenge — like choosing the number of clusters in k-means.
Ignoring the Gaussian assumption. GMMs work beautifully when your data is approximately Gaussian. They fail silently when it isn't — always check your data before applying GMMs.

16.1.4 Student Questions and Answers

Q: Is variance covered as part of covariance in the GMM parameter list? Because we said a GMM has mean, variance, covariance, and mixing coefficient.

A: Yes. When you talk about a single random variable, you have mean and variance. When you talk about a combination of Gaussian variables — multivariate — it is not going to be only variance. It is a variance-covariance matrix. So the covariance matrix subsumes the variance. For the univariate case ( $d = 1$ ), the covariance matrix collapses to just $σ^{2}$ .

Q: If we are working with a 1D numerical example (only one variable), why do we not see the covariance matrix in the formula?

A: To avoid complexity. In a 1D numerical example, we just want to understand the algorithm. The full covariance matrix comes in when you have more than two variables. In the 1D case, you just use $σ^{2}$ . In general — when you implement a GMM for real data — you use the full covariance matrix.

Q: Don't we already know the number of Gaussian components $K$ ? If there are two companies with two share prices, $K = 2$ — why is choosing $K$ a challenge?

A: In that simple example, yes, you know $K = 2$ . Those are clear cases. But in real data, you cannot expect the mixture to be that clean. Real data may have many overlapping components, and you do not know $K$ in advance. You have to choose it — just like choosing the number of clusters in k-means. That is one of the two main challenges of GMMs.

Recap. A GMM is a weighted blend of Gaussian bell curves. Each component has a mean, a covariance, and a mixing coefficient. Soft clustering via GMM gives you honest probabilities instead of forced binary assignments. But you need to know $K$ and your data should roughly follow a mixture of Gaussians.

Bridge. Once you have the GMM formula, the hard problem is finding the best parameters — the means, covariances, and mixing coefficients. Direct maximum likelihood doesn't have a closed form because of the sum-inside-a-log. The EM algorithm (next section) solves this by alternating between guessing which component each point came from and updating the parameters.

Real-World & Domain Connection. GMMs power soft clustering in customer segmentation — a shopper who buys both luxury and discount items isn't just "luxury" or "budget," they're a mix. In computer vision, GMMs model background pixels in video surveillance: each pixel's color over time is a mixture of Gaussians, and anything that doesn't fit the mixture is flagged as a moving object. In quantitative finance, GMMs model asset returns from mixed regimes (bull market, bear market, sideways) without needing to know which regime each day belongs to.

16.2 Expectation Maximization (EM) Algorithm for GMM

Hook. You have a GMM with unknown parameters. You want the best means, covariances, and mixing coefficients. Maximum likelihood sounds perfect — until you try to solve it and hit a sum buried inside a logarithm. No algebra trick gets you a clean answer. What do you do? You don't solve it directly. You guess, improve, guess again, improve — that's EM.

16.2.1 Why We Need EM

Purpose. A GMM has three sets of parameters: the means $μ_{k}$ , the covariance matrices $Σ_{k}$ , and the mixing coefficients $π_{k}$ . You want to find the values that maximize the likelihood of your observed data. The log-likelihood of a GMM looks innocent but hides a trap:

lo g p (X ∣ π, μ, Σ) = i = 1 \sum N lo g (k = 1 \sum K π_{k} N (x_{i} ∣ μ_{k}, Σ_{k}))

The sum over $k$ sits inside the logarithm. Differentiating this directly gives coupled equations with no closed-form solution. As the professor put it: "Because of this mathematical representation, it becomes too complex. Very difficult to solve this." EM solves this by introducing a latent variable — "which component did each point come from?" — and alternating between guessing the latent variable and optimizing the parameters.

16.2.2 The EM Algorithm — Inputs, Outputs, and Steps

Inputs & Outputs.

Input: A dataset ${x_{1}, \dots, x_{N}}$ , the number of components $K$ , and a convergence threshold.
Output: Estimated parameters $π^_{k}, μ^_{k}, Σ^_{k}$ for $k = 1, \dots, K$ , plus the responsibility matrix $γ (z_{ik})$ for every point-component pair.

Steps. The professor described it as: "First we initialize the parameters. Then two steps. E step and M step."

Step 0 — Initialization. Pick starting values for all parameters. Random means $μ_{k}$ , identity or diagonal covariances $Σ_{k}$ , and uniform mixing coefficients $π_{k} = 1/ K$ . The mixing coefficients must sum to 1 — if you pick random values, normalize them: $π_{k} \leftarrow π_{k} / \sum_{j} π_{j}$ .

Step 1 — E-step (Expectation). Using the current parameters, compute the responsibility $γ (z_{ik})$ — the posterior probability that data point $i$ came from component $k$ :

γ (z_{ik}) = \frac{π _{k} N ( x _{i} ∣ μ _{k} , Σ _{k} )}{\sum _{j = 1}^{K} π _{j} N ( x _{i} ∣ μ _{j} , Σ _{j} )}

The denominator is the total GMM density at $x_{i}$ — it normalizes the responsibilities so that for each point $i$ , $\sum_{k} γ (z_{ik}) = 1$ . The numerator is component $k$ 's contribution. This is Bayes' theorem: prior $π_{k}$ , likelihood $N (x_{i} ∣ μ_{k}, Σ_{k})$ , posterior $γ (z_{ik})$ .

Step 2 — M-step (Maximization). Using the responsibilities from the E-step, re-estimate the parameters to maximize the expected log-likelihood:

π_{k}^{new} = \frac{N _{k}}{N}, where N_{k} = i = 1 \sum N γ (z_{ik})

μ_{k}^{new} = \frac{1}{N _{k}} i = 1 \sum N γ (z_{ik}) x_{i}

Σ_{k}^{new} = \frac{1}{N _{k}} i = 1 \sum N γ (z_{ik}) (x_{i} - μ_{k}^{new}) (x_{i} - μ_{k}^{new})^{⊤}

Each update is a weighted average. $N_{k}$ is the "effective number of points" assigned to component $k$ . The mean update is a responsibility-weighted average of the data. The covariance update is a responsibility-weighted outer product of deviations. The professor summarized: "Based on E step we go further and update the parameters. In turn, this maximizes the likelihood."

Iterate. Go back to E-step with the new parameters, then M-step again. "Keep on iterating between E step, M step and so on till it reaches the convergence."

Final assignment. Once converged, assign each data point to the component with the highest responsibility $γ (z_{ik})$ . "Wherever the probability is highest, we assign that data point to the corresponding cluster because the probability is more for it."

Trace — one iteration on a tiny 2-point, 2-component problem. Suppose $K = 2$ , two 1D points: $x_{1} = 2.0$ , $x_{2} = 5.0$ . Initialize: $π_{1} = π_{2} = 0.5$ , $μ_{1} = 1.0$ , $μ_{2} = 6.0$ , $σ_{1}^{2} = σ_{2}^{2} = 1.0$ .

E-step. Compute responsibilities for $x_{1} = 2.0$ :

N (2.0 ∣ 1.0, 1.0) = \frac{1}{2 π} e^{- (2.0 - 1.0)^{2} /2} = \frac{1}{2 π} e^{- 0.5} \approx 0.2420

N (2.0 ∣ 6.0, 1.0) = \frac{1}{2 π} e^{- (2.0 - 6.0)^{2} /2} = \frac{1}{2 π} e^{- 8} \approx 0.00013

Denominator: $0.5 \times 0.2420 + 0.5 \times 0.00013 = 0.12107$ . So:

γ (z_{11}) = \frac{0.5 \times 0.2420}{0.12107} \approx 0.999, γ (z_{12}) \approx 0.001

Point $x_{1}$ almost certainly belongs to component 1. For $x_{2} = 5.0$ : the situation reverses — $γ (z_{21}) \approx 0.001$ , $γ (z_{22}) \approx 0.999$ .

M-step. $N_{1} = 0.999 + 0.001 = 1.0$ , $N_{2} = 0.001 + 0.999 = 1.0$ . (Effective counts sum to $N = 2$ .)

μ_{1}^{new} = \frac{0.999 \times 2.0 + 0.001 \times 5.0}{1.0} \approx 2.003

μ_{2}^{new} = \frac{0.001 \times 2.0 + 0.999 \times 5.0}{1.0} \approx 4.997

The means have shifted toward their respective points. $π_{1} = π_{2} = 1.0/2 = 0.5$ (unchanged). Covariances update similarly. The algorithm would iterate until $μ_{1} \to 2.0$ , $μ_{2} \to 5.0$ .

Sense-check: After one iteration, $μ_{1}$ moved from 1.0 to ~2.0 — towards $x_{1}$ . $μ_{2}$ moved from 6.0 to ~5.0 — towards $x_{2}$ . The EM algorithm is doing exactly what it should: pulling each component's mean toward the points it's responsible for.

When to Use / Alternatives.

Use EM for GMM when you need soft cluster assignments and your data is roughly Gaussian. EM is the standard approach — it's guaranteed to improve the likelihood at each step (though it may converge to a local maximum).
Alternatives: K-means is simpler and faster when clusters are spherical and you only need hard assignments — in fact, k-means can be seen as a limiting case of GMM-EM where all covariances approach zero. For non-Gaussian data, consider DBSCAN (density-based) or spectral clustering.
Local optima trap: EM converges to a local maximum of the likelihood, not the global one. Different random initializations can give different results. Run EM multiple times with different starting points and pick the one with the highest final likelihood.

Symbol	Meaning	Domain
$γ (z_{ik})$	Responsibility — posterior probability point $i$ came from component $k$	$[0, 1]$
$x_{i}$	The $i$ -th data point	$R^{d}$
$N$	Total number of data points	$N \geq 1$
$N_{k}$	Effective number of points assigned to component $k$	$N_{k} = \sum_{i} γ (z_{ik})$

16.2.3 GMM Challenges and Limitations

Assumptions & Scope. EM for GMMs inherits all the GMM assumptions from section 16.1. The Gaussian assumption is paramount — if your data is not a mixture of Gaussians, EM will still run and give you answers, but they'll be wrong. EM also assumes the data points are independent and identically distributed (IID). The number of components $K$ must be fixed before running. EM guarantees the likelihood never decreases, but it can converge to a saddle point or a poor local maximum — it does not guarantee finding the global optimum.

Visual Intuition. Picture a scatter plot with two overlapping clouds of points. Draw two ellipses — one for each Gaussian component — centered at initial random positions. The E-step colors every point with a blend of the two ellipse colors, weighted by how close the point is to each ellipse center. The M-step then shifts, rotates, and resizes each ellipse to match the weighted average of the points it "owns." With each iteration, the ellipses settle into the natural clusters. The likelihood curve is like a landscape with hills and valleys — EM is a hiker who takes the steepest uphill step from their starting position, but might end up on a foothill instead of the summit.

Pitfalls.

Thinking EM gives the global optimum. It doesn't. EM finds a local maximum. Run it multiple times with different random starts — the highest final likelihood wins.
Confusing exam mode with implementation mode. In exam problems, parameters are given — you just compute responsibilities. In code, you must iterate E and M until convergence. Don't mix these up.
Forgetting to normalize mixing coefficients. Initialize $π_{k}$ however you want, but always divide by the sum so they add to 1.
Singular covariance. If a component collapses onto a single point (or a few collinear points), its covariance matrix can become singular and the likelihood blows up. This is a known failure mode of GMM-EM — use regularization (add a small constant to the diagonal of $Σ_{k}$ ) to prevent it.

16.2.4 Student Questions and Answers

Q: In numerical problems, the parameter values ( $π_{k}$ , $μ_{k}$ , $σ_{k}^{2}$ ) are given directly — we do not have to iterate EM. Is that correct?

A: Yes. In exam numerical problems, the parameters and mixing coefficients are provided. You just compute the posterior probability (responsibility). But when you go for implementation — actually coding a GMM — you start with random initializations. Then you loop: E-step, M-step, E-step, M-step, updating each time until convergence.

Q: When initializing the mixing coefficients, how do we ensure $π_{1} + π_{2} = 1$ ?

A: You have to take care of that constraint. If $π_{1} = 0.5$ , then $π_{2} = 0.5$ . The sum of all mixing coefficients must be 1. In random initialization, you can pick values and normalize them: divide each by their sum.

Recap. EM solves the GMM parameter estimation problem by alternating between the E-step (compute soft assignments — "guess which component") and the M-step (update parameters — "fit components to their assigned points"). It finds a local maximum of the likelihood, not necessarily the global one.

Bridge. The E-step's responsibility formula is the core computational unit of EM. It's built directly from Bayes' theorem. The next section (16.3) shows you exactly how that mapping works — prior, likelihood, posterior — and gives the 2-component formula you'll use in exam problems.

Real-World & Domain Connection. EM isn't just for GMMs — it's one of the most widely used algorithms in statistics for any problem with latent (hidden) variables. It appears in hidden Markov models (speech recognition), missing data imputation (clinical trials where patients drop out), and topic models like Latent Dirichlet Allocation (document clustering). The GMM-EM combination specifically is the default choice for soft clustering in Python's scikit-learn (sklearn.mixture.GaussianMixture) and is used in bioinformatics for population structure inference from genetic markers.

16.3 GMM — Posterior Probability (Responsibility) Computation

Hook. You know a data point's value and you know the GMM parameters. But you don't know which component generated the point. How do you decide? You don't guess blindly — you compute the probability that each component was the source, given what you observed. That probability is the responsibility.

16.3.1 The Responsibility Formula — Connecting to Bayes' Theorem

Intuition. Think of a detective at a crime scene. Two suspects could have committed the crime (component 1 and component 2). The detective has a prior belief about each suspect's guilt (the mixing coefficients $π_{1}$ and $π_{2}$ ). Then they find a piece of evidence $x$ — say, a fingerprint. The likelihood is "how likely would this fingerprint be if suspect 1 committed the crime?" and similarly for suspect 2. Bayes' theorem combines the prior with the likelihood to give the posterior — the updated probability that each suspect is guilty, given the evidence. That posterior is exactly the GMM responsibility.

The analogy breaks because in a GMM there's no actual "guilty" component — every point truly came from exactly one component (in the generative model), but we can never know which. Still, the detective updating beliefs is the right mental picture.

The professor drew a direct line from Bayes' theorem to the GMM responsibility formula. Here is the full derivation:

Step 1 — Bayes' theorem for two classes. Consider two classes $A$ and $B$ . For evidence $E$ :

P (A ∣ E) = \frac{P ( A ) \cdot P ( E ∣ A )}{P ( E )} = \frac{P ( A ) \cdot P ( E ∣ A )}{P ( A ) \cdot P ( E ∣ A ) + P ( B ) \cdot P ( E ∣ B )}

This is the standard Bayes formula with the total probability $P (E)$ expanded in the denominator.

Step 2 — Map GMM terms to Bayes terms. Let $A$ = "the point came from Gaussian component 1" and $B$ = "came from component 2." Then:

Prior: $P (A) = π_{1}$ (mixing coefficient — how common component 1 is overall)
Prior: $P (B) = π_{2}$
Likelihood: $P (E ∣ A) = N (x ∣ μ_{1}, σ_{1}^{2})$ (how probable the observed $x$ is under component 1's bell curve)
Likelihood: $P (E ∣ B) = N (x ∣ μ_{2}, σ_{2}^{2})$

Step 3 — The responsibility formula. Substitute into Bayes:

γ (z_{1}) = P (component 1 ∣ x) = \frac{π _{1} \cdot N ( x ∣ μ _{1} , σ _{1}^{2} )}{π _{1} \cdot N ( x ∣ μ _{1} , σ _{1}^{2} ) + π _{2} \cdot N ( x ∣ μ _{2} , σ _{2}^{2} )}

For component 2, the numerator changes but the denominator stays the same:

γ (z_{2}) = \frac{π _{2} \cdot N ( x ∣ μ _{2} , σ _{2}^{2} )}{π _{1} \cdot N ( x ∣ μ _{1} , σ _{1}^{2} ) + π _{2} \cdot N ( x ∣ μ _{2} , σ _{2}^{2} )}

The denominator is the total GMM density $p (x)$ at that point — it's the same for all components. It guarantees that $γ (z_{1}) + γ (z_{2}) = 1$ . The numerator is the component-specific contribution.

Formalize — the general K-component responsibility. For $K$ components, the responsibility of component $k$ for data point $x$ is:

γ (z_{k}) = \frac{π _{k} \cdot N ( x ∣ μ _{k} , Σ _{k} )}{\sum _{j = 1}^{K} π _{j} \cdot N ( x ∣ μ _{j} , Σ _{j} )}

The professor summarized it: "This is drawn from our basic understanding of conditional probability and Bayes' theorem. Total probability in the denominator, component probability in the numerator."

Worked mini-example. A 2-component 1D GMM: $π_{1} = 0.6$ , $μ_{1} = 94$ , $σ_{1}^{2} = 50$ ; $π_{2} = 0.4$ , $μ_{2} = 90$ , $σ_{2}^{2} = 25$ . Compute the responsibility for $x = 95$ .

First, compute each Gaussian density at $x = 95$ :

N (95 ∣ 94, 50) = \frac{1}{50 \cdot 2 π} exp (- \frac{( 95 - 94 ) ^{2}}{2 \cdot 50}) = \frac{1}{100 π} e^{- 1/100} \approx \frac{1}{17.725} \cdot 0.99005 \approx 0.05585

N (95 ∣ 90, 25) = \frac{1}{25 \cdot 2 π} exp (- \frac{( 95 - 90 ) ^{2}}{2 \cdot 25}) = \frac{1}{50 π} e^{- 25/50} \approx \frac{1}{12.533} \cdot 0.60653 \approx 0.04839

Now the weighted numerators and denominator:

π_{1} \cdot N_{1} = 0.6 \times 0.05585 \approx 0.03351

π_{2} \cdot N_{2} = 0.4 \times 0.04839 \approx 0.01936

Denominator = 0.03351 + 0.01936 = 0.05287

Finally, the responsibilities:

γ (z_{1}) = \frac{0.03351}{0.05287} \approx 0.634, γ (z_{2}) = \frac{0.01936}{0.05287} \approx 0.366

Verdict: $x = 95$ is about 63% likely from component 1 and 37% from component 2. The point is closer to $μ_{1} = 94$ (distance 1) than to $μ_{2} = 90$ (distance 5), and component 1 has the larger prior (0.6 vs 0.4), so the 63% assignment makes sense.

Assumptions & Scope. The responsibility formula assumes you already have the GMM parameters ( $π_{k}, μ_{k}, σ_{k}^{2}$ ). It is the E-step of EM — you use current parameters to compute soft assignments. The formula is valid for any $K \geq 2$ and any dimension $d$ . In the 1D case ( $d = 1$ ), replace the multivariate Gaussian with the univariate one. The formula always gives probabilities that sum to 1 across components for each point — this is a built-in sanity check. If your $γ (z_{k})$ values don't sum to 1, you made an arithmetic error.

Visual Intuition. Draw the x-axis with two bell curves: a taller one at $μ_{1} = 94$ (weight 0.6) and a shorter one at $μ_{2} = 90$ (weight 0.4). At $x = 95$ , drop a vertical line. This line intersects the scaled curve for component 1 at height $0.6 \times N (95∣94, 50) \approx 0.0335$ and the scaled curve for component 2 at height $0.4 \times N (95∣90, 25) \approx 0.0194$ . The total height of the GMM curve at $x = 95$ is their sum, $0.0529$ . The responsibility $γ (z_{1})$ is the ratio of the component-1 height to the total height — it's the fraction of the total density that component 1 contributes at that x-location.

Pitfalls.

Forgetting the denominator is the same for all components. Compute it once, reuse it. Many students recompute it separately for each component and introduce rounding errors.
Mixing up which Gaussian uses which parameters. Component 1's $N_{1}$ always uses $μ_{1}$ and $σ_{1}^{2}$ . It sounds obvious, but under exam pressure, students swap them.
Neglecting the mixing coefficient. The responsibility is NOT just $N_{1} / (N_{1} + N_{2})$ . You must multiply by the priors $π_{k}$ . A component with a small prior gets a lower responsibility even if the data fits it well.
Using the wrong density formula. For 1D, use $1/ (σ 2 π) e^{- (x - μ)^{2} / (2 σ^{2})}$ . Don't accidentally use the multivariate form with determinants and matrix inverses.

16.3.2 Student Questions and Answers

Q: The denominator formula — is it really $π_{1} N_{1} + π_{2} N_{2}$ ? Earlier it seemed like only $π_{1} N_{1}$ was written in the denominator.

A: No — the denominator is the total probability. It must include both components: $π_{1}$ times the density of the first Gaussian plus $π_{2}$ times the density of the second. That is the total probability. The numerator is only the part for the component you are computing the responsibility for. "This is like our Bayes theorem. Total probability in the denominator, component probability in the numerator."

Recap. The responsibility $γ (z_{k})$ is the posterior probability that component $k$ generated point $x$ . It's Bayes' theorem with mixing coefficients as priors and Gaussian densities as likelihoods. The denominator is the total GMM density — same for all components — guaranteeing the responsibilities sum to 1.

Bridge. Armed with the responsibility formula, you can now solve any GMM classification problem where the parameters are given. The next section (16.4) walks through two full numerical examples — computing responsibilities and assigning points to clusters.

Real-World & Domain Connection. The responsibility concept extends beyond GMMs to any mixture model. In speech recognition, each phoneme is modeled as a GMM, and the responsibility tells you how likely each frame of audio came from each phoneme. In medical diagnosis, a patient's lab result $x$ could come from a "healthy" Gaussian or a "diseased" Gaussian — the responsibility is the probability the patient has the disease given the test result. This is literally Bayesian diagnosis, and it's used in clinical decision support systems.

16.4 GMM Worked Numerical Examples

Hook. You've seen the Bayes-to-GMM mapping. Now the real test: given raw numbers — mixing coefficients, means, variances, and a data point — can you compute the responsibility and classify the point? These are the exact kinds of problems you'll see on the exam. No Z-table, no integration — just exponentials and arithmetic.

16.4.1 Example 1 — Responsibility for $x = 95$

A 1-dimensional GMM has two components with these parameters:

Component 1: $π_{1} = 0.6$ , $μ_{1} = 94$ , $σ_{1}^{2} = 50$
Component 2: $π_{2} = 0.4$ , $μ_{2} = 90$ , $σ_{2}^{2} = 25$

A new data point $x = 95$ arrives. Compute the responsibility that this point belongs to component 1, then classify it.

Step 1 — Write the univariate Gaussian density. For any component with mean $μ$ and variance $σ^{2}$ :

f (x ∣ μ, σ^{2}) = \frac{1}{σ 2 π} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

You'll use this twice — once per component.

Step 2 — Compute each component's density at $x = 95$ .

For component 1 ( $μ_{1} = 94$ , $σ_{1}^{2} = 50$ ):

\frac{( 95 - 94 ) ^{2}}{2 \cdot 50} = \frac{1}{100} = 0.01

\frac{1}{50 \cdot 2 π} = \frac{1}{100 π} = \frac{1}{17.725} \approx 0.05642

f (95 ∣ 94, 50) = 0.05642 \cdot e^{- 0.01} = 0.05642 \cdot 0.99005 \approx 0.05586

For component 2 ( $μ_{2} = 90$ , $σ_{2}^{2} = 25$ ):

\frac{( 95 - 90 ) ^{2}}{2 \cdot 25} = \frac{25}{50} = 0.5

\frac{1}{25 \cdot 2 π} = \frac{1}{50 π} = \frac{1}{12.533} \approx 0.07979

f (95 ∣ 90, 25) = 0.07979 \cdot e^{- 0.5} = 0.07979 \cdot 0.60653 \approx 0.04840

The professor noted: "We don't require any Z distribution table. There's simply 95 minus 94 whole square by 50, e power minus of something, that value. That's it." No integration, no table lookup — just the exponential function.

Step 3 — Weight by mixing coefficients and compute total probability.

π_{1} \cdot f_{1} = 0.6 \times 0.05586 \approx 0.03351

π_{2} \cdot f_{2} = 0.4 \times 0.04840 \approx 0.01936

p (95) = 0.03351 + 0.01936 = 0.05287

Step 4 — Compute responsibilities.

γ (z_{1}) = \frac{0.03351}{0.05287} \approx 0.634

γ (z_{2}) = \frac{0.01936}{0.05287} \approx 0.366

Step 5 — Classify. Since $γ (z_{1}) = 0.634 > γ (z_{2}) = 0.366$ , assign $x = 95$ to component 1. As the professor said: "Based on the probability we assign wherever the probability is highest."

Sense-check. $x = 95$ is 1 unit from $μ_{1} = 94$ and 5 units from $μ_{2} = 90$ . Component 1 also has the larger prior (0.6 vs 0.4). Both facts push the same way — component 1 should win. The 63.4% responsibility is about right for a point that's close but not right at $μ_{1}$ .

16.4.2 Example 2 — Classifying $x = 130$

Two Gaussian components with these parameters:

$π_{1} = 0.3$ , $μ_{1} = 120$ , $σ_{1}^{2} = 100$
$π_{2} = 0.7$ , $μ_{2} = 170$ , $σ_{2}^{2} = 100$

A point $x = 130$ is given. Which component does it belong to?

Step 1 — Compute each density at $x = 130$ .

For component 1 ( $μ_{1} = 120$ , $σ_{1}^{2} = 100$ ):

\frac{( 130 - 120 ) ^{2}}{2 \cdot 100} = \frac{100}{200} = 0.5

\frac{1}{100 \cdot 2 π} = \frac{1}{200 π} = \frac{1}{25.066} \approx 0.03989

f (130 ∣ 120, 100) = 0.03989 \cdot e^{- 0.5} = 0.03989 \cdot 0.60653 \approx 0.02420

For component 2 ( $μ_{2} = 170$ , $σ_{2}^{2} = 100$ ):

\frac{( 130 - 170 ) ^{2}}{2 \cdot 100} = \frac{1600}{200} = 8.0

\frac{1}{100 \cdot 2 π} = 0.03989 (same normalizing constant)

f (130 ∣ 170, 100) = 0.03989 \cdot e^{- 8} = 0.03989 \cdot 0.000335 \approx 0.0000134

Step 2 — Weight and compute total probability.

π_{1} \cdot f_{1} = 0.3 \times 0.02420 \approx 0.007260

π_{2} \cdot f_{2} = 0.7 \times 0.0000134 \approx 0.000009

p (130) = 0.007260 + 0.000009 \approx 0.007269

Step 3 — Compute responsibility.

γ (z_{1}) = \frac{0.007260}{0.007269} \approx 0.999

γ (z_{2}) = \frac{0.000009}{0.007269} \approx 0.001

Step 4 — Classify. The professor stated the probability for cluster 1 came out to approximately 0.9. With $σ_{2}^{2} = 100$ we get 0.999. Either way, $γ (z_{1})$ is overwhelmingly higher. The conclusion: $x = 130$ belongs to cluster 1.

"Probability is more for this — therefore conclusion: X equal to 130 belongs to cluster 1."

Sense-check. $x = 130$ is only 10 units from $μ_{1} = 120$ but 40 units from $μ_{2} = 170$ . Even though component 2 has a larger prior (0.7 vs 0.3), the distance advantage is enormous — the exponential in the Gaussian penalizes large squared distances severely. The classification is unambiguous.

Assumptions & Scope. Both examples are 1D — no covariance matrix, just $σ^{2}$ . This is intentional: exam problems stay 1D to avoid matrix algebra. For real data with $d > 1$ , replace the univariate Gaussian density with the multivariate form and use the full covariance matrix. The responsibility formula structure (weighted likelihood / total probability) remains identical regardless of dimension.

Visual Intuition for Example 2. Draw two bell curves on the same axis. Component 1: centered at 120, spread $σ = 10$ , prior weight 0.3. Component 2: centered at 170, same spread, prior weight 0.7. At $x = 130$ , the component-1 curve is near its peak — about 0.024 — while the component-2 curve is way out in its left tail at $40/ σ = 4$ standard deviations away. Even multiplying by the larger prior (0.7), component 2 contributes almost nothing (0.000009). The GMM density at $x = 130$ is essentially all from component 1. The soft assignment is practically hard in this case — 99.9% vs 0.1%.

Pitfalls.

Forgetting the $\frac{1}{σ 2 π}$ factor in the Gaussian density. Students sometimes use only the exponential part. The normalizing constant matters — without it, your densities aren't probabilities and your responsibilities won't sum to 1 correctly.
Computing $\frac{( x - μ ) ^{2}}{2 σ ^{2}}$ instead of just the numerator. The factor of 2 in the denominator of the exponent is part of the Gaussian formula. Forgetting it doubles your computed distance.
Not squaring the difference. $(x - μ)^{2}$ , not $∣ x - μ ∣$ . In Example 2: $(130 - 170)^{2} = 1600$ , not 40.
Using the Z-table. The professor explicitly said: "We don't require any Z distribution table." GMM responsibility problems use raw Gaussian density evaluations, not cumulative probabilities. If you reach for the Z-table, you're solving the wrong problem.

Exam note. Expect a GMM numerical of this kind: given mixing coefficients, means, and variances for two components, plus a data point, compute responsibilities using the Bayes-like formula. No Z-table needed — just exponentials. Classify the point to the component with the higher responsibility. If you are comfortable with these two examples, you can solve any GMM responsibility problem on the exam.

Bridge. GMMs and EM give you a way to model data as a mixture of distributions and find soft cluster assignments. But what if you don't know the distribution's parameters at all? The next topic — Maximum Likelihood Estimation (section 16.5) — is the general method for estimating parameters from data, and it's the foundation that EM itself rests on.

Real-World & Domain Connection. The GMM responsibility calculation is the inference step — given a trained model, classify new points. In fraud detection, a transaction's features are plugged into a GMM of normal vs fraudulent behavior, and the responsibility tells you the fraud probability. In astronomy, stars are classified by their spectral features into stellar types using GMMs; the responsibility handles stars at the boundary between types.

16.5 Maximum Likelihood Estimation (MLE) — Foundations

Hook. You have a coin but don't know if it's fair. You flip it 10 times and get 7 heads. What's your best guess for the true heads probability? You could say 0.7 — that's the proportion you observed. But why is that the "best" guess? Maximum likelihood estimation gives you the mathematical answer — and the recipe works for any distribution, not just coins.

16.5.1 From Bayes to MLE

Intuition. Imagine you're a detective with two suspects and a piece of evidence. Bayes' theorem combines your prior suspicion (how likely each suspect is a priori) with the evidence (how likely the evidence would be if each suspect were guilty). But what if you have no prior suspicion — both suspects are equally likely? Then you just pick the suspect who makes the evidence most probable. That's MLE: ignore the priors, pick the parameter that makes the observed data most likely.

The analogy: a photographer adjusting focus. The lens position is the parameter. The blurriness of the image is the negative log-likelihood. You turn the focus ring (change the parameter) until the image is sharpest (likelihood is maximized). No priors — just data and parameter.

The professor traced the path from Bayes' theorem to MLE in three logical steps:

Step 1 — Bayes for classification. You have evidence $E$ , two classes $A$ and $B$ :

P (A ∣ E) = \frac{P ( A ) \cdot P ( E ∣ A )}{P ( E )}, P (B ∣ E) = \frac{P ( B ) \cdot P ( E ∣ B )}{P ( E )}

You pick the class with the higher posterior probability.

Step 2 — Drop the denominator (MAP). Both expressions share the same denominator $P (E)$ . For comparing which is larger, it cancels out. As the professor said: "If I remove the denominator, the maximum remains the maximum." Now compare $P (A) \cdot P (E ∣ A)$ vs $P (B) \cdot P (E ∣ B)$ . This is the MAP (Maximum A Posteriori) decision rule — it still uses priors $P (A)$ and $P (B)$ .

Step 3 — Drop the priors (MLE). If $P (A) = P (B)$ — equal priors, or priors are unknown — you can drop them too. What remains:

Choose the class that maximizes P (E ∣ A) vs P (E ∣ B)

But $P (E ∣ A)$ is the likelihood — the probability of observing the data $E$ given the parameter (class) $A$ . Maximizing it directly is maximum likelihood estimation.

Formalize. MLE answers: given a dataset ${x_{1}, x_{2}, \dots, x_{n}}$ assumed to come from a distribution with unknown parameter $θ$ , what value of $θ$ makes the observed data most probable?

The likelihood function $L (θ)$ is the joint probability of all observations, treated as a function of the parameter:

L (θ) = P (x_{1}, x_{2}, \dots, x_{n} ∣ θ) = i = 1 \prod n P (x_{i} ∣ θ)

The product form assumes observations are independent — a standard assumption in MLE. The MLE $θ^_{MLE}$ is the value that maximizes $L (θ)$ :

θ^_{MLE} = ar g θ max L (θ)

In practice, we maximize the log-likelihood $ℓ (θ) = lo g L (θ)$ because the log turns the product into a sum:

ℓ (θ) = i = 1 \sum n lo g P (x_{i} ∣ θ)

The professor summarized: "That results in an expression which is maximization of probability distribution. That is where maximum likelihood is."

16.5.2 The MLE Procedure — General Recipe

The six-step MLE recipe. The professor gave this step-by-step procedure:

Identify the probability distribution that models your data. Bernoulli for binary outcomes, Binomial for counts of successes, Poisson for count data, Normal for continuous measurements.
Write the likelihood function $L (θ)$ — the joint probability of all observations as a function of the parameter(s). This is the product of individual probabilities: $L (θ) = \prod_{i} P (x_{i} ∣ θ)$ .
Take the natural log to get $ℓ (θ) = lo g L (θ)$ . Products become sums: $ℓ (θ) = \sum_{i} lo g P (x_{i} ∣ θ)$ . Exponents drop down: $lo g (p^{k}) = k lo g p$ .
Differentiate $ℓ (θ)$ with respect to the parameter $θ$ to get $d ℓ / d θ$ .
Set the derivative to zero and solve for $θ$ : $d ℓ / d θ = 0$ .
(Optional) Check the second derivative. If $d^{2} ℓ / d θ^{2} < 0$ at your solution, you have a maximum. "If you want, you can go further with second derivative and validate: is it the maximum or minimum? Second derivative less than 0 means maximum."

Assumptions & Scope. MLE assumes:

Correct model specification: the data actually comes from the distribution you chose. MLE under a wrong distribution gives wrong estimates, and you won't know it from the math alone.
Independence: observations are independent of each other. If data points are correlated (time series, clustered data), the product form $L (θ) = \prod P (x_{i})$ is wrong and the MLE can be biased.
Large samples: MLE estimates are consistent (converge to the true value as $n \to \infty$ ) and asymptotically normal, but in small samples they can be biased. The Normal variance MLE uses $n$ in the denominator, but the unbiased version uses $n - 1$ .
Regularity conditions: the log-likelihood must be differentiable and the parameter space must not depend on the data. These almost always hold for the distributions in this course.

Visual Intuition. Picture a graph with the parameter $θ$ on the x-axis and the likelihood $L (θ)$ on the y-axis. For a Bernoulli with 3 successes in 10 trials, the curve looks like a hill: it starts at 0 when $p = 0$ (impossible to get successes if true probability is 0), rises to a peak at $p = 0.30$ , then falls back to 0 at $p = 1$ . The MLE is the x-coordinate of the peak. The log-likelihood $ℓ (θ)$ has the same peak location but is easier to work with — it's a smooth concave curve that you can differentiate.

Symbol	Meaning	Domain
$θ$	Generic parameter to be estimated	Depends on distribution
$L (θ)$	Likelihood function: $P (data ∣ θ)$	$L (θ) \geq 0$
$ℓ (θ)$	Log-likelihood: $lo g L (θ)$	$ℓ (θ) \leq 0$ (discrete)
$θ^_{MLE}$	Maximum likelihood estimate	Same domain as $θ$

16.5.3 Common Distributions and Their MLE Targets

Distribution	Parameter(s) to estimate	MLE formula
Bernoulli	$p$	$p^= \frac{\sum x _{i}}{n}$ (sample proportion)
Binomial	$p$	$p^= \frac{k}{n}$ (sample proportion)
Poisson	$λ$	$λ^= X ˉ$ (sample mean)
Normal	$μ$ and $σ^{2}$	$μ^= X ˉ$ , $σ^^{2} = \frac{1}{n} \sum (x_{i} - X ˉ)^{2}$

Pitfalls.

Maximizing $L (θ)$ directly instead of $ℓ (θ)$ . Products of probabilities become astronomically small for even moderate $n$ . The log-likelihood prevents numerical underflow and makes differentiation tractable. Always take the log.
Forgetting to set the derivative to zero. The derivative gives the critical point. You still need to solve the resulting equation — don't stop at writing the derivative.
Confusing the likelihood with the probability. $L (θ) = P (data ∣ θ)$ is a function of $θ$ with the data fixed, not a function of the data. It is not a probability distribution over $θ$ — integrating $L (θ)$ over $θ$ does not give 1.
Using MLE when you have strong prior information. MLE ignores priors. If you know, from physics or domain knowledge, that a parameter must be in a certain range, use MAP or Bayesian methods instead.

Recap. MLE finds the parameter that makes the observed data most probable. Start with Bayes, drop the denominator (constant), drop the priors (assume equal), and maximize the likelihood. The recipe: identify distribution → write likelihood → take log → differentiate → set to zero → solve.

Bridge. The MLE recipe is abstract until you see it applied to concrete data. The next section (16.6) works through Bernoulli, Binomial, and Poisson MLE problems with real numbers — the exact types of problems you'll face on the exam.

Real-World & Domain Connection. MLE is the workhorse of statistical estimation. Pharmaceutical companies use MLE to estimate drug efficacy rates from clinical trial data (Bernoulli/binomial). Call centers use Poisson MLE to estimate arrival rates and staff accordingly. Insurance companies use Normal MLE to model claim amounts. The central limit theorem guarantees that MLE estimates are approximately Normal for large samples, which is why confidence intervals and hypothesis tests — the topics of sections 16.7-16.8 — are built on MLE estimates.

16.6 MLE Worked Examples

Hook. The MLE recipe in section 16.5 is abstract — "write the likelihood, take log, differentiate, solve." Now you'll see it applied to real data across three distributions. Each example follows the same six steps, but the details change with the distribution. Master these patterns and you can handle any MLE problem on the exam.

16.6.1 Bernoulli — Bike Helmets (Flaw Detection)

A sample of 10 new bike helmets is tested. The 1st, 3rd, and 10th helmets are flawed. The rest are not. Assume the flaw status follows a Bernoulli distribution. Find the MLE for $p$ , the probability that a helmet is flawed.

Step 1 — Define the random variables. Let $X_{i} = 1$ if helmet $i$ is flawed, $X_{i} = 0$ otherwise.

X_{1} = 1, X_{2} = 0, X_{3} = 1, X_{4} = 0, X_{5} = 0, X_{6} = 0, X_{7} = 0, X_{8} = 0, X_{9} = 0, X_{10} = 1

Three flawed helmets (1s), seven non-flawed helmets (0s). So $\sum x_{i} = 3$ , $n = 10$ .

Step 2 — Write the Bernoulli likelihood. A single Bernoulli observation has probability:

P (X_{i} = x_{i} ∣ p) = p^{x_{i}} (1 - p)^{1 - x_{i}}

For 10 independent observations, the joint likelihood is the product:

L (p) = i = 1 \prod 10 P (X_{i} = x_{i} ∣ p) = i = 1 \prod 10 p^{x_{i}} (1 - p)^{1 - x_{i}}

Since each $x_{i}$ is either 0 or 1, this product collapses:

L (p) = p^{\sum x_{i}} \cdot (1 - p)^{n - \sum x_{i}} = p^{3} (1 - p)^{7}

The professor: "Now we need to find what is the maximum value — optimize the maximum likelihood."

Step 3 — Take the log. Products become sums:

ℓ (p) = lo g L (p) = lo g (p^{3} (1 - p)^{7}) = 3 lo g p + 7 lo g (1 - p)

Step 4 — Differentiate.

\frac{d ℓ}{d p} = \frac{d}{d p} [3 lo g p + 7 lo g (1 - p)] = \frac{3}{p} - \frac{7}{1 - p}

Step 5 — Set to zero and solve.

\frac{3}{p} - \frac{7}{1 - p} = 0 ⟹ \frac{3}{p} = \frac{7}{1 - p}

3 (1 - p) = 7 p ⟹ 3 - 3 p = 7 p ⟹ 3 = 10 p ⟹ p^= 0.30

Step 6 — (Optional) Second derivative check.

\frac{d ^{2} ℓ}{d p ^{2}} = - \frac{3}{p ^{2}} - \frac{7}{( 1 - p ) ^{2}}

At $p = 0.30$ : both terms are negative, so $d^{2} ℓ / d p^{2} < 0$ — confirmed maximum.

Answer: The MLE for the flaw probability is $p^= 0.30$ .

Sense-check. 3 out of 10 helmets were flawed. The MLE is the sample proportion $3/10 = 0.30$ , exactly what common sense would suggest. For Bernoulli/Binomial, MLE always equals the sample proportion.

16.6.2 Binomial MLE — Derivation

Derivation. For a binomial distribution with $n$ trials and $k$ successes, the probability mass function is:

P (X = k ∣ p) = (k n) p^{k} (1 - p)^{n - k}

The binomial coefficient $(k n)$ does not depend on $p$ , so it drops out when we differentiate the log-likelihood. Using $θ$ for the parameter:

ℓ (θ) = lo g (k n) + k lo g θ + (n - k) lo g (1 - θ)

\frac{d ℓ}{d θ} = \frac{k}{θ} - \frac{n - k}{1 - θ} = 0

\frac{k}{θ} = \frac{n - k}{1 - θ} ⟹ k (1 - θ) = (n - k) θ ⟹ k - k θ = n θ - k θ ⟹ k = n θ

θ^= \frac{k}{n}

The MLE for the binomial parameter $p$ is the sample proportion: $p^= k / n$ .

16.6.3 Poisson MLE — Derivation

Derivation. For $n$ independent Poisson observations $x_{1}, x_{2}, \dots, x_{n}$ :

L (λ) = i = 1 \prod n \frac{e ^{- λ} λ ^{x_{i}}}{x _{i} !} = e^{- nλ} \cdot \frac{λ ^{\sum x_{i}}}{\prod x _{i} !}

ℓ (λ) = - nλ + (\sum x_{i}) lo g λ - \sum lo g (x_{i}!)

\frac{d ℓ}{d λ} = - n + \frac{\sum x _{i}}{λ} = 0

λ^= \frac{\sum x _{i}}{n} = X ˉ

The MLE for the Poisson rate $λ$ is the sample mean.

16.6.4 Poisson Worked Example — Defective Hard Drives

The number of defective hard drives produced daily follows a Poisson distribution. Counts from 10 days: 7, 3, 1, 2, 3, 4, 2, 1, 2, 1. The sum is $7 + 3 + 1 + 2 + 3 + 4 + 2 + 1 + 2 + 1 = 26$ .

Step 1 — Compute the MLE. $λ^= X ˉ = 26/10 = 2.6$ . As the professor said: "The maximum likelihood estimate for lambda is nothing but X bar."

Step 2 — Use the estimate to compute probabilities. Find $P (X = 0 or X = 1)$ , the probability of 0 or 1 defects on a given day:

P (X = 0) = \frac{e ^{- 2.6} \cdot ( 2.6 ) ^{0}}{0 !} = e^{- 2.6}

P (X = 1) = \frac{e ^{- 2.6} \cdot ( 2.6 ) ^{1}}{1 !} = 2.6 \cdot e^{- 2.6}

P (X = 0 or X = 1) = e^{- 2.6} + 2.6 \cdot e^{- 2.6} = e^{- 2.6} (1 + 2.6) = 3.6 \cdot e^{- 2.6}

Using $e^{- 2.6} \approx 0.0743$ :

P (0 or 1 defects) \approx 3.6 \times 0.0743 = 0.267

Sense-check. The average is 2.6 defects per day. Getting 0 or 1 defects is below average — about a 27% chance, which is plausible.

16.6.5 Binomial 0/1 Sample — MLE for $p$

Consider six independent observations, each either 0 or 1: $0, 1, 0, 0, 1, 0$ . Here $P (X = 1) = p$ and $P (X = 0) = 1 - p$ . Find the MLE for $p$ .

Step 1 — Count. Two ones, four zeros. So $k = 2$ , $n = 6$ .

Step 2 — Write the likelihood. Each observation contributes $(1 - p)$ if 0 and $p$ if 1:

L (p) = (1 - p) \cdot p \cdot (1 - p) \cdot (1 - p) \cdot p \cdot (1 - p) = p^{2} (1 - p)^{4}

Step 3 — Take log.

ℓ (p) = 2 lo g p + 4 lo g (1 - p)

Step 4 — Differentiate and set to zero.

\frac{d ℓ}{d p} = \frac{2}{p} - \frac{4}{1 - p} = 0

\frac{2}{p} = \frac{4}{1 - p} ⟹ 2 (1 - p) = 4 p ⟹ 2 - 2 p = 4 p ⟹ 2 = 6 p ⟹ p^= \frac{1}{3} \approx 0.333

Sense-check. 2 successes out of 6 trials gives $2/6 = 1/3$ . The MLE is the sample proportion, as expected.

Exam warning from the professor. If a solved example similar to this appears in the exam but with different numbers, do not copy blindly. "You will get zero marks if you submit a solution that does not match the actual data in your exam question." Always re-derive with the numbers in your specific question. The method is the same — the numbers change.

16.6.6 Student Questions and Answers

Q: For the Poisson example where the counts are 7, 3, 1, 2, ... — is the sum 26 and divided by 10 gives 2.6?

A: Yes. The sample mean $X ˉ = 2.6$ is the MLE for $λ$ . Then you use $λ = 2.6$ in the Poisson formula to compute $P (X = 0)$ and $P (X = 1)$ , and add them.

Assumptions & Scope. All MLE derivations above assume independent observations. For the Poisson, the data must be counts (non-negative integers) — MLE for $λ$ doesn't make sense for negative or continuous data. For Bernoulli/Binomial, observations must be binary. The MLE for the Normal distribution ( $μ^= X ˉ$ , $σ^^{2} = \frac{1}{n} \sum (x_{i} - X ˉ)^{2}$ ) follows the same log-differentiate-solve pattern but requires two partial derivatives.

Visual Intuition for MLE. For the Bernoulli bike-helmet example, imagine plotting $L (p) = p^{3} (1 - p)^{7}$ against $p$ from 0 to 1. The curve starts at 0 (at $p = 0$ ), rises to a peak at $p = 0.30$ , then falls back to 0 (at $p = 1$ ). The peak is sharp but not symmetric — the curve is taller on the left side (fewer successes) than the right. The log-likelihood $ℓ (p) = 3 lo g p + 7 lo g (1 - p)$ has the same peak location but is more symmetric and easier to differentiate.

Pitfalls.

Forgetting the log step. Directly differentiating $p^{3} (1 - p)^{7}$ with the product rule is messy and error-prone. Always take the log first.
Dropping the binomial coefficient too early. In the binomial MLE, $(k n)$ is a constant with respect to $p$ , so it vanishes in the derivative. But you must recognize that it's there in the likelihood function — it just doesn't affect the maximization.
Treating $n$ as unknown in binomial MLE. In this course, $n$ is known (the number of trials). Only $p$ is estimated. MLE for unknown $n$ is a much harder discrete optimization problem.
Copying solutions without checking numbers. The professor explicitly warned against this — your exam data will differ. Re-derive every time.

Recap. MLE for Bernoulli/Binomial gives $p^= k / n$ (sample proportion). MLE for Poisson gives $λ^= X ˉ$ (sample mean). The recipe is always: write the product likelihood, take log, differentiate, set to zero, solve. The pattern is the same across distributions.

Exam note. You can expect an MLE numerical: a small dataset plus a distribution (Bernoulli/Binomial/Poisson). Write the likelihood from the data, take log, differentiate, solve for the parameter. The Bernoulli bike-helmets problem and the binomial 0/1 sample are the representative types.

Bridge. MLE gives you point estimates — single best values for parameters. But estimates come with uncertainty. The next topic — hypothesis testing (section 16.7) — uses those estimates to answer the question: "Is this parameter value consistent with the data, or should we reject it?"

Real-World & Domain Connection. MLE is implemented in every statistical software package. When you run glm() in R or statsmodels in Python, the default fitting method is MLE. Manufacturing quality control uses Bernoulli MLE to estimate defect rates from inspection samples (the bike-helmet problem is a miniature version of real factory quality assurance). Call centers use Poisson MLE to forecast call volumes and schedule staff. In clinical trials, binomial MLE estimates drug response rates, directly feeding into the hypothesis tests that determine whether a new treatment is approved.

16.7 Hypothesis Testing — Model Identification and Decision Framework Review

Hook. You're handed an exam problem: "A manufacturer claims their brake pads last more than 920 hours. A sample of 44 pads is tested at 5% significance..." What test do you use? Z or t? One-tailed or two-tailed? Means or proportions? If you freeze here, the rest of the problem — no matter how well you compute — is wrong. The professor's main message: identify the model first, then compute. The formulas are available; knowing which one to pick is the skill.

16.7.1 The Core Skill — Identifying Which Model Applies

Intuition. Think of hypothesis testing like a toolbox. You have a Z-test wrench, a t-test screwdriver, a proportion test pliers. The problem describes a job — "tighten this bolt." Your first task isn't tightening; it's picking the right tool. The professor's four questions are your tool-identification checklist.

The professor emphasized: "You should not waste your time in referring which model. You should not get confused. Otherwise all the formulas are available. Tables are available."

Every hypothesis testing problem reduces to four binary decisions:

Means or proportions? If the data is averages/measurements (hours, weight, price) → means. If it's percentages/counts out of a total (defect rate, survival rate) → proportions.
One population or two? One sample → one-population test. Comparing two groups (brand A vs brand B, before vs after) → two-population test.
Large sample or small? For means: $n \geq 30$ → large (Z). For two means: $n_{1} + n_{2} - 2 \geq 30$ → large (Z). Otherwise → small (t).
One-tailed or two-tailed? Directional claim (greater than, less than, superior) → one-tailed. No-direction claim (different, equal, change) → two-tailed.

Once you answer these four, the formula and critical value follow mechanically.

16.7.2 The Decision Tree for Choosing a Test

The complete decision framework.

For means:

Scenario	Sample Size Criterion	Test	Test Statistic	Critical Value
One mean	$n \geq 30$	Z-test	$Z = \frac{X ˉ - μ _{0}}{σ / n}$	Z-table
One mean	$n < 30$	t-test	$t = \frac{X ˉ - μ _{0}}{s / n}$	t-table, df $= n - 1$
Two means	$n_{1} + n_{2} - 2 \geq 30$	Z-test	$Z = \frac{( X ˉ _{1} - X ˉ _{2} ) - ( μ _{1} - μ _{2} ) _{0}}{\frac{σ _{1}^{2}}{n _{1}} + \frac{σ _{2}^{2}}{n _{2}}}$	Z-table
Two means	$n_{1} + n_{2} - 2 < 30$	t-test	Pooled or unpooled t	t-table, df $= n_{1} + n_{2} - 2$

For proportions (always large sample, always Z):

Scenario	Test Statistic
One proportion	$Z = \frac{p ^ - P _{0}}{P _{0} Q _{0} / n}$
Two proportions	$Z = \frac{( p ^ _{1} - p ^ _{2} ) - ( P _{1} - P _{2} ) _{0}}{p ^ q ^ ( \frac{1}{n _{1}} + \frac{1}{n _{2}} )}$

Paired t-test: For before-and-after data — same individuals measured twice. Examples: drug efficacy (before vs after treatment), training program effectiveness (pre-test vs post-test). The test works on the differences $d_{i} = X_{after, i} - X_{before, i}$ , treating them as a single sample and testing $H_{0} : μ_{d} = 0$ .

16.7.3 Critical Rule — Two-Means Sample Size

The rule students miss most. "We should not see the sample sizes separately." For two means, compute:

n_{1} + n_{2} - 2

If this $\geq 30$ → Z-test. If $< 30$ → t-test.

$n_{1} = 20$ , $n_{2} = 20$ : each is below 30 individually, but $20 + 20 - 2 = 38 > 30$ → Z-test.
$n_{1} = 10$ , $n_{2} = 10$ : $10 + 10 - 2 = 18 < 30$ → t-test, df = 18.

The $- 2$ comes from losing one degree of freedom per sample for estimating each mean. This rule catches many students off guard.

16.7.4 Degrees of Freedom for t-Tests

Test Type	Degrees of Freedom
One sample	df $= n - 1$
Two samples	df $= n_{1} + n_{2} - 2$

To use the t-table: find the row matching your df, then the column matching your $α$ (and tail type). If $n = 20$ (one sample), df = 19 — look up row 19.

16.7.5 Z Critical Values Quick Reference

Memorize or keep a reference card. These six values cover most exam problems:

$α$	One-Tailed $Z_{α}$	Two-Tailed $Z_{α /2}$
0.01 (1%)	$\pm 2.33$	$\pm 2.58$
0.05 (5%)	$\pm 1.645$	$\pm 1.96$
0.10 (10%)	$\pm 1.28$	$\pm 1.645$

For one-tailed: the sign matches the direction. Right-tailed → positive critical value. Left-tailed → negative. For two-tailed: both $\pm$ values are rejection regions.

"If anything other than these values is needed, then you have to reference the full Z-table and figure it out."

16.7.6 Hypothesis Formulation Rules

The iron rule: $H_{0}$ must always contain equality. "H naught always comes with equality symbol."

Claim Wording	$H_{0}$	$H_{1}$	Tail Type
"The mean exceeds 920"	$μ = 920$	$μ > 920$	Right-tailed
"The mean is less than 750"	$μ = 750$	$μ < 750$	Left-tailed
"The average delivery time is 3 days"	$μ = 3$	$μ \neq = 3$	Two-tailed
"There is a significant difference between brands"	$μ_{1} - μ_{2} = 0$	$μ_{1} - μ_{2} \neq = 0$	Two-tailed
"Product A is superior to product B"	$μ_{A} - μ_{B} = 0$	$μ_{A} - μ_{B} > 0$	Right-tailed
"Product A lasts 34 hours more than B"	$μ_{A} - μ_{B} = 34$	$μ_{A} - μ_{B} > 34$	Right-tailed

The claim always goes to $H_{1}$ unless it already contains pure equality (=), in which case it goes to $H_{0}$ and $H_{1}$ gets $\neq =$ .

16.7.7 "Do Not Reject" vs "Accept"

Statisticians prefer "do not reject $H_{0}$ " over "accept $H_{0}$ ." The logic: failing to reject is not the same as proving it true — you just don't have enough evidence against it. For this course's exams, "accept $H_{0}$ " or "reject $H_{0}$ " is fine. Structure your conclusion: (1) state the decision about $H_{0}$ (accept/reject), then (2) state what this means for the original claim.

"Reject H naught means accept H1. The conclusion is first: accept H naught or reject H naught."

16.7.8 Hypothesis Testing — Step-by-Step Procedure

The complete hypothesis test workflow:

Identify the model. Answer the four questions: means/proportions? one/two populations? large/small sample? one-tail/two-tail?
State hypotheses. $H_{0}$ (with equality) and $H_{1}$ .
Determine the critical value. From the Z-table or t-table at the given $α$ .
Compute the test statistic. Plug sample data into the formula.
Compare and decide. If test statistic falls in the rejection region → reject $H_{0}$ . Otherwise → do not reject $H_{0}$ .
State the conclusion. About $H_{0}$ first, then about the original claim.

Assumptions & Scope.

Z-tests assume known population standard deviation $σ$ . In practice, $σ$ is rarely known — but exam problems typically provide it for Z-test scenarios. If only the sample standard deviation $s$ is given and $n < 30$ , use the t-test.
Central Limit Theorem (CLT): Z-tests rely on the CLT — the sampling distribution of $X ˉ$ is approximately Normal when $n \geq 30$ , regardless of the population shape. Below 30, you need the population to be approximately Normal to use t.
Independence: All tests assume observations are independent. Paired t-tests are the exception — they handle dependent (paired) data.
Proportions: Require large samples for the Normal approximation. Rule of thumb: $n P_{0} \geq 5$ and $n Q_{0} \geq 5$ . If these fail, use exact binomial tests instead.

Visual Intuition. Draw a Normal curve centered at the null value. Shade the rejection region(s) at the tail(s). The test statistic is a point on the x-axis. If it falls in the shaded zone, reject $H_{0}$ . For a right-tailed test at $α = 0.05$ , shade the rightmost 5% of the curve (starting at $Z = 1.645$ ). For a two-tailed test at $α = 0.05$ , shade the leftmost 2.5% and rightmost 2.5% (starting at $Z = - 1.96$ and $Z = 1.96$ ). The critical value is the boundary of the shaded zone.

Pitfalls.

Checking each sample separately for the two-means rule. Don't. Always compute $n_{1} + n_{2} - 2$ . Two samples of 20 each → Z, not t.
Putting the inequality in $H_{0}$ . $H_{0}$ always has equality (=). If the claim is "greater than," that goes to $H_{1}$ . If you reverse this, your entire test is backwards.
Using the wrong tail. "Is different from" → two-tailed. "Is greater than" → right-tailed. "Is less than" → left-tailed. A common mistake: using a two-tailed test when the claim is directional, which doubles your critical value and makes it harder to reject.
Forgetting that proportion tests always use Z. There is no "t-test for proportions" in this course. Proportions have their own Z formula with $P Q / n$ .
Stopping after computing the test statistic. You must compare it to the critical value and state a conclusion about $H_{0}$ . A test statistic without a decision is incomplete.

16.7.9 Student Questions and Answers

Several students asked about the two-means hypothesis formulation rule.

Q: In some questions about the test for equality of two means, the claim says "less than or equal to" — but we still set $H_{0}$ as equality and test difference as zero. Is that the rule?

A: Yes. $H_{0}$ always contains equality. If the claim is "the average life of product A is superior to product B," that translates to $μ_{A} - μ_{B} > 0$ . But $H_{0}$ becomes $μ_{A} - μ_{B} = 0$ (equality). $H_{1}$ is $μ_{A} - μ_{B} > 0$ . If the claim says "product A lasts 34 hours more than product B," then $H_{0} : μ_{A} - μ_{B} = 34$ and $H_{1} : μ_{A} - μ_{B} > 34$ . The actual claimed difference replaces 0, but $H_{0}$ still has equality.

Q: How do I find the degrees of freedom when the sample size is less than 30 for a single sample? What do I look up in the t-table?

A: For a single sample, df = $n - 1$ . If $n = 20$ , then df = 19. In the t-table, find the row for 19, then the column for your alpha value. That gives the critical t-value. For two samples, df = $n_{1} + n_{2} - 2$ .

Q: In the two-means Z formula, $μ_{1} - μ_{2}$ appears. What if $μ_{1}$ and $μ_{2}$ are not individually given?

A: You do not need individual $μ_{1}$ and $μ_{2}$ values. Under $H_{0} : μ_{1} - μ_{2} = 0$ , the difference is just 0. If $H_{0}$ says the difference is some specific value (like 30), then $μ_{1} - μ_{2} = 30$ . The formula uses the difference, not the individual values.

Symbol	Meaning	Domain
$X ˉ$	Sample mean	Any real number
$μ$	Population mean	Any real number
$σ$	Population standard deviation	$σ > 0$
$s$	Sample standard deviation	$s > 0$
$n$	Sample size	$n \geq 2$
$p^$	Sample proportion	$[0, 1]$
$P$	Population proportion (under $H_{0}$ )	$[0, 1]$
$Q$	$1 - P$	$[0, 1]$
$α$	Level of significance	Typically 0.01, 0.05, 0.10
df	Degrees of freedom	$\geq 1$

Recap. Hypothesis testing is a six-step workflow: identify the model (means/proportions, one/two populations, large/small, tail direction) → state $H_{0}$ (with equality) and $H_{1}$ → find critical value → compute test statistic → compare → conclude. The two-means sample size rule ( $n_{1} + n_{2} - 2$ ) and the Z critical values (1.645, 1.96, 2.58) are the highest-frequency traps.

Exam note. This is the most important exam skill — not computing, but identifying. "Just have a glance — which model? How to work?" Know the Z critical values for 1%, 5%, 10%. $H_{0}$ always has equality. The two-means rule: $n_{1} + n_{2} - 2$ , not individual samples.

Bridge. The framework is the map. The next section (16.8) walks the territory — eight fully worked hypothesis testing examples that apply every cell of the decision tree.

Real-World & Domain Connection. Hypothesis testing is the engine of evidence-based decision making. Pharmaceutical companies use two-sample t-tests to compare new drugs against placebos. Manufacturing plants use one-sample Z-tests to verify that production lines meet specifications. Marketing teams use proportion tests to measure campaign effectiveness. Election pollsters use two-proportion Z-tests to compare candidate support across demographics. Every time you read "statistically significant" in a study, a hypothesis test — built on this exact framework — is behind that claim.

16.8 Worked Hypothesis Testing Examples — Means and Proportions

Hook. Section 16.7 gave you the decision framework — the map. Now you'll walk the territory. Seven worked examples spanning one-mean, two-mean, sampling distribution, and proportion problems. Each one applies the same six-step workflow: identify → state hypotheses → find critical value → compute test statistic → compare → conclude.

16.8.1 Example A — Delivery Time (One Mean, Two-Tailed, Large Sample)

A company claims the average delivery time is 3 days. A random sample of 50 deliveries is taken. Sample mean $X ˉ = 3.2$ days, population standard deviation $σ = 0.6$ days. Test the claim at 5% significance.

Step 1 — Model identification. One mean. $n = 50 > 30$ → large sample → Z-test. Claim: "is 3 days" — pure equality, no direction → two-tailed.

Step 2 — State hypotheses.

H_{0} : μ = 3, H_{1} : μ \neq = 3

Step 3 — Critical value. $α = 0.05$ , two-tailed → $Z_{crit} = \pm 1.96$ .

Step 4 — Test statistic.

Z = \frac{X ˉ - μ _{0}}{σ / n} = \frac{3.2 - 3.0}{0.6/ 50} = \frac{0.2}{0.6/7.071} = \frac{0.2}{0.0849} \approx 2.36

Step 5 — Compare. $2.36 > 1.96$ — falls in the rejection region.

Step 6 — Conclude. Reject $H_{0}$ . The claim that the average delivery time is exactly 3 days is not supported by the data. The sample suggests the true mean is higher.

Sense-check. The sample mean (3.2) is fairly far from the claimed 3.0 — about 2.36 standard errors away. With 50 observations, that's enough evidence to reject.

16.8.2 Example B — Brake Pads (One Mean, Right-Tailed, Large Sample)

A manufacturer claims the mean life of their brake pads exceeds 920 hours. A random sample of 44 pads is tested: $X ˉ = 935$ hours, $σ = 54$ hours. Test at 5%.

Step 1 — Model identification. One mean. $n = 44 > 30$ → Z-test. Claim: "exceeds 920" → right-tailed.

Step 2 — State hypotheses.

H_{0} : μ = 920, H_{1} : μ > 920

Step 3 — Critical value. $α = 0.05$ , right-tailed → $Z_{crit} = 1.645$ .

Step 4 — Test statistic.

Z = \frac{935 - 920}{54/ 44} = \frac{15}{54/6.633} = \frac{15}{8.141} \approx 1.842

Step 5 — Compare. $1.842 > 1.645$ → reject $H_{0}$ .

Step 6 — Conclude. Accept the claim. The data supports that the mean life exceeds 920 hours (at 5% significance).

Sense-check. The sample mean (935) is 15 hours above the claimed threshold, about 1.84 standard errors. That's enough at $α = 0.05$ one-tailed, but it would fail a two-tailed test at 1% (where $Z_{crit} = 2.58$ ).

16.8.3 Example C — Light Bulbs (One Mean, Left-Tailed, Large Sample)

The claim: the mean life of light bulbs is less than 750 hours. Sample: $n = 36$ , $X ˉ = 740$ hours, $σ = 30$ hours. Test at 1%.

Step 1 — Model identification. One mean. $n = 36 > 30$ → Z-test. Claim: "less than 750" → left-tailed.

The professor explained the hypothesis placement: "Less than 750 — we can't take it as null hypothesis because equality is not there. Push it as alternative hypothesis. Null hypothesis becomes equality."

Step 2 — State hypotheses.

H_{0} : μ = 750, H_{1} : μ < 750

Step 3 — Critical value. $α = 0.01$ , left-tailed → $Z_{crit} = - 2.33$ .

Step 4 — Test statistic.

Z = \frac{740 - 750}{30/ 36} = \frac{- 10}{30/6} = \frac{- 10}{5} = - 2.00

Step 5 — Compare. $- 2.00 > - 2.33$ — the test statistic is NOT more extreme than the critical value. Do not reject $H_{0}$ .

Step 6 — Conclude. At 1% significance, there is not enough evidence to support the claim that the mean life is less than 750 hours. The sample mean (740) is lower, but the evidence isn't strong enough at the strict 1% level.

Sense-check. At $α = 0.05$ (left-tailed, $Z_{crit} = - 1.645$ ), we would reject $H_{0}$ . The stricter alpha (1% vs 5%) changes the conclusion — a perfect illustration of how significance level drives the decision.

16.8.4 Example D — Two Brands (Two Means, Two-Tailed, Large Sample)

Two brands are compared. Sample data: Brand 1: $n_{1} = 40$ , $X ˉ_{1} = 85$ , $σ_{1} = 8$ . Brand 2: $n_{2} = 40$ , $X ˉ_{2} = 80$ , $σ_{2} = 7$ . Is there a significant difference at 5%? At 1%?

Step 1 — Model identification. Two means. $n_{1} + n_{2} - 2 = 40 + 40 - 2 = 78 \geq 30$ → Z-test. Claim: "significant difference" → two-tailed.

Step 2 — State hypotheses.

H_{0} : μ_{1} - μ_{2} = 0, H_{1} : μ_{1} - μ_{2} \neq = 0

Step 3 — Critical values.

At 5%: $Z_{crit} = \pm 1.96$
At 1%: $Z_{crit} = \pm 2.58$

Step 4 — Test statistic.

Z = \frac{( 85 - 80 ) - 0}{\frac{8 ^{2}}{40} + \frac{7 ^{2}}{40}} = \frac{5}{\frac{64}{40} + \frac{49}{40}} = \frac{5}{1.6 + 1.225} = \frac{5}{2.825} = \frac{5}{1.681} \approx 2.97

Step 5 — Compare.

At 5%: $2.97 > 1.96$ → reject $H_{0}$ .
At 1%: $2.97 > 2.58$ → reject $H_{0}$ .

Step 6 — Conclude. There is a significant difference between the two brands at both 5% and 1% significance levels. Brand 1's mean is significantly higher.

Sense-check. The difference is 5 units within a standard error of about 1.68 — that's nearly 3 standard errors. Strong evidence. At both common alpha levels, we reject.

16.8.5 Example E — Sampling Distribution of the Mean

Vehicles have a population mean age $μ = 96$ months and $σ = 16$ months. A random sample of $n = 36$ vehicles is taken. Find the probability that the sample mean age falls between 90 and 100 months.

Step 1 — Model identification. Sampling distribution of the mean. $n = 36 > 30$ → CLT applies → use Z.

Step 2 — Convert bounds to Z-scores.

Z = \frac{X ˉ - μ}{σ / n} = \frac{X ˉ - 96}{16/6} = \frac{X ˉ - 96}{2.667}

For $X ˉ = 90$ : $Z = \frac{90 - 96}{2.667} = \frac{- 6}{2.667} = - 2.25$

For $X ˉ = 100$ : $Z = \frac{100 - 96}{2.667} = \frac{4}{2.667} = 1.50$

Step 3 — Compute the probability.

P (90 \leq X ˉ \leq 100) = P (- 2.25 \leq Z \leq 1.50) = F (1.50) - F (- 2.25)

Using the symmetry rule $F (- Z) = 1 - F (Z)$ :

= F (1.50) - [1 - F (2.25)] = F (1.50) + F (2.25) - 1

From the Z-table: $F (1.50) = 0.9332$ , $F (2.25) = 0.9878$ .

= 0.9332 + 0.9878 - 1 = 0.9210

Sense-check. About 92% chance the sample mean falls in a roughly $\pm 1.5 σ$ band around the population mean — plausible for $n = 36$ .

16.8.6 Example F — Proportion: Cancer Drug Efficacy

60% of cancer patients are cured by a new drug ( $P = 0.60$ ). In a random sample of $n = 120$ patients, what is the probability that 50% or more will be cured?

Step 1 — Model identification. Single proportion. $n = 120$ is large. Sampling distribution of $p^$ .

Step 2 — Compute Z for $p^= 0.50$ .

Z = \frac{p ^ - P}{P Q / n} = \frac{0.50 - 0.60}{( 0.60 ) ( 0.40 ) /120}

Compute the denominator:

\frac{0.24}{120} = 0.002 = 0.04472

Z = \frac{- 0.10}{0.04472} \approx - 2.236

The professor rounded this to $- 2.2$ .

Step 3 — Find the probability.

P (p^\geq 0.50) = P (Z \geq - 2.236)

Since the Normal distribution is symmetric: $P (Z \geq - 2.236) = 1 - F (- 2.236) = F (2.236)$ .

From the Z-table: $F (2.24) \approx 0.9875$ . So:

P (p^\geq 0.50) \approx 0.9875

Sense-check. There's about a 98.75% chance that 50% or more of 120 patients will be cured, even though the true cure rate is only 60%. The large sample size makes it very likely that $p^$ stays close to 0.60.

16.8.7 Example G — Disease Survival (One Proportion, Right-Tailed)

The claim: 85% of people survive a certain disease ( $P = 0.85$ ). A sample of $n = 20$ people finds 18 survivors ( $p^= 18/20 = 0.90$ ). Does this support that the survival rate is more than 85%?

Step 1 — Model identification. Single proportion. The professor clarified: "85% is population proportion — that is the generalized one. 18 out of 20 is sample proportion." Claim: "more than 85%" → right-tailed.

Step 2 — State hypotheses.

H_{0} : P = 0.85, H_{1} : P > 0.85

Step 3 — Critical value. Assuming $α = 0.05$ : right-tailed $Z_{crit} = 1.645$ .

Step 4 — Test statistic.

Z = \frac{p ^ - P _{0}}{P _{0} Q _{0} / n} = \frac{0.90 - 0.85}{( 0.85 ) ( 0.15 ) /20} = \frac{0.05}{0.1275/20} = \frac{0.05}{0.006375} = \frac{0.05}{0.0798} \approx 0.627

Step 5 — Compare. $0.627 < 1.645$ → do not reject $H_{0}$ .

Step 6 — Conclude. The sample (18/20 = 90%) does not provide strong enough evidence that the true survival rate exceeds 85%. The difference could easily be due to random chance with only 20 patients.

Note on sample size: $n = 20$ is small. Check $n P_{0} = 20 \times 0.85 = 17 \geq 5$ and $n Q_{0} = 20 \times 0.15 = 3$ — the second value is below 5, so the Normal approximation is borderline. In practice, you'd use an exact binomial test. For exam purposes at this level, the Z-test approach is acceptable.

Sense-check. Going from 85% to 90% with only 20 patients is just 1 extra survivor above the expected 17. That's weak evidence — the non-rejection makes sense.

16.8.8 Comparison — Hypothesis Testing Patterns Summary

Example	Type	Tail	$n$	$α$	$Z_{crit}$	$Z_{calc}$	Decision
A: Delivery	One mean	Two	50	5%	$\pm 1.96$	2.36	Reject $H_{0}$
B: Brake pads	One mean	Right	44	5%	1.645	1.842	Reject $H_{0}$
C: Light bulbs	One mean	Left	36	1%	$- 2.33$	$- 2.00$	Do not reject
D: Two brands	Two means	Two	40+40	5%/1%	$\pm 1.96$ / $\pm 2.58$	2.97	Reject both
E: Vehicles	Sampling dist	Symmetric	36	—	—	—	$P \approx 0.921$
F: Cancer drug	Proportion	Symmetric	120	—	—	—	$P \approx 0.9875$
G: Disease	One proportion	Right	20	5%	1.645	0.627	Do not reject

16.8.9 Student Questions and Answers

Q: For the sampling distribution problems, the formula is the same whether $n = 36$ or $n = 49$ , right?

A: Yes. Both are greater than 30, so both use the Z distribution. The central limit theorem says: whenever the sample size reaches 30 or more, you can use the Z distribution. This holds regardless of the population distribution. That is the clue — central limit theorem.

Q: How do we tell whether the given proportion is the sample proportion $p^$ or population proportion $P$ ?

A: Look at the phrasing. The generalized statement is population — "85% of people survive" is $P$ . The specific data from the sample is $p^$ . 20 people were selected, 18 survived — so $p^= 18/20 = 0.90$ . If it is a broad claim about the whole population, it is $P$ . If it is from the data collected in the problem, it is $p^$ .

Q: What if the level of significance $α$ is not mentioned in the exam question?

A: Pick your own $α$ . Choose any standard value — 1%, 5%, or 10% — and state it. Answer accordingly. Do not ask the exam supervisor or proctor. You can choose any alpha value and write your own conclusion. There is no need to ask anyone.

16.8.10 Exam Strategy Tips

Identify the model first. "Just have a glance — which model? How to work?" Before computing, classify the problem.
All formulas are available. In an open-book exam, don't memorize — know which formula to use and when.
Keep the Z critical values table handy. 1%, 5%, 10% for both tails cover most problems.
For t-tests, use the full t-table. Row = df, column = $α$ . No shortcut.
Conclusion starts with $H_{0}$ . "Accept $H_{0}$ or reject $H_{0}$ ." Then translate to the original claim.
Tomorrow's session: more testing problems, ANOVA overview, time series review, correlation summary.

Pitfalls.

Using Z when you should use t. If $n < 30$ for one sample or $n_{1} + n_{2} - 2 < 30$ for two samples, and $σ$ is unknown, use t — not Z. The t-distribution has fatter tails and a different critical value.
Wrong tail for the claim. "More than" → right-tailed. "Less than" → left-tailed. "Different" → two-tailed. Getting this wrong flips your entire rejection region.
Forgetting to divide by $n$ in the denominator. The standard error is $σ / n$ , not just $σ$ . Without the $n$ , your test statistic is too small and you'll rarely reject.
Not stating a conclusion. Computing the test statistic is not enough. Compare to critical value, state "reject $H_{0}$ " or "do not reject $H_{0}$ ," then interpret.
Misidentifying $P$ vs $p^$ . $P$ is the population value (from the claim). $p^$ is computed from the sample data.

Recap. Seven worked examples span the full hypothesis testing landscape: one-mean Z (delivery, brake pads, light bulbs), two-mean Z (two brands), sampling distribution (vehicles), proportion Z (cancer drug), and one-proportion test (disease survival). The pattern is identical: identify → hypothesize → critical value → test statistic → compare → conclude.

Exam note. If $α$ is not given, pick your own (1%, 5%, or 10%) and state it. Know the Z critical values. Always compute $n_{1} + n_{2} - 2$ for two means. And never copy a solved example blindly — verify the numbers match your exam question.

Bridge. Tomorrow's session extends this framework to ANOVA (comparing more than two means), time series analysis, and correlation — all built on the same hypothesis testing logic.

Real-World & Domain Connection. These examples mirror real business decisions. The delivery time test (Example A) is what logistics companies run daily to monitor service level agreements. The brake pad test (Example B) is a miniature version of automotive safety compliance testing — regulators require statistical evidence that components meet specifications. The two-brands comparison (Example D) is marketing A/B testing. The cancer drug proportion (Example F) is a simplified clinical trial power calculation. Every example maps to a real decision someone gets paid to make.

Exam Guidance Summary

Exam note — GMM numericals. Expect a problem with mixing coefficients, means, and variances for two Gaussian components and a data point $x$ . Compute the responsibility (posterior probability) for each component using the Bayes-like formula from section 16.3. No Z-table needed — just exponentials. Classify the point to the component with higher responsibility. The examples in section 16.4 are the template.

Exam note — MLE numericals. A small dataset plus a distribution (Bernoulli, Binomial, or Poisson). Write the likelihood function from the data, take log, differentiate, set to zero, solve for the parameter. The Bernoulli bike-helmets problem ( $p^= 0.30$ ) and the binomial 0/1 sample problem ( $p^= 1/3$ ) from section 16.6 are the representative types. Do not copy solutions blindly — verify the numbers in your exam question match before using a solved example.

Exam note — Hypothesis testing. The most common question type. You must identify: (a) means or proportions, (b) one or two populations, (c) large or small sample, (d) one-tailed or two-tailed. Then write $H_{0}$ (always with equality) and $H_{1}$ correctly. Compute the test statistic, compare to the critical value, and state your conclusion about $H_{0}$ first. The worked examples in section 16.8 cover every pattern.

Exam note — Critical values and rules to memorize.

Z critical values: 1% one-tail $\pm 2.33$ , 1% two-tail $\pm 2.58$ ; 5% one-tail $\pm 1.645$ , 5% two-tail $\pm 1.96$ ; 10% one-tail $\pm 1.28$ , 10% two-tail $\pm 1.645$ .
Two-means sample size rule: compute $n_{1} + n_{2} - 2$ . If $\geq 30$ → Z; if $< 30$ → t. Never check each sample separately.
Degrees of freedom: one sample df = $n - 1$ ; two samples df = $n_{1} + n_{2} - 2$ .
If $α$ is not given, pick your own (1%, 5%, or 10%) and state it explicitly.

Tomorrow's topics: More hypothesis testing problems, ANOVA overview, time series review, correlation summary. Bring your doubts — the professor will spend time on Q&A.

Key Industry Applications

Gaussian Mixture Models power soft clustering, density estimation, and anomaly detection wherever clusters overlap. Customer segmentation that captures mixed buying behaviors — a shopper who buys both luxury and discount items isn't forced into one bucket. Financial return modeling from mixed market regimes (bull, bear, sideways). Background subtraction in video surveillance, where each pixel's color over time is a GMM and anomalies trigger motion alerts. GMMs are the standard when you need probabilistic rather than hard binary cluster assignments.
The EM algorithm is a general optimization technique for any problem with latent variables — not just GMMs. Hidden Markov models in speech recognition use EM to learn phone-to-acoustic mappings. Missing data imputation in clinical trials uses EM when patients drop out. Latent Dirichlet Allocation for topic modeling uses EM to discover themes in document collections. Wherever you have "I can see the data but not the hidden labels," EM is the standard solution.
Maximum Likelihood Estimation is the default parameter estimation method across all of statistics. Manufacturing quality control uses Bernoulli MLE to estimate defect rates from inspection samples. Call centers use Poisson MLE to forecast call volumes and schedule staff. Insurance actuarial models use Normal MLE for claim amounts. Reliability engineering uses MLE to estimate component lifetimes from failure data. Every glm() in R and statsmodels fit in Python uses MLE under the hood.
Hypothesis testing is the engine of evidence-based business decisions. Comparing two suppliers (two-means Z-test). Validating product performance claims (one-mean Z-test). Measuring marketing campaign conversion rates (proportion test). Evaluating training program effectiveness with before-and-after measurements (paired t-test). Every "statistically significant" result in a business report or scientific paper traces back to the hypothesis testing framework covered in this lecture.

ISM Lecture 16 notes · Gaussian Mixture Models, Maximum Likelihood Estimation, and Hypothesis Testing

Introduction to Statistical Methods· postgraduate· 2026-07-07

Sections Breakdown

1Gaussian Mixture Models — Recap and Mathematical Representation

Weighted sum of Gaussian distributions, hard vs soft clustering, mixing coefficients, covariance matrix

2Expectation Maximization (EM) Algorithm for GMM

E-step and M-step iteration, responsibility computation, parameter updates, local maxima

3GMM — Posterior Probability (Responsibility) Computation

Bayes' theorem mapping to GMM, K-component responsibility formula

4GMM Worked Numerical Examples

Two full examples computing responsibilities for given parameters and classifying data points

5Maximum Likelihood Estimation (MLE) — Foundations

From Bayes to MLE, six-step recipe, likelihood and log-likelihood functions

6MLE Worked Examples

Bernoulli bike helmets, Binomial derivation, Poisson defective hard drives, Binomial 0/1 sample

7Hypothesis Testing — Model Identification and Decision Framework

Means vs proportions, one vs two populations, Z vs t, one-tail vs two-tail, H0 equality rule

8Worked Hypothesis Testing Examples — Means and Proportions

Seven examples: delivery time, brake pads, light bulbs, two brands, sampling distribution, cancer drug, disease survival

9Exam Guidance Summary

Exam tips for GMM numericals, MLE numericals, hypothesis testing identification, critical values

10Key Industry Applications

Real-world uses of GMMs, EM, MLE, and hypothesis testing across industries

Postgraduate students in statistics and data science

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Gaussian Mixture Models

Must-know: A GMM is a weighted sum of K Gaussian components. Each component has a mixing coefficient pi_k, mean mu_k, and covariance Sigma_k. Soft clustering gives probabilities instead of hard assignments.

$p (x) = k = 1 \sum K π_{k} N (x ∣ μ_{k}, Σ_{k})$

⚠ Top pitfall: Forgetting that mixing coefficients must sum to 1 — if you initialize without normalizing, the GMM is not a valid probability distribution.

Self-check: What are the three parameter families per component in a GMM?

Connects to: EM Algorithm, Responsibility Computation, Soft Clustering

EM Algorithm for GMM

Must-know: EM alternates between E-step (compute responsibilities using current parameters) and M-step (update parameters using responsibilities). It finds a LOCAL maximum of the likelihood — run multiple times with different starts.

$γ (z_{ik}) = \frac{π _{k} N ( x _{i} ∣ μ _{k} , Σ _{k} )}{\sum _{j = 1}^{K} π _{j} N ( x _{i} ∣ μ _{j} , Σ _{j} )}$

⚠ Top pitfall: Thinking EM gives the global optimum — it doesn't. Always run multiple random initializations and pick the highest final likelihood.

Self-check: What's the difference between exam mode (parameters given, just compute responsibilities) and implementation mode (iterate E and M until convergence)?

Connects to: GMM, Responsibility, MLE

GMM Responsibility (Posterior Probability)

Must-know: Responsibility is the posterior probability that component k generated point x. It's Bayes' theorem: prior = pi_k (mixing coefficient), likelihood = N(x|mu_k, sigma_k^2), denominator = total GMM density.

$γ (z_{1}) = \frac{π _{1} \cdot N ( x ∣ μ _{1} , σ _{1}^{2} )}{π _{1} \cdot N ( x ∣ μ _{1} , σ _{1}^{2} ) + π _{2} \cdot N ( x ∣ μ _{2} , σ _{2}^{2} )}$

⚠ Top pitfall: Neglecting the mixing coefficient — responsibility is NOT just N1/(N1+N2). Multiply by priors pi_k. Also forgetting the 1/(sigma*sqrt(2*pi)) normalizing factor in the Gaussian density.

Self-check: Why must the responsibilities for a single data point sum to 1 across all components?

Connects to: Bayes' Theorem, EM E-step, GMM

Maximum Likelihood Estimation

Must-know: MLE finds the parameter that makes observed data most probable. Recipe: identify distribution → write likelihood (product) → take log (sum) → differentiate → set to zero → solve. Start from Bayes, drop denominator (MAP), drop priors (MLE).

$ℓ (θ) = i = 1 \sum n lo g P (x_{i} ∣ θ)$

⚠ Top pitfall: Maximizing L(theta) directly instead of log L(theta). Products become astronomically small; the log prevents numerical underflow and makes differentiation tractable.

Self-check: What are the three logical steps from Bayes' theorem to MLE?

Connects to: Bernoulli MLE, Binomial MLE, Poisson MLE

Bernoulli/Binomial MLE

Must-know: For Bernoulli/Binomial, MLE of p = k/n (sample proportion). Write likelihood as p^k * (1-p)^(n-k), take log, differentiate, solve. The binomial coefficient drops out during differentiation.

$p^= \frac{\sum x _{i}}{n} (Bernoulli) p^= \frac{k}{n} (Binomial)$

⚠ Top pitfall: Forgetting the log step — differentiating p^3(1-p)^7 with the product rule is messy. Always take log first to turn the product into a sum.

Self-check: In the Bernoulli bike-helmets problem (3 flaws out of 10), what is the likelihood function and why does the MLE equal 0.30?

Connects to: MLE, Poisson MLE, Hypothesis Testing

Poisson MLE

Must-know: For Poisson, MLE of lambda = X-bar (sample mean). The factorial terms x_i! drop out during differentiation since they don't depend on lambda.

$λ^= \frac{\sum x _{i}}{n} = X ˉ$

⚠ Top pitfall: Using Poisson MLE for non-count data — Poisson only applies to non-negative integers. MLE for lambda doesn't make sense for negative or continuous data.

Self-check: Given daily defect counts 7,3,1,2,3,4,2,1,2,1, what is the MLE for lambda and how do you compute P(0 or 1 defects)?

Connects to: MLE, Bernoulli MLE, Distribution Modeling

Hypothesis Testing Framework

Must-know: Identify the model FIRST: (a) means or proportions, (b) one or two populations, (c) large (n>=30 → Z) or small (n<30 → t), (d) one-tailed or two-tailed. H0 ALWAYS has equality. Two-means rule: compute n1+n2-2, NOT individual samples.

$Z = \frac{X ˉ - μ _{0}}{σ / n} (one mean, Z-test)$

⚠ Top pitfall: Checking each sample separately for two-means rule. Always compute n1+n2-2. Two samples of 20 each → 38 >= 30 → Z-test, NOT t-test.

Self-check: For a claim 'the mean exceeds 920', write H0 and H1 and state the tail type.

Connects to: Z-test, t-test, Proportion Test, MLE

Z-Test and Proportion Test Workflow

Must-know: Proportion tests ALWAYS use Z, never t. Memorize Z critical values: 5% one-tail ±1.645, 5% two-tail ±1.96; 1% one-tail ±2.33, 1% two-tail ±2.58. If alpha not given, pick your own (1%, 5%, or 10%) and state it.

$Z = \frac{p ^ - P _{0}}{P _{0} Q _{0} / n} (one proportion)$

⚠ Top pitfall: Stopping after computing the test statistic. You must compare it to the critical value and state a conclusion about H0 first, then interpret for the original claim.

Self-check: A sample of 20 patients shows 18 survive (p-hat=0.90) vs claimed P=0.85. With Z=0.627 at alpha=0.05, what's the decision and why?

Connects to: Hypothesis Testing, MLE, Sampling Distribution

Previous LectureLecture 15 · Time Series Analysis — Smoothing and Forecasting

Next LectureLecture 17 · Hypothesis Testing Review, ANOVA, Chi-Square, Time Series, and GMM

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Gaussian Mixture Models, Maximum Likelihood Estimation, and Hypothesis Testing Review

16.1 Gaussian Mixture Models — Recap and Mathematical Representation

16.1.1 What a Gaussian Mixture Model Is

16.1.2 Hard Clustering vs Soft Clustering

16.1.3 The Covariance Matrix — Why It Appears

16.1.4 Student Questions and Answers

16.2 Expectation Maximization (EM) Algorithm for GMM

16.2.1 Why We Need EM

16.2.2 The EM Algorithm — Inputs, Outputs, and Steps

16.2.3 GMM Challenges and Limitations

16.2.4 Student Questions and Answers

16.3 GMM — Posterior Probability (Responsibility) Computation

16.3.1 The Responsibility Formula — Connecting to Bayes' Theorem

16.3.2 Student Questions and Answers

16.4 GMM Worked Numerical Examples

16.4.1 Example 1 — Responsibility for x=95

16.4.2 Example 2 — Classifying x=130

16.5 Maximum Likelihood Estimation (MLE) — Foundations

16.5.1 From Bayes to MLE

16.5.2 The MLE Procedure — General Recipe

16.5.3 Common Distributions and Their MLE Targets

16.6 MLE Worked Examples

16.6.1 Bernoulli — Bike Helmets (Flaw Detection)

16.6.2 Binomial MLE — Derivation

16.6.3 Poisson MLE — Derivation

16.6.4 Poisson Worked Example — Defective Hard Drives

16.6.5 Binomial 0/1 Sample — MLE for p

16.6.6 Student Questions and Answers

16.7 Hypothesis Testing — Model Identification and Decision Framework Review

16.7.1 The Core Skill — Identifying Which Model Applies

16.7.2 The Decision Tree for Choosing a Test

16.7.3 Critical Rule — Two-Means Sample Size

16.7.4 Degrees of Freedom for t-Tests

16.7.5 Z Critical Values Quick Reference

16.7.6 Hypothesis Formulation Rules

16.7.7 "Do Not Reject" vs "Accept"

16.7.8 Hypothesis Testing — Step-by-Step Procedure

16.7.9 Student Questions and Answers

16.8 Worked Hypothesis Testing Examples — Means and Proportions

16.8.1 Example A — Delivery Time (One Mean, Two-Tailed, Large Sample)

16.8.2 Example B — Brake Pads (One Mean, Right-Tailed, Large Sample)

16.8.3 Example C — Light Bulbs (One Mean, Left-Tailed, Large Sample)

16.8.4 Example D — Two Brands (Two Means, Two-Tailed, Large Sample)

16.8.5 Example E — Sampling Distribution of the Mean

16.8.6 Example F — Proportion: Cancer Drug Efficacy

16.8.7 Example G — Disease Survival (One Proportion, Right-Tailed)

16.8.8 Comparison — Hypothesis Testing Patterns Summary

16.8.9 Student Questions and Answers

16.8.10 Exam Strategy Tips

Exam Guidance Summary

Key Industry Applications

ISM Lecture 16 notes · Gaussian Mixture Models, Maximum Likelihood Estimation, and Hypothesis Testing

Sections Breakdown

Exam Revision Notes

Gaussian Mixture Models

EM Algorithm for GMM

GMM Responsibility (Posterior Probability)

Maximum Likelihood Estimation

Bernoulli/Binomial MLE

Poisson MLE

Hypothesis Testing Framework

Z-Test and Proportion Test Workflow

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat

16.4.1 Example 1 — Responsibility for $x = 95$

16.4.2 Example 2 — Classifying $x = 130$

16.6.5 Binomial 0/1 Sample — MLE for $p$