Machine Learning

Logistic Regression — Complete Lecture Notes

📅 Published: 2026-06-28

🎓 Level: postgraduate

👥 Audience: Postgraduate students in Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Linear Regression Model and Cost Function — covered in Lecture 3 & 4
Gradient Descent Algorithm — covered in Lecture 4
Feature Engineering and Scaling — covered in Lecture 3 & 4
Classification vs Regression — covered in Lecture 1
Overfitting and Underfitting — covered in Lecture 4

Logistic Regression — Complete Enriched Lecture Notes

1. Logistic Regression — Core Concepts

Hook. You have a stack of loan applications. You want a yes/no answer: approve or deny. But what if the algorithm gave you a number — say 0.87 — instead of a hard yes? That number is the model's confidence: "I am 87% sure this applicant will repay." Logistic regression is the tool that produces that number. And here is the surprise: its training rule is mathematically identical to linear regression's. Same equation, completely different job.

Intuition. Think of a dimmer switch versus a light switch. A light switch is either ON or OFF — that is a hard classifier. A dimmer gives you any brightness between OFF and ON — that is what logistic regression does: it outputs a probability, a number between 0 and 1. You can always snap the dimmer to ON (probability > 0.5) or OFF (probability < 0.5) by picking a threshold, but the raw output keeps the nuance.

The engine under the hood is the sigmoid function, which looks like an S-curve. Whatever raw score you feed in — big positive, big negative, zero — the sigmoid politely squashes it into (0, 1). Like a bouncer who never lets anyone outside the club, no matter how extreme the input.

Analogy mapping: Raw score (logit) = how far you are from the decision boundary. Sigmoid = the bouncer who converts that distance into a "probability of entry." The analogy breaks at the extremes: a real bouncer might let in someone standing right at the rope (0.5 probability), but the sigmoid is perfectly symmetric and mathematical — it has no bad days.

Formalize — Building the Model Step by Step.

Step 1: The linear score. Just like linear regression, we form a weighted sum of the inputs:

$z = θ_{0} + θ_{1} x_{1} + θ_{2} x_{2} + \dots + θ_{n} x_{n} = θ^{⊤} x$

where $x_{0} = 1$ by convention (so $θ_{0}$ becomes the bias/intercept term). We call $z$ the logit or score. It can be any real number: $- \infty < z < \infty$ . This raw number is NOT a probability — it is unbounded.

Step 2: The sigmoid squashing function. To convert $z$ into a probability, we pass it through the logistic sigmoid:

$h_{θ} (x) = σ (z) = \frac{1}{1 + e ^{- z}}$

This function has three critical properties:

Range: $σ (z) \in (0, 1)$ for all real $z$ . It never hits exactly 0 or 1.
Center: $σ (0) = 0.5$ . At the decision boundary ( $z = 0$ ), the model is maximally uncertain.
Derivative: $\frac{d σ}{d z} = σ (z) (1 - σ (z))$ . This clean self-derivative is what makes the gradient descent math work out so neatly — the $σ (1 - σ)$ factor cancels out when we take the derivative of the log-loss. [Verified against Bishop §4.3.2, Eq. 4.88.]

Step 3: The cost function — why not use mean squared error? If we plugged $σ (θ^{⊤} x)$ into the MSE cost from linear regression, the sigmoid nonlinearity would make the cost function non-convex — full of local minima where gradient descent could get stuck. The fix is the log loss (also called cross-entropy):

$J (θ) = - \frac{1}{m} i = 1 \sum m [y^{(i)} ln h_{θ} (x^{(i)}) + (1 - y^{(i)}) ln (1 - h_{θ} (x^{(i)}))]$

Why this form? It comes from maximum likelihood. For each training example, the model's prediction $h_{θ} (x)$ is interpreted as $P (y = 1 ∣ x; θ)$ . The likelihood of seeing the entire dataset is:

$L (θ) = i = 1 \prod m h_{θ} (x^{(i)})^{y^{(i)}} \cdot (1 - h_{θ} (x^{(i)}))^{1 - y^{(i)}}$

Taking the negative log (which turns products into sums and flips maximization into minimization) gives exactly the cross-entropy above. This cost function is convex for logistic regression, so gradient descent is guaranteed to find the global minimum. [Verified against Bishop §4.3.2, Eq. 4.89–4.90.]

Step 4: The gradient and update rule. Using the sigmoid derivative $σ^{'} = σ (1 - σ)$ and the chain rule, the gradient of $J$ with respect to $θ_{j}$ simplifies dramatically:

$\frac{\partial J}{\partial θ _{j}} = \frac{1}{m} i = 1 \sum m (h_{θ} (x^{(i)}) - y^{(i)}) x_{j}^{(i)}$

The $σ (1 - σ)$ factor from the chain rule cancels with the structure of the log-loss. The result is stunningly simple: the gradient is the average of (prediction âˆ’ actual) Ã— input. This is exactly the same form as linear regression's gradient. [Verified against Bishop §4.3.2, Eq. 4.91.] The update rule follows:

$θ_{j} := θ_{j} - α \frac{1}{m} i = 1 \sum m (h_{θ} (x^{(i)}) - y^{(i)}) x_{j}^{(i)}$

The only difference from linear regression is inside $h_{θ}$ :

Linear regression: $h_{θ} (x) = θ^{⊤} x$ (raw linear output)
Logistic regression: $h_{θ} (x) = σ (θ^{⊤} x)$ (sigmoid of linear output)

Derivative walkthrough — the cancellation that makes everything work:

$\frac{\partial J}{\partial θ _{j}} where \frac{\partial h _{i}}{\partial θ _{j}} ∴ \frac{\partial J}{\partial θ _{j}} = - \frac{1}{m} i = 1 \sum m [y^{(i)} \frac{1}{h _{i}} \frac{\partial h _{i}}{\partial θ _{j}} + (1 - y^{(i)}) \frac{1}{1 - h _{i}} (- \frac{\partial h _{i}}{\partial θ _{j}})] = - \frac{1}{m} i = 1 \sum m [\frac{y ^{(i)}}{h _{i}} - \frac{1 - y ^{(i)}}{1 - h _{i}}] \frac{\partial h _{i}}{\partial θ _{j}} = σ (z_{i}) (1 - σ (z_{i})) \cdot x_{j}^{(i)} = h_{i} (1 - h_{i}) \cdot x_{j}^{(i)} = - \frac{1}{m} i = 1 \sum m [\frac{y ^{(i)} - h _{i}}{h _{i} ( 1 - h _{i} )}] \cdot h_{i} (1 - h_{i}) \cdot x_{j}^{(i)} = \frac{1}{m} i = 1 \sum m (h_{i} - y^{(i)}) x_{j}^{(i)}$

The $h_{i} (1 - h_{i})$ in the numerator and denominator cancel — the signature elegance of the sigmoid + cross-entropy pair. This is why we use this combination: the math simplifies beautifully.

Worked Example — A Tiny 2D Classification.

You have 3 training examples for predicting "Pass Exam" (1 = pass, 0 = fail) from hours studied ( $x_{1}$ ):

$i$	$x_{1}$ (hours)	$y$ (pass?)
1	1	0
2	3	0
3	5	1

Initial weights: $θ_{0} = 0$ , $θ_{1} = 0$ . Learning rate $α = 0.1$ .

Iteration 1:

For all records with $θ = [0, 0]$ : $z = 0 + 0 \cdot x_{1} = 0$ , so $h_{θ} = σ (0) = 0.5$ .
Errors: all three predictions are 0.5, actuals are [0, 0, 1].
Gradient for $θ_{0}$ : $\frac{1}{3} [(0.5 - 0) + (0.5 - 0) + (0.5 - 1)] = \frac{1}{3} [0.5 + 0.5 - 0.5] = \frac{0.5}{3} \approx 0.167$
Gradient for $θ_{1}$ : $\frac{1}{3} [(0.5 - 0) \cdot 1 + (0.5 - 0) \cdot 3 + (0.5 - 1) \cdot 5] = \frac{1}{3} [0.5 + 1.5 - 2.5] = \frac{- 0.5}{3} \approx - 0.167$
Update: $θ_{0} := 0 - 0.1 \cdot 0.167 = - 0.0167$ , $θ_{1} := 0 - 0.1 \cdot (- 0.167) = 0.0167$

After iteration 1: $θ = [- 0.0167, 0.0167]$ . The tiny weights make sense: with all predictions at 0.5, the net error is small. The model has barely moved.

Sense-check: With $θ_{1} > 0$ , more hours â†’ higher pass probability. That matches intuition. The model will strengthen this signal over many iterations as the log-loss pushes the boundary to separate the classes.

Scope: When logistic regression works and when it breaks.

Assumptions:

Binary target: The output $y$ takes values in ${0, 1}$ . For multi-class, you need OvR/OvO strategies or softmax regression (covered in Section 13).
Linear decision boundary: The model draws a single straight line (hyperplane) through feature space. If the true classes are separated by a circle or a spiral, logistic regression cannot fit them — unless you first engineer nonlinear features (polynomials, interactions, basis expansions $ϕ (x)$ ).
Independent errors: Training examples are assumed IID (independent and identically distributed). If your data has temporal correlation (time series), this assumption is violated.
No perfect separation: If the data is linearly separable, maximum likelihood drives the weights to infinity — the sigmoid becomes a step function, and the model becomes overconfident. [Bishop §4.3.2 warns about this explicitly.] Regularization (Section 5) prevents this.

What goes wrong when assumptions fail:

Nonlinear boundary â†’ underfitting, poor accuracy. Fix: add polynomial features.
Correlated samples â†’ biased coefficient estimates. Fix: use time-series-aware models.
Perfect separation â†’ infinite weights without regularization. Fix: always use L2 regularization.

Visual Intuition. Picture the sigmoid curve: the x-axis is $z$ (the logit, ranging from $- \infty$ to $+ \infty$ ), and the y-axis is $σ (z)$ (probability, 0 to 1). The curve is S-shaped. At $z = 0$ , it crosses 0.5 — this is the decision threshold. For $z > 2$ , the curve is above 0.88 and flattening out toward 1. For $z < - 2$ , it is below 0.12 and flattening toward 0. The steepest slope is at $z = 0$ , where small changes in the score produce the largest changes in probability. The takeaway: the model is most sensitive near the decision boundary, and most confident far from it.

Now overlay the data points: positive examples cluster on the right (high $z$ ), negatives on the left (low $z$ ). The sigmoid is the "soft" wall between them — a wall you can see through, that gives probabilities instead of hard assignments.

Pitfalls — Common beginner traps the professor flags.

"It has regression in the name, so it predicts numbers." No. Logistic regression predicts probabilities of class membership. It is a classification algorithm. The word "regression" comes from the mathematical update rule matching linear regression — not from the task it solves.

Using MSE as the cost function. If you minimize mean squared error with a sigmoid output, the cost surface is non-convex. Gradient descent may land in a local minimum. Always use log-loss (cross-entropy) for logistic regression. This is why the professor spent time deriving the log-loss in the previous class.

Forgetting to apply the sigmoid. The raw score $z = θ^{⊤} x$ is NOT the prediction. You must pass it through $σ (z)$ to get a probability. A score of $z = 2$ means strong positive signal, but the probability is $σ (2) \approx 0.88$ , not 2.

Interpreting weights as direct effects on probability. A weight $θ_{j} = 0.3$ does NOT mean "increasing $x_{j}$ by 1 increases the probability by 0.3." The effect is nonlinear and depends on where you are on the sigmoid curve. For odds-based interpretation (the correct way), see Section 4.

The exam trap — scikit-learn's C parameter. In code, C = 1/Î». High C = weak regularization, low C = strong regularization. It is the inverse. The exam may test this. Full details in Section 5.

Recap. Logistic regression = linear score + sigmoid squash + log-loss minimization. The gradient update rule is identical to linear regression's, but the hypothesis function is the sigmoid. This marriage of a linear core with a nonlinear output function makes it the simplest principled probabilistic classifier. Bridge: Now that we understand the mathematical engine, we apply it to a real problem — sentiment analysis in Section 2 — where we see how raw text gets converted into the feature vectors this engine expects.

Real-World & Domain Connection. Logistic regression is the workhorse of credit scoring. When a bank decides whether to approve your loan, a logistic regression model (or its close cousin) is often what computes the "credit risk score." Each input feature — income, debt-to-income ratio, number of late payments — gets a weight. The sigmoid converts the weighted sum into a default probability. Regulators like these models because the weights are interpretable: you can explain exactly why an application was denied. In the broader ML field, logistic regression serves as the baseline classifier — the first model you try before reaching for anything more complex. If a deep neural net barely beats logistic regression on your problem, you probably do not need the neural net. It is also the final layer of a binary classification neural network: that last neuron with sigmoid activation is logistic regression sitting on top of learned features.

Symbol Registry (Reference)

Symbol	Meaning	Notation	Type
$x$	Feature vector	$x \in R^{d}$	vector
$θ$ or $w$	Weight vector (parameters)	$θ \in R^{d}$	vector
$θ_{0}$ or $w_{0}$	Bias term (intercept)	scalar	scalar
$z$	Score / logit	$z = θ^{⊤} x$	scalar in $R$
$σ (z)$	Sigmoid function	$σ (z) = \frac{1}{1 + e ^{- z}}$	scalar in $(0, 1)$
$h_{θ} (x)$	Hypothesis (predicted probability)	$h_{θ} (x) = σ (θ^{⊤} x)$	scalar in $(0, 1)$
$y$	True label	$y \in {0, 1}$	scalar
$m$	Number of training examples	scalar	scalar
$α$	Learning rate	scalar	scalar
$J (θ)$	Cost function (cross-entropy)	scalar	scalar

Comparison: Linear vs Logistic Regression

Aspect	Linear Regression	Logistic Regression
Task	Predict continuous values	Classify into binary classes
Hypothesis $h_{θ} (x)$	$θ^{⊤} x$ (identity)	$σ (θ^{⊤} x)$ (sigmoid)
Cost function	Mean squared error (MSE)	Log loss / cross-entropy
Update rule form	$θ_{j} := θ_{j} - α \frac{1}{m} \sum (h_{θ} - y) x_{j}$	Same form
Output domain	$(- \infty, \infty)$	$(0, 1)$
Closed-form solution?	Yes (normal equations)	No (requires iterative optimization)
Convex cost?	Always	Yes, but only with cross-entropy (MSE makes it non-convex)

2. Sentiment Analysis Application

Hook. A computer cannot "read" a movie review. It sees only numbers. Yet with six carefully chosen numbers per review, logistic regression can tell you whether the reviewer loved or hated the film — with 70% confidence. The magic is not in the algorithm; it is in feature engineering: the art of turning words into numbers the math can digest.

Intuition. Imagine you are a detective with a checklist. For each suspect, you tick boxes: "smiled" (positive sign), "avoided eye contact" (negative sign), "said 'no'" (red flag), "used 'I' and 'you' a lot" (emotional involvement). You do not read the whole testimony — you count the signals. That is exactly what logistic regression does: a human designs the checklist (features), and the model learns how many points each check is worth (weights).

Analogy mapping: Features = items on your detective checklist. Weights = how many points each clue is worth toward "guilty" (positive sentiment). The sigmoid converts the total score into a probability. The analogy breaks because the checklist items are rigid — the model cannot invent new clues. If a review says "not bad" but your checklist only counts "bad" as a negative word, you will misclassify it. This is why feature engineering is hard.

Formalize — From Text to Numbers.

Logistic regression requires a fixed-length numerical vector $x \in R^{d}$ . Raw text has variable length and is symbolic. Feature engineering bridges this gap by defining $d$ measurable properties of the text.

The professor's sentiment model uses $d = 6$ features:

Feature	Description	Rationale
$x_{1}$	Count of positive words	Direct signal of positive sentiment
$x_{2}$	Count of negative words	Direct signal of negative sentiment
$x_{3}$	Binary: 1 if "no" appears	The word "no" often flips sentiment ("no good")
$x_{4}$	Count of 1st/2nd person pronouns (I, you, we)	Indicates personal/emotional tone
$x_{5}$	Binary: 1 if "not" appears	Negation marker ("not happy" â†’ negative)
$x_{6}$	$ln (total word count)$	Normalizes for document length

Why $ln$ for word count? A 1000-word review should not get 10Ã— more "sentiment score" than a 100-word review just because it is longer. The log transform compresses the scale: $ln (100) \approx 4.6$ , $ln (1000) \approx 6.9$ — only a 50% difference, not 10Ã—. This prevents raw length from dominating the prediction.

Worked Example — Full Prediction Walkthrough.

Given: A review with: 3 positive words, 2 negative words, contains "no" (1), 3 pronouns, no "not" (0), $ln (word count) = 4.19$ .

Feature vector: $x = [3, 2, 1, 3, 0, 4.19]$

Trained weights: $w = [2.5, - 5.0, - 1.2, 0.5, - 0.8, 0.7]$ , bias $w_{0} = 0.1$ .

Step 1 — Score (logit):

$z = w_{0} + w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} + w_{4} x_{4} + w_{5} x_{5} + w_{6} x_{6} = 0.1 + (2.5 \times 3) + (- 5.0 \times 2) + (- 1.2 \times 1) + (0.5 \times 3) + (- 0.8 \times 0) + (0.7 \times 4.19) = 0.1 + 7.5 - 10.0 - 1.2 + 1.5 + 0 + 2.933 = 0.833$

Step 2 — Sigmoid:

$h_{θ} (x) = σ (0.833) = \frac{1}{1 + e ^{- 0.833}} \approx 0.70$

Interpretation: 70% probability the review is positive. Since $0.70 > 0.5$ , classify as positive.

Sense-check: The score $z = 0.833$ is small and positive. The positive words (+7.5) barely outweigh the negative words (âˆ’10.0) and the "no" penalty (âˆ’1.2). The log-word-count bonus (+2.933) tips it positive. This feels right: the review has slightly more positive signals than negative ones, so moderate confidence makes sense.

Weight wisdom — reading what the model learned:

$w_{1} = + 2.5$ : Each positive word adds 2.5 to the score â†’ strong push toward "positive."
$w_{2} = - 5.0$ : Each negative word subtracts 5.0 â†’ negative words are twice as powerful as positive ones. The model is pessimistic — it weights negative signals more heavily.
$w_{3} = - 1.2$ : The word "no" is a mild negative signal.
$w_{4} = + 0.5$ : Pronouns slightly push toward positive (emotional reviews tend to be opinionated).
$w_{5} = - 0.8$ : "Not" is a mild negative indicator.
$w_{6} = + 0.7$ : Longer reviews slightly favor positive classification.

Scope: Feature engineering is model-agnostic. These six features could feed any classifier — logistic regression, decision trees, SVMs. The model just sees numbers. The quality of your features determines the ceiling of your model's performance. The professor chose these features because they are interpretable and computationally cheap. Modern NLP uses word embeddings (dense vectors learned from data) which capture richer semantics but are harder to interpret.

Pitfalls.

Negation blindness. Counting "good" as positive and "not" as present still misses that "not good" means BAD. Simple bag-of-words features cannot capture word order or negation scope. This is why sentiment analysis is harder than it looks.
Domain mismatch. Positive/negative word lists trained on movie reviews may fail on medical text ("the tumor is positive for malignancy" — "positive" means bad here).
Feature scale mismatch. If $x_{6}$ (log word count) were replaced with raw word count (e.g., 1000 vs 4.19), its weight $w_{6} = 0.7$ would contribute $0.7 \times 1000 = 700$ to the score — completely swamping all other features. Log scaling prevents this.

Recap. Sentiment analysis = feature engineering + logistic regression. The model does not read text; it counts signals. The weights tell you which signals matter and how much. Bridge: The same weight-interpretation logic applies to any logistic regression model — in Section 4, we learn to read weights through the lens of odds for even sharper interpretation.

Real-World & Domain Connection. Sentiment analysis powers brand monitoring. Companies like Coca-Cola and Apple track Twitter mentions in real time. A logistic regression classifier (or its deep-learning descendant) flags every tweet as positive, negative, or neutral. The aggregate sentiment trend is a leading indicator of PR crises and campaign effectiveness. Logistic regression is favored when regulators demand explainability: you can point to the exact words (features) that drove a classification, unlike a black-box neural net.

3. Gradient Descent in Logistic Regression — Worked Iteration

Hook. You initialize random weights and ask the model to predict. It confidently says "YES" to everything — even the three candidates who were rejected. One round of gradient descent later, the model has already changed its mind on two features, flipping their weights from positive to negative. How does it know to do that? The answer is in the gradient — the error signal that whispers to each weight: "too high" or "too low."

Purpose. Gradient descent for logistic regression minimizes the cross-entropy cost $J (θ)$ by iteratively adjusting weights in the direction of steepest descent. Unlike linear regression, there is no closed-form solution — the sigmoid nonlinearity forces us to iterate. [Bishop §4.3.3 confirms: no closed form for logistic regression; the error function is concave with a unique minimum, solved via iterative methods like IRLS or gradient descent.]

Inputs: Training set of $m$ examples ${(x^{(i)}, y^{(i)})}_{i = 1}^{m}$ , initial weights $θ$ , learning rate $α$ .

Outputs: Optimized weights $θ$ that minimize $J (θ)$ .

Steps — One Iteration of Batch Gradient Descent.

Forward pass: For every training example $i$ , compute $z^{(i)} = θ^{⊤} x^{(i)}$ then $h_{θ}^{(i)} = σ (z^{(i)})$ .
Error computation: For each example, compute the error $e^{(i)} = h_{θ}^{(i)} - y^{(i)}$ .
Gradient accumulation: For each weight $θ_{j}$ , accumulate $\frac{1}{m} \sum_{i = 1}^{m} e^{(i)} x_{j}^{(i)}$ .
Weight update: $θ_{j} := θ_{j} - α \cdot gradient_{j}$ .
Repeat from step 1 with the new weights until convergence.

Why this works: The gradient $\frac{\partial J}{\partial θ _{j}} = \frac{1}{m} \sum (h_{θ} - y) x_{j}$ points uphill on the cost surface. Subtracting $α$ times the gradient moves downhill. When $h_{θ} > y$ (over-predicting), the error is positive, so the weight decreases. When $h_{θ} < y$ (under-predicting), the error is negative, so the weight increases.

Trace — The Job Offer Dataset.

Dataset (6 records, features = [CGPA, IQ], target = job offered?):

i	CGPA	IQ	Job? (y)
1	4.0	110	1
2	2.5	90	0
3	3.5	105	1
4	3.0	95	1
5	2.0	85	0
6	2.8	88	0

Initial: $θ = [0.5, 0.5, 0.5]$ , $α = 0.3$ .

Iteration 1 — Forward pass. Every $z^{(i)} = 0.5 + 0.5 \cdot CGPA + 0.5 \cdot IQ$ is large (>40 for all records), so every $h_{θ}^{(i)} = σ (large) \approx 1.000$ .

Errors: $e^{(i)} = h_{θ}^{(i)} - y^{(i)}$ :

Records 1,3,4: $1.000 - 1 = 0.000$ (correct predictions, no correction needed)
Records 2,5,6: $1.000 - 0 = 1.000$ (wrong — predicted yes, actual no)

Weight updates:

$θ_{0}$ : gradient = $\frac{1}{6} (0 + 1 + 0 + 0 + 1 + 1) = \frac{3}{6} = 0.5$ â†’ $θ_{0} := 0.5 - 0.3 \times 0.5 = 0.35$

$θ_{1}$ (CGPA): gradient = $\frac{1}{6} (0 \cdot 4.0 + 1 \cdot 2.5 + 0 \cdot 3.5 + 0 \cdot 3.0 + 1 \cdot 2.0 + 1 \cdot 2.8) = \frac{7.3}{6} \approx 1.217$ â†’ $θ_{1} := 0.5 - 0.3 \times 1.217 \approx 0.135$

$θ_{2}$ (IQ): gradient = $\frac{1}{6} (0 \cdot 110 + 1 \cdot 90 + 0 \cdot 105 + 0 \cdot 95 + 1 \cdot 85 + 1 \cdot 88) = \frac{263}{6} \approx 43.83$ â†’ $θ_{2} := 0.5 - 0.3 \times 43.83 \approx - 12.65$

After iteration 1: $θ \approx [0.35, 0.135, - 12.65]$ . The IQ weight plummeted because IQ values are large (~100), so the same error gets multiplied by a large $x_{j}$ . This is why the professor's result $[0.4, - 0.4, - 0.6]$ differs — the exact numbers depend on the precise dataset values. The direction is what matters: weights drop, especially for large-magnitude features.

Sense-check: $θ_{2}$ becoming strongly negative means the model is learning "high IQ â†’ LESS likely to get the job" — purely a pattern in this small dataset, not a causal claim.

Three Variants of Gradient Descent.

Type	Data per step	Update frequency	Noise	Use case
Batch	All $m$ examples	Once per epoch	None (deterministic)	Small datasets, stable convergence
Mini-batch	$b$ examples ( $1 < b < m$ )	$m / b$ times per epoch	Moderate	Default for most ML
Stochastic (SGD)	1 example	$m$ times per epoch	High (noisy)	Large datasets, online learning

The professor's examples use batch GD for clarity. In practice, mini-batch SGD is the workhorse.

Student Q&A (deduplicated).

Q: After the first iteration, do we recompute $h_{θ} (x)$ before the second iteration? A: Yes — this is essential. After updating all $θ$ values, you must run the forward pass again with the new weights. Then compute new errors, new gradients, and new updates. This loop (forward â†’ error â†’ gradient â†’ update) repeats every iteration. Several students asked this — it is the most common point of confusion in gradient descent homework.

Pitfalls.

Forgetting to recompute $h_{θ}$ . If you reuse old predictions after updating weights, you are not doing gradient descent — you are doing something nonsensical. Every iteration starts with a fresh forward pass.
Feature scale dominates gradients. Notice how $θ_{2}$ (IQ, values ~100) changed far more than $θ_{1}$ (CGPA, values ~4). Large-range features get large gradients. Always scale features before gradient descent (see Section 12).
Learning rate too large. With $α = 0.3$ and unscaled IQ, $θ_{2}$ jumped from 0.5 to âˆ’12.65 in one step. This is unstable. A smaller $α$ or feature scaling would help.
Exam trap — the professor's exact numbers. The professor's result $[0.4, - 0.4, - 0.6]$ comes from a specific dataset. On an exam, you will be given the exact dataset and must compute the numbers precisely. Do not memorize the result — learn the process.

Recap. One gradient descent iteration = forward pass â†’ error â†’ gradient â†’ update. The gradient automatically corrects over-optimistic predictions by lowering weights. The update rule is identical to linear regression's, but $h_{θ}$ is the sigmoid. Bridge: After training, we have a set of weights. Section 4 teaches us how to read those weights — what does $θ_{1} = 0.3$ actually mean for a real prediction?

Real-World & Domain Connection. Batch gradient descent on the full dataset is rare in industry. Stochastic gradient descent (SGD) and its variants (Adam, RMSprop) train every deep neural network you have heard of — from ChatGPT to image recognition. The core idea — "compute error, take a small step downhill" — is the same whether you have 6 examples or 6 billion. Logistic regression with SGD was the first production ML system at Google (for ad click prediction, circa 2007).

4. Model Interpretation and Odds

Hook. A trained logistic regression model spits out an equation like $z = 0.4 + 0.3 \times CGPA - 0.45 \times IQ$ . Three numbers. But buried in those three numbers is a complete story about who gets the job and why. Reading that story requires understanding two things: what the sign tells you (direction) and what $e^{θ}$ tells you (magnitude, via odds). This is the interpretability superpower that makes logistic regression the go-to model in regulated industries.

Intuition — Probability vs Odds. Probability and odds are two ways to say the same thing, like Celsius and Fahrenheit. Probability: "80% chance of rain." Odds: "4 to 1 in favor of rain." Same information, different number. People confuse them constantly. Here is the rule:

Probability $P \in [0, 1]$ : what fraction of the time something happens.
Odds $= \frac{P}{1 - P} \in [0, \infty)$ : the ratio of "it happens" to "it does not."

Analogy: A biased coin. Probability of heads = 0.75 means in 100 flips, ~75 are heads. Odds = 0.75/0.25 = 3, meaning "3 heads for every 1 tail." The math: odds = P/(1âˆ’P). To go back: P = odds/(1+odds).

Formalize — The Logit-Odds Connection.

Start from the sigmoid: $P (y = 1 ∣ x) = \frac{1}{1 + e ^{- z}}$ where $z = θ^{⊤} x$ .

Solve for $z$ :

$P P (1 + e^{- z}) P + P e^{- z} P e^{- z} e^{- z} e^{z} = \frac{1}{1 + e ^{- z}} = 1 = 1 = 1 - P = \frac{1 - P}{P} = \frac{P}{1 - P}$

So $e^{z} = odds$ . Taking $ln$ of both sides: $z = ln (odds)$ . This is why $z$ is called the log-odds or logit.

Now plug in $z = θ^{⊤} x$ :

$ln (\frac{P}{1 - P}) = θ_{0} + θ_{1} x_{1} + \dots + θ_{n} x_{n}$

This is the log-odds form of logistic regression. It shows that each $θ_{j}$ has an additive effect on the log-odds, and therefore a multiplicative effect on the odds:

$Odds = e^{θ_{0}} \cdot e^{θ_{1} x_{1}} \cdot \dots \cdot e^{θ_{n} x_{n}}$

When $x_{j}$ increases by 1, the odds get multiplied by $e^{θ_{j}}$ . This called the odds ratio for feature $j$ .

Worked Examples.

Model: $z = 0.4 + 0.3 \times CGPA - 0.45 \times IQ$

Prediction for CGPA=5, IQ=6:

$z = 0.4 + 1.5 - 2.7 = - 0.8 \Rightarrow P = σ (- 0.8) = \frac{1}{1 + e ^{0.8}} \approx 0.31$

Classify as no job (0.31 < 0.5). Sense-check: IQ's negative weight outweighs CGPA's positive weight â†’ prediction tilts negative. Correct.

Odds interpretation for CGPA ( $θ_{1} = 0.3$ ):

$e^{0.3} \approx 1.35$

"Each +1 point of CGPA multiplies the odds of getting the job by 1.35." If your current odds are 2:1 (probability = 0.67), a 1-point CGPA increase makes odds â‰ˆ 2.7:1 (probability â‰ˆ 0.73). The effect is multiplicative on odds, not additive on probability.

Odds interpretation for IQ ( $θ_{2} = - 0.45$ ):

$e^{- 0.45} \approx 0.638$

"Each +1 point of IQ multiplies the odds by 0.638" — approximately a 36% reduction in odds. A negative weight means $e^{θ} < 1$ , so odds shrink.

Critical distinction: $θ_{j} = 0.3$ does NOT mean "probability increases by 0.3." The effect on probability depends on where you start on the S-curve. A 1-unit increase in CGPA has a different probability impact if your current probability is 0.1 vs 0.5 vs 0.9.

Scope: Odds ratios apply only when features are independent. If CGPA and IQ are correlated (multicollinearity), the odds ratio for CGPA "holding IQ constant" becomes less reliable. The number $e^{0.3} = 1.35$ still comes out of the math, but its real-world interpretation as "the isolated effect of CGPA" is weakened.

Visual Intuition. Plot the log-odds line: x-axis = CGPA (say 0–10), y-axis = $ln (odds)$ . This is a straight line with slope 0.3. Now plot the probability curve: x-axis = CGPA, y-axis = $P$ . This is an S-curve, steepest at $P = 0.5$ . The same 0.3 slope in log-odds space becomes a varying slope in probability space — steep near 0.5, flat near 0 or 1. The takeaway: log-odds space is linear and additive; probability space is not.

Student Q&A (deduplicated).

Q: How do we interpret a negative weight? Several students asked variants of this. A: A negative $θ_{j}$ means as $x_{j}$ increases, the log-odds decrease, so the probability of the positive class goes down. For IQ with $θ_{2} = - 0.45$ : higher IQ â†’ lower job-offer probability in this dataset. This is correlation, not causation. The data happened to show that pattern. The odds ratio $e^{- 0.45} \approx 0.64$ quantifies it precisely: each IQ point multiplies odds by 0.64.

Q: What does "4 to 1 odds" actually mean? A: If the probability of passing is $P$ , and odds are 4:1, then $P / (1 - P) = 4$ , so $P = 4/5 = 0.8$ . It means: over many independent trials, you expect 4 passes for every 1 failure. In a single trial, it means the event is 4Ã— as likely to happen as not.

Q: Are the two classes mutually exclusive? A: Yes — a candidate either gets the job or does not. But the model can express mixed signals via the score $z$ . When positive features push $z$ up and negative features push $z$ down, the net $z$ can land near 0, giving $P \approx 0.5$ — the model is torn, not contradictory.

Recap. Read weights through the lens of odds: sign = direction, $e^{θ_{j}}$ = multiplicative effect on odds. The log-odds $z$ is linear in the features; the probability $P$ is not. Bridge: Interpretation is great — but a model with massive weights is overfit and uninterpretable. Section 5 introduces regularization, which keeps weights small enough to trust.

Real-World & Domain Connection. Odds ratios are the lingua franca of medical statistics. When you read "smoking increases the odds of lung cancer by a factor of 15," that 15 is $e^{θ_{smoking}}$ from a logistic regression. Epidemiologists report odds ratios because they are independent of the baseline risk — the factor of 15 applies whether your baseline risk is 0.1% or 10%. Logistic regression dominates medical literature for precisely this interpretability.

5. Regularization in Logistic Regression

Hook. You train a logistic regression model and get 99.8% accuracy on your training data. You celebrate. Then you test it on new data — 62%. Your model memorized the training set instead of learning. This is overfitting, and the cure is regularization: a mathematical speed limiter that says "you may fit the data, but you may not use gigantic weights to do it."

Intuition — The Speed Limiter Analogy. A car with no speed limiter can go 200 km/h on a straight road — but it will crash on the first curve. A speed limiter set to 80 km/h forces the driver to find a path that works at moderate speed everywhere. Regularization is that limiter for weights. It says: "You cannot set $θ_{1} = 500$ just to perfectly separate these 10 points. Keep the weights small, and find a boundary that works reasonably for everyone."

Analogy break point: A real speed limiter hard-caps the speed. Regularization applies a soft penalty — large weights are allowed only if they reduce the error enough to justify the penalty. It is a trade-off, not a hard limit.

Formalize — Adding a Penalty to the Cost.

The regularized cost function:

$J (θ) = Log-loss (data fit) - \frac{1}{m} i = 1 \sum m [y^{(i)} ln h_{θ} (x^{(i)}) + (1 - y^{(i)}) ln (1 - h_{θ} (x^{(i)}))] + Penalty λ \cdot R (θ)$

The penalty $R (θ)$ does NOT include $θ_{0}$ (the bias). Regularizing the bias would force the decision boundary through the origin, which is rarely desired.

L1 (Lasso): $R (θ) = \sum_{j = 1}^{n} ∣ θ_{j} ∣$ — sum of absolute values.

Drives some weights to exactly zero â†’ automatic feature selection.
Use when you suspect many features are irrelevant.
The gradient becomes: $\frac{\partial J}{\partial θ _{j}} = \frac{1}{m} \sum (h_{θ} - y) x_{j} + λ \cdot sign (θ_{j})$ for $θ_{j} \neq = 0$ .

L2 (Ridge): $R (θ) = \frac{1}{2} \sum_{j = 1}^{n} θ_{j}^{2}$ — sum of squared values (the $\frac{1}{2}$ is for cleaner derivatives).

Shrinks all weights toward zero but never to exactly zero.
Use as the default — it is differentiable everywhere and numerically stable.
The gradient becomes: $\frac{\partial J}{\partial θ _{j}} = \frac{1}{m} \sum (h_{θ} - y) x_{j} + λ θ_{j}$ .

The new update rule with L2: $θ_{j} := θ_{j} - α (\frac{1}{m} \sum (h_{θ} - y) x_{j} + λ θ_{j})$ .

Regularization prevents overfitting with linearly separable data: [Bishop §4.3.2 warns] When data is perfectly separable, maximum likelihood drives $∥ w ∥ \to \infty$ — the sigmoid becomes a step function, and every training point gets $P = 1$ for its true class. The model is infinitely confident about noise. Regularization's penalty on $∥ w ∥$ prevents this divergence.

The C Parameter Trap (Exam-Critical).

Scikit-learn's LogisticRegression uses C instead of $λ$ :

$C = \frac{1}{λ}$

This is the inverse of regularization strength:

`C` value	$λ$ equivalent	Regularization	Model behavior
`C = 0.01`	$λ = 100$	Very strong	Simple model, high bias, low variance
`C = 1.0`	$λ = 1$	Moderate	Default, balanced
`C = 100`	$λ = 0.01$	Very weak	Complex model, low bias, high variance, risk of overfitting

The trap: "I want strong regularization so I'll set C=100." Wrong. That gives the weakest regularization. High C = low penalty = large weights allowed.

Notation warning: Some textbooks use $α$ for the regularization parameter and others use $λ$ . Scikit-learn uses $C = 1/ α$ internally. The professor uses $λ$ . On the exam, follow the professor's notation. In code, remember $C = 1/ λ$ .

Scope: When to use which regularizer.

L2 (Ridge) is the safe default. Differentiable, stable, works with any optimizer. Use when you do not know which features matter.
L1 (Lasso) for feature selection. Use when you have many features and suspect most are noise. L1 will zero out the useless ones automatically.
Elastic Net (L1 + L2 combined): Best of both worlds. Available in scikit-learn as penalty='elasticnet' with the l1_ratio parameter.
No regularization is almost never correct. Even with plenty of data, a tiny L2 penalty ( $λ = 1 0^{- 4}$ ) improves numerical stability.

Pitfalls.

The C-inverse trap. High C = weak regularization. Low C = strong. This is the single most common scikit-learn mistake. The exam will test this.
Regularizing the bias. Always exclude $θ_{0}$ from the penalty. If you penalize the bias, the model cannot shift the decision boundary freely, degrading performance.
Not scaling before regularizing. L1 and L2 penalize weight magnitudes. If CGPA is in [0,10] and IQ is in [50,150], the IQ weight looks "larger" and gets penalized more — even if IQ is genuinely more important. Always standardize features before applying regularization.
$λ$ too large. If every weight is driven to zero, the model always predicts the majority class. This is underfitting. Tune $λ$ using cross-validation.

Recap. Regularization = penalty on large weights. L1 zeros out features; L2 shrinks them. In scikit-learn, C = 1/Î» — the inverse relationship is exam-critical. Bridge: With a trained, regularized model in hand, we now ask: how good is it? Section 6 introduces the confusion matrix, the first tool for answering that question.

Real-World & Domain Connection. In credit scoring, regulators require models that do not overfit to historical biases. An L1-regularized logistic regression might zero out protected attributes (race, gender, zip code) while keeping legitimate predictors (income, debt ratio). This automatic feature selection is auditable: you can show regulators exactly which features the model uses and prove that prohibited variables were excluded. The same L1 penalty also helps when you have thousands of potential features (every word in a document, every gene in a microarray) — L1 picks the few that matter.

6. Evaluating Classifiers — The Confusion Matrix

Hook. Your model says "cancer" and the patient does not have it — stressful, expensive, but fixable with a second test. Your model says "no cancer" and the patient does have it — potentially fatal. These two mistakes have the same name ("wrong") but wildly different costs. The confusion matrix is the tool that separates these two kinds of wrong so you can decide which one matters more.

Intuition — The Report Card Analogy. A confusion matrix is a 2Ã—2 report card. The rows are what the teacher (reality) says. The columns are what the student (model) answered. The diagonal (TP, TN) = correct answers. The off-diagonal (FP, FN) = mistakes — but two different kinds of mistakes. It is like a medical test: a false positive is a false alarm; a false negative is a missed detection. Same word "wrong," opposite consequences.

Formalize — The 2Ã—2 Confusion Matrix.

Predicted: Positive	Predicted: Negative
Actual: Positive	TP — True Positive	FN — False Negative (Type II Error)
Actual: Negative	FP — False Positive (Type I Error)	TN — True Negative

Memory aid — read each term as "[truly/falsely] predicted as [positive/negative]":

TP: actually yes, said yes â†’ correct hit.
TN: actually no, said no â†’ correct rejection.
FP: actually no, said yes â†’ false alarm. Type I Error. Think: "False Positive = Falsely cried 'Positive!'"
FN: actually yes, said no â†’ missed detection. Type II Error. Think: "False Negative = Falsely said 'Negative.'"

Total records = TP + FP + FN + TN. All metrics in Sections 7–10 are built from these four numbers.

Worked Example — Car Price Classification.

A model predicts whether a used car is "Low Price" (positive class) or "High Price" (negative class). Results on 100 cars:

Predicted: Low	Predicted: High	Total
Actual: Low	TP = 15	FN = 5	20
Actual: High	FP = 10	TN = 70	80
Total	25	75	100

15 low-price cars correctly flagged (TP).
5 low-price cars missed (FN) — the model said "high price" but they were actually bargains.
10 high-price cars wrongly flagged as low (FP) — false alarms.
70 high-price cars correctly identified (TN).

Sense-check: Out of 20 actual low-price cars, the model found 15 (75% hit rate = recall). Out of 25 "low" predictions, 15 were real (60% precision). Both metrics need improvement, but the matrix tells us exactly where.

The Asymmetric Cost of Errors — This Is a Design Decision.

Scenario	Positive =	FP cost	FN cost	Which is worse?
Cancer screening	Has cancer	Unnecessary biopsy (stress, cost)	Missed cancer (life-threatening)	FN
Spam filter (auto-delete)	Is spam	Delete legitimate email (career-ending)	Spam in inbox (annoying)	FP
Nuclear facility access	Authorized person	Let in intruder (catastrophic)	Lock out employee (inconvenience)	FP
Fraud detection	Fraudulent transaction	Investigate legitimate tx (operational cost)	Miss fraud (financial loss)	FN

The rule: You, the ML engineer, decide which error is worse based on the real-world cost — not the math. Then choose the metric (precision vs recall) that penalizes the worse error. There is no universal answer.

Pitfalls.

Confusing which class is "positive." The positive class is whatever you define it to be — usually the rarer or more important class (disease = positive, fraud = positive). Swapping positive and negative flips TPâ†”TN and FPâ†”FN.
Thinking FP and FN are equally bad. They never are. The confusion matrix exists precisely because the two errors have different costs.
Forgetting that Type I = FP, Type II = FN. Mnemonic: Type I = you Falsely rejected the null (False Positive). Type II = you Failed to reject the null when you should have (False Negative).

Recap. The confusion matrix separates "correct" into TP/TN and "wrong" into FP/FN. The two types of wrong have different real-world costs. Bridge: The simplest summary of the matrix is accuracy — all correct divided by all predictions. Section 7 shows why this simplest summary can be the most misleading.

Real-World & Domain Connection. Confusion matrices are the standard output of every medical diagnostic test evaluation. When the FDA approves a new COVID test, the submission includes a confusion matrix against a gold-standard PCR test. Sensitivity (recall) and specificity (TNR) are reported in every medical journal. The same framework applies to evaluating any binary classifier — from spam filters to facial recognition systems.

7. Accuracy and Its Pitfalls

Hook. You build a fraud detector. It predicts "not fraud" for every transaction. Accuracy: 99.7%. You celebrate — until you realize it caught zero fraudulent transactions. This is the accuracy paradox: the most dangerously useless model can have the highest accuracy. Accuracy is the metric everyone asks for and the metric you should almost never trust alone.

Intuition. Accuracy answers: "What fraction of my predictions were right?" That sounds good — until one class is rare. If 99.7% of transactions are legitimate, a model that always says "legitimate" is 99.7% accurate and 0% useful. Accuracy is like grading a student who answered only the easy questions: the score looks great, but it tells you nothing about whether they can handle the hard ones.

Formalize.

$Accuracy = \frac{T P + T N}{T P + T N + F P + F N} = \frac{correct predictions}{all predictions}$

Accuracy treats every correct prediction equally — whether it was an easy majority-class case or a hard minority-class case. This is its fatal flaw.

The Accuracy Paradox (proved by example):

Scenario	Total	Actual Positive	Actual Negative	Dumb model predicts	TP	TN	FP	FN	Accuracy
Balanced	200	100	100	All "Positive"	100	0	100	0	50%
Imbalanced	1000	10	990	All "Negative"	0	990	0	10	99%

In the balanced case, the dumb model is exposed (50%). In the imbalanced case, the same dumb strategy earns 99%. The model learned nothing — it simply guessed the majority class. [The T1_5 reference on evaluating hypotheses confirms: sample error can be deceptively low when the hypothesis space is rich and the model overfits.]

Worked Example — The Fraud Detection Trap.

You have 10,000 credit card transactions: 30 are fraudulent (0.3%), 9,970 are legitimate (99.7%).

Model A (lazy): Always predicts "legitimate."

TP = 0, FN = 30 (missed ALL fraud)
FP = 0, TN = 9,970 (perfect on legitimate)
Accuracy = 9,970/10,000 = 99.7% â† Looks amazing!
Recall = 0/30 = 0% â† Actually useless!

Model B (real): Catches 25/30 fraud cases, but flags 50 legitimate transactions.

TP = 25, FN = 5
FP = 50, TN = 9,920
Accuracy = (25+9,920)/10,000 = 99.45% â† Lower than Model A!
Recall = 25/30 = 83.3% â† Actually useful!

The trap: Model A has higher accuracy (99.7% vs 99.45%) but is worthless. Model B is the one you deploy. Never rank models by accuracy on imbalanced data.

Scope: When accuracy IS acceptable. Accuracy works when classes are roughly balanced (40/60 split or better) AND both types of error have similar costs. Example: classifying cat vs dog images in a dataset with 500 cats and 500 dogs. Accuracy is fine here. The moment one class drops below ~20% or the costs become asymmetric, switch to F1, precision, or recall.

Pitfalls — The Professor's Exam Flags.

99% training accuracy = overfitting, not success. The professor explicitly warned: if you see 99% accuracy on training data, be suspicious. The model has memorized, not learned. Expect much lower test accuracy.
50% accuracy is not necessarily bad. On a genuinely hard, balanced problem, 50% might mean the model is trying and there is room for improvement. It is more honest than a fake 99%.
Never report accuracy without the class distribution. "95% accuracy" means nothing without knowing the positive/negative split. Always report: "95% accuracy (5% positive, 95% negative)."
Exam trap — the accuracy paradox question. The exam likes showing a confusion matrix on imbalanced data and asking: "Is accuracy a good metric here?" The answer is always NO — explain why and suggest F1 or recall instead.

Recap. Accuracy = (TP+TN)/total. It is the most intuitive metric and the most dangerous one on imbalanced data. Always check the class distribution before trusting it. Bridge: If accuracy fails when one class is rare, what should we use instead? Section 8 introduces precision, recall, and F1 — the trio that does not care about class balance.

Real-World & Domain Connection. In 2015, a famous paper showed that several published medical AI models with >90% accuracy were no better than random guessing when evaluated properly. The culprit? Severe class imbalance (disease prevalence <1%) and reliance on accuracy as the sole metric. This led to widespread adoption of AUC and F1 as mandatory reporting metrics in medical AI journals. Accuracy is now considered insufficient for any paper submission in most ML venues when data is imbalanced.

8. Precision, Recall, and F1 Score

Hook. A spam filter that deletes every email it thinks is spam. Great precision: 99% of deleted emails are truly spam. Terrible recall: it catches only 20% of actual spam. Meanwhile, the other 80% of spam floods your inbox. Which model is "better" — the cautious deleter or a reckless one that catches 99% of spam but also deletes 10% of your real email? The answer depends on whether you hate spam in your inbox more than you fear losing a real message. Precision and recall force you to choose.

Intuition — The Fisherman Analogy. You are fishing in a lake with a net.

Precision: Of all the fish in your net, what fraction are the kind you want? A small, carefully placed net (high precision, low recall).
Recall: Of all the desirable fish in the lake, what fraction did you catch? A giant trawler net (high recall, low precision — you catch everything, including trash).

You cannot maximize both with one net. A small net misses fish (low recall). A giant net catches garbage (low precision). The F1 score is the compromise — it penalizes nets that are terrible at either one.

Formalize — The Metrics.

Metric	Formula	Question It Answers
Precision (PPV)	$\frac{T P}{T P + F P}$	"When I say 'yes,' am I right?"
Recall (Sensitivity, TPR)	$\frac{T P}{T P + F N}$	"Did I find all the 'yes' cases?"
FPR (False Positive Rate)	$\frac{F P}{F P + T N}$	"What fraction of actual 'no' did I wrongly flag?"
Specificity (TNR)	$\frac{T N}{T N + F P}$	"What fraction of actual 'no' did I correctly clear?"
F1 Score	$2 \cdot \frac{P \cdot R}{P + R}$	"How balanced are my precision and recall?"

Why harmonic mean for F1? The harmonic mean $\frac{2 P R}{P + R}$ is always â‰¤ the arithmetic mean $\frac{P + R}{2}$ . When P and R are far apart, the harmonic mean plummets toward the smaller one. This is the desired behavior: a model with P=0.99 and R=0.01 should score near 0, not 0.5.

$If P = 0.99, R = 0.01 : F 1 = 2 \cdot \frac{0.99 \cdot 0.01}{0.99 + 0.01} = \frac{0.0198}{1.0} = 0.0198 \approx 0$ $Arithmetic mean would give: \frac{0.99 + 0.01}{2} = 0.5 — deceptively decent!$

Worked Example — The F1 Penalty in Action.

From the car price example (Section 6): TP=15, FP=10, FN=5, TN=70.

Precision = 15/(15+10) = 15/25 = 0.60 (60%)
Recall = 15/(15+5) = 15/20 = 0.75 (75%)
F1 = 2 Ã— (0.60 Ã— 0.75)/(0.60 + 0.75) = 2 Ã— 0.45/1.35 = 0.667

The F1 is closer to the lower number (0.60) than the higher (0.75) — the harmonic mean penalizes the weaker metric.

Sense-check: The model is decent but not great. It catches 75% of low-price cars (good recall) but 40% of its "low" flags are false alarms (middling precision). F1=0.67 reflects this balance honestly.

The Precision-Recall Trade-off — Comparison.

Model Type	Threshold	Precision	Recall	When to use
Conservative	High (predict "yes" only when very sure)	High	Low	Spam auto-delete, nuclear access
Aggressive	Low (predict "yes" liberally)	Low	High	Cancer screening, fraud detection
Balanced	Tuned via cross-validation	Moderate	Moderate	Most applications

You tune this trade-off by moving the decision threshold away from 0.5. Lower threshold â†’ more "yes" predictions â†’ higher recall, lower precision. Higher threshold â†’ fewer "yes" predictions â†’ higher precision, lower recall.

Pitfalls.

Precision=1.0 with recall=0.0 is useless. A model that predicts "positive" exactly once, correctly, has perfect precision — and found 1% of the positives. Always report both numbers.
F1 assumes precision and recall are equally important. If recall is 10Ã— more important than precision (cancer screening), use F_beta with Î²>1. The general form: $F_{β} = (1 + β^{2}) \frac{P \cdot R}{β ^{2} P + R}$ . For Î²=2, recall is weighted 2Ã— more than precision.
Don't compare F1 across datasets with different class ratios. F1 depends on the positive class prevalence. An F1 of 0.8 on a balanced dataset is not the same as 0.8 on a 1:100 imbalanced dataset.

Recap. Precision = accuracy of positive predictions. Recall = coverage of actual positives. F1 = harmonic mean that penalizes imbalance between the two. Choose based on which error costs more. Bridge: Section 9 puts these metrics to work by comparing three real classifier personalities — cautious, aggressive, and ideal — to show that "best" depends entirely on context.

Real-World & Domain Connection. In information retrieval (Google Search), precision = "of the results I showed, how many were relevant?" and recall = "of all relevant documents, how many did I show?" Google optimizes for precision@10 (the top 10 results) because users rarely go past page 1. In e-discovery (legal document review for lawsuits), recall is paramount — missing one relevant document can cost millions in sanctions. Same metrics, different priorities.

9. Comparing Classifiers — Three Use Cases

Hook. Three models, same data, same algorithm — three completely different personalities. One is paranoid and rarely says "yes." One is reckless and says "yes" to almost everything. One is nearly perfect. Which would you deploy in a hospital? In a nuclear facility? In your email spam filter? The answer is different for each.

Comparison — Three Models on Balanced Data (100 Pos, 100 Neg).

Metric	Model 1 (Cautious)	Model 2 (Aggressive)	Model 3 (Ideal)
TP	50	99	99
FP	1	10	1
TN	99	90	99
FN	50	1	1
Precision	50/51 = 0.98	99/109 = 0.90	99/100 = 0.99
Recall	50/100 = 0.50	99/100 = 0.99	99/100 = 0.99
F1	0.66	0.94	0.99
Personality	"Only when I'm sure"	"Better safe than sorry"	"Too good to be true"

Model 1 — The Cautious Classifier (Precision=0.98, Recall=0.50).

When this model says "yes," it is right 98% of the time. But it only says "yes" to half the actual positives — it missed 50 of them. This is a high-precision, low-recall profile.

Deploy this for: Nuclear facility retinal scan. You would rather occasionally lock out a real employee (low recall, inconvenience) than ever let in an intruder (even one FP could be catastrophic). The model's "yes" must be essentially infallible.

Professor's characterization: "Too cautious — afraid of making a false positive."

Model 2 — The Aggressive Classifier (Precision=0.90, Recall=0.99).

This model catches 99% of all actual positives — only 1 slipped through. But it also flagged 10 negatives as positive (false alarms). This is a high-recall, moderate-precision profile.

Deploy this for: Cancer screening. Missing 1 cancer patient (FN) could be fatal. Flagging 10 healthy people causes stress and unnecessary biopsies — but a follow-up test clears them. The cost of FN >> cost of FP.

Professor's characterization: "Willing to flag many things as positive to catch almost all actual positives."

Model 3 — The "Ideal" Classifier (Precision=0.99, Recall=0.99, F1=0.99).

Almost perfect. Only 1 mistake in each direction.

Professor's warning: "If you see these numbers in a real project, be doubtful. It is too good to be true."

On an exam, Model 3 is the best by the numbers. In practice, Model 2 (F1â‰ˆ0.94) is far more realistic and often the better choice because its recall-priority aligns with real-world costs.

The Universal Decision Rule.

Application	Positive Class	Costlier Error	Choose Model Type	Prioritize Metric
Cancer screening	Has disease	FN (missed diagnosis)	Aggressive	High Recall
Spam auto-delete	Is spam	FP (lost real email)	Cautious	High Precision
Nuclear facility	Authorized	FP (intruder)	Cautious	High Precision
Fraud detection	Fraudulent	FN (missed fraud)	Aggressive	High Recall

The iron rule: Identify the costlier error â†’ choose the metric that penalizes it â†’ tune the model accordingly. Context determines everything.

Scope: Medium and High Skew Cases. As class imbalance increases, the same principles apply but metrics become even more critical. Accuracy becomes progressively more misleading. F1, precision, and recall remain reliable. For high-skew cases (1:100 or worse), consider precision-recall curves instead of ROC curves (see Section 10).

Student Q&A (deduplicated).

Q: How do we know if a feature is negatively correlated? A: Look at the sign of $θ_{j}$ . Negative sign â†’ as $x_{j}$ increases, probability of the positive class decreases. The model learned this from the training data. It is correlation, not causation.

Recap. No single "best" model exists independent of context. The cautious model (high precision) and the aggressive model (high recall) serve different masters. Choose based on which error you can least afford. Bridge: The ROC curve (Section 10) visualizes the entire precision-recall trade-off across all possible thresholds — so you can pick your operating point with your eyes open.

Real-World & Domain Connection. In credit scoring, US law (Equal Credit Opportunity Act) requires lenders to provide the specific reasons for denial. A high-precision logistic regression model with interpretable weights can output: "Denied because: high debt-to-income ratio, recent late payment." A black-box model with higher F1 but no explanations is legally non-compliant. Precision + interpretability beats raw F1 in regulated industries.

10. ROC Curve and AUC

Hook. Your logistic regression model outputs probabilities, not hard labels. You can set the threshold at 0.5, 0.3, or 0.9 — each choice gives different TP, FP, TN, FN counts. Which threshold is best? The ROC curve answers this by showing you every possible threshold at once, letting you pick the trade-off that matches your problem.

Intuition — The Security Camera Analogy. A motion-detection camera has a sensitivity dial. Turn it to 10: it catches every moving leaf (high TPR) but also triggers on shadows (high FPR). Turn it to 1: it triggers only on large objects (low FPR) but misses cats (low TPR). The ROC curve is the plot of "how many real events did you catch" vs "how many false alarms did you trigger" as you sweep the dial from 1 to 10. The best camera has a curve that rises quickly to the top-left — it catches real events without triggering on noise.

Purpose. The ROC curve visualizes classifier performance across ALL possible decision thresholds. It does not commit to one threshold — it shows the full spectrum of (FPR, TPR) pairs.

Inputs: A ranked list of test instances with predicted scores/probabilities and true labels.

Outputs: A curve in (FPR, TPR) space and a scalar AUC âˆˆ [0, 1].

Steps — Building the ROC Curve.

Given 10 instances sorted by descending score:

Rank	Score	True Class
1	0.95	Yes
2	0.85	Yes
3	0.85	Yes
4	0.85	No
5	0.80	Yes
6	0.80	No
7	0.60	No
8	0.43	Yes
9	0.43	No
10	0.25	Yes

Totals: 6 actual Yes, 4 actual No.

Sweep through thresholds from high to low:

Threshold	Predict Positive?	TP	FP	FN	TN	TPR	FPR
1.00 (above all)	None	0	0	6	4	0.00	0.00
0.90 (between 0.95 and 0.85)	Rank 1 only	1	0	5	4	0.17	0.00
0.83 (between 0.85 and 0.80)	Ranks 1-4	3	1	3	3	0.50	0.25
0.70 (between 0.80 and 0.60)	Ranks 1-6	4	2	2	2	0.67	0.50
0.50 (between 0.60 and 0.43)	Ranks 1-7	4	3	2	1	0.67	0.75
0.30 (between 0.43 and 0.25)	Ranks 1-9	5	4	1	0	0.83	1.00
0.00 (below all)	All	6	4	0	0	1.00	1.00

Plot these (FPR, TPR) points and connect them. The curve starts at (0,0), ends at (1,1), and bows toward the top-left. Sense-check: At threshold 0.90 (very strict), TPR=0.17, FPR=0.00 — the model catches few positives but makes zero false alarms. At threshold 0.30 (very loose), TPR=0.83, FPR=1.00 — it catches most positives but also flags every negative. The curve traces the full trade-off.

AUC — Area Under the Curve.

AUC	Meaning
1.00	Perfect separation — every positive ranks above every negative
0.90	Excellent — strong separation, some overlap
0.75	Fair — moderate separation
0.50	Random — the diagonal; model is guessing
<0.50< /strong>	Worse than random — flip the predictions and you get >0.50

Probabilistic interpretation: AUC = probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance. AUC = 0.8 means: "Pick a random Yes and a random No — the model ranks the Yes higher 80% of the time."

Critical Points.

Point	TPR	FPR	The model is...
(0,0)	0	0	Predicting EVERYTHING as negative
(1,1)	1	1	Predicting EVERYTHING as positive
(0,1)	1	0	Perfect — all positives caught, zero false alarms

The diagonal (0,0)â†’(1,1) = random guessing. Any usable model must bow above the diagonal.

Comparing Models with ROC. Plot both ROC curves on the same axes. The model whose curve is consistently closer to (0,1) is better. If curves cross, neither dominates — one may be better at low FPR (important for precision-critical applications), the other at high TPR (important for recall-critical applications). Choose based on your FPR tolerance.

Pitfalls.

ROC can be optimistic on highly imbalanced data. When negatives vastly outnumber positives, FPR = FP/(FP+TN) has a huge denominator (TN), so FPR stays tiny even with many FPs. The ROC curve looks great while the precision-recall curve would expose the problem. For severe imbalance (1:100 or worse), prefer the precision-recall curve.
AUC aggregates over thresholds you would never use. An AUC of 0.9 might come from great performance at FPR=0.5 (useless in practice) and terrible performance at FPR=0.01 (where you actually operate). Always inspect the curve shape, not just the number.
AUC < 0.5 means the model learned the pattern backwards. The model systematically says "yes" for negatives and "no" for positives. Flip its predictions and AUC becomes 1 âˆ’ original_AUC.
Scikit-learn: from sklearn.metrics import roc_curve, roc_auc_score — both are standard imports. The exam may ask about these.

Recap. ROC = TPR vs FPR across all thresholds. AUC = single-number summary; 0.5 = random, 1.0 = perfect. Use ROC for threshold selection and model comparison on balanced/moderately imbalanced data. Bridge: A model with great AUC can still fail if the classes are severely imbalanced. Section 11 tackles class imbalance head-on — why it breaks everything and how SMOTE fixes it.

Real-World & Domain Connection. ROC curves originated in WWII radar engineering — the "Receiver Operating Characteristic" measured how well a radar operator distinguished enemy aircraft (signal) from noise. The same framework now evaluates every medical diagnostic test. The AUC of a PSA test for prostate cancer is ~0.68 — far from perfect, which is why it is used alongside other tests. When the FDA evaluates a new diagnostic, the ROC curve and AUC are mandatory submissions.

11. Class Imbalance — Problems and Solutions

Hook. Your dataset has 9,970 legitimate transactions and 30 fraudulent ones. Without any special handling, your logistic regression model may learn exactly one rule: "always say legitimate." Accuracy: 99.7%. Fraud caught: 0. The math is not broken — it is working exactly as designed, minimizing the overall error by ignoring a class that barely affects the loss. This is the class imbalance problem.

Intuition — The Needle in a Haystack. Searching for fraud in normal transactions is like searching for a needle in a haystack. If you only care about the total weight of what you find, ignoring the needle costs you almost nothing — the hay dominates. A lazy classifier optimizes total weight (overall loss) and happily ignores the needle. The fix: either make the needle heavier (cost-sensitive learning), make more needles (SMOTE), or measure success by whether you found needles (recall/F1), not total hay weight (accuracy).

Solution 1 — Resampling. Before reaching for algorithms, check if your data collection is biased. Did you sample 10,000 normal transactions but only 30 fraud cases because fraud is genuinely rare, or because your collection window was too short? If possible, collect more minority-class data. This is the cleanest fix — but often impossible (you cannot manufacture more fraud cases).

Solution 2 — SMOTE (Synthetic Minority Oversampling Technique).

SMOTE creates new, synthetic minority examples by interpolating between existing ones:

Pick a minority example $x$ .
Find its $k$ nearest minority-class neighbors in feature space.
Randomly pick one neighbor $x_{nn}$ .
Create a synthetic point: $x_{n e w} = x + r \cdot (x_{nn} - x)$ where $r \sim Uniform (0, 1)$ .
Assign the minority class label to $x_{n e w}$ .

Why interpolation works: Two real cancer patients with blood marker values 150 and 180 probably bracket a plausible range of cancer-patient values. A synthetic patient at 165 is more realistic than a duplicate at 150. SMOTE fills in the convex hull of the minority class rather than just repeating known points.

Why not duplicate? Duplicating $x$ 100 times adds no new information — the model sees the same point 100 times. SMOTE creates genuinely new points the model has never seen.

Availability: from imblearn.over_sampling import SMOTE in Python.

Solution 3 — Change the Metric. Stop reporting accuracy. Report:

Recall when you must catch all positives (disease screening)
Precision when false alarms are unacceptable (spam deletion)
F1 for overall balance
Precision-Recall curve (not ROC) for severely imbalanced data, because ROC's FPR denominator (TN + FP) is dominated by TN, making FPR deceptively small

Solution 4 — Algorithm Choice. Logistic regression minimizes overall loss — it is naturally sensitive to imbalance. Alternatives:

Decision trees: Split on class purity, so minority class can still drive splits if it is "pure" in a region.
SVMs: Can use class weights to penalize minority-class errors more heavily.
Ensemble methods (Random Forest, XGBoost): Naturally handle imbalance through bagging/boosting and support class_weight='balanced'.

For logistic regression specifically: Use class_weight='balanced' in scikit-learn, which automatically weights errors inversely proportional to class frequencies.

Student Q&A (deduplicated).

Q: Can we just copy minority records to balance the data? A: No — duplicating identical records does not add new information. The model sees the same feature vector 100 times and learns nothing new. SMOTE interpolates between existing minority examples to create genuinely new synthetic records, giving the model a richer and more diverse minority-class distribution to learn from.

Pitfalls.

SMOTE creates unrealistic points if features are correlated. If height and weight are correlated, interpolating between (150cm, 80kg) and (180cm, 70kg) might give (165cm, 75kg) — plausible. But if you also have "number of pregnancies" as a feature, SMOTE might create a male patient with 1.3 pregnancies. Always check feature semantics before applying SMOTE.
SMOTE before train/test split. Always split first, then SMOTE only the training set. If you SMOTE the whole dataset, your test set will contain synthetic points, giving falsely optimistic results.
Undersampling throws away data. Randomly deleting majority-class examples loses information. Only use undersampling when you have massive data and the majority class is genuinely redundant.

Recap. Class imbalance = the minority class barely affects the loss, so the model ignores it. Fixes: SMOTE (synthesize minority examples), change the metric (F1/recall/precision), use balanced class weights, or switch algorithms. Bridge: Another factor that silently degrades logistic regression is mismatched feature scales — Section 12 explains why CGPA=5 and IQ=120 cannot be fed raw into gradient descent.

Real-World & Domain Connection. PayPal's fraud detection system processes millions of transactions hourly with a fraud rate below 0.1%. They use a combination of SMOTE-like techniques and cost-sensitive learning (weighting fraud errors 100Ã— more than legitimate errors). Pure accuracy would be >99.9% for a model that always says "legitimate" — and would catch zero fraud. Their metric is recall at a fixed precision threshold: catch as many fraud cases as possible while keeping the false alarm rate low enough that the manual review team is not overwhelmed.

12. Feature Scaling

Hook. You run gradient descent with CGPA (range 0–10) and IQ (range 50–150). The IQ weight changes 15Ã— faster than the CGPA weight — not because IQ is more important, but because IQ values are 15Ã— larger. The optimizer zigzags violently and may never converge. The fix is simple and mandatory: scale your features.

Intuition — The Uneven Staircase. Imagine descending a staircase where one step is 1 inch high and the next is 15 feet high. You would stumble. Gradient descent on unscaled features does exactly this: the cost surface is shaped like a long, narrow valley. The gradient points mostly across the valley (oscillating) rather than along it (descending). Scaling makes the valley roughly circular, so every step moves you efficiently toward the bottom.

Two Scaling Methods.

Standardization (Z-score): $x^{'} = \frac{x - μ}{σ}$ — centers at 0 with unit variance. Best default. Used in sklearn.preprocessing.StandardScaler.

Min-Max Normalization: $x^{'} = \frac{x - x _{m i n}}{x _{m a x} - x _{m i n}}$ — scales to [0, 1]. Use when you need non-negative inputs.

Why scaling matters for logistic regression specifically:

Gradient descent convergence: Unscaled features â†’ elongated cost contours â†’ slow, oscillating convergence. [The gradient descent reference (R2_AppE) confirms: gradient descent follows steepest descent; unscaled features make this path inefficient.]
Regularization fairness: L1/L2 penalties treat all weights equally. If IQ values are 15Ã— larger than CGPA, the IQ weight is penalized 15Ã— more — not because it should be, but because of the scale difference.
Interpretability: Scaled coefficients are comparable. After standardization, $θ_{1} = 0.5$ and $θ_{2} = - 0.3$ directly tell you CGPA has a larger standardized effect than IQ.

Worked Example — The Job-Offer Data.

Raw data: CGPA âˆˆ [2.0, 4.0], IQ âˆˆ [85, 110]. Without scaling, the IQ gradient is ~25Ã— larger (average IQ â‰ˆ 100 vs average CGPA â‰ˆ 3). The optimizer takes tiny steps for CGPA and giant leaps for IQ — oscillating across the narrow valley.

After standardization (subtract mean, divide by std):

CGPA: mean â‰ˆ 3.0, std â‰ˆ 0.7 â†’ CGPA' âˆˆ [âˆ’1.4, 1.4]
IQ: mean â‰ˆ 95, std â‰ˆ 10 â†’ IQ' âˆˆ [âˆ’1.0, 1.5]

Now both features have comparable range ~[âˆ’2, 2]. Gradients are comparable magnitude. The optimizer descends smoothly.

Sense-check: Scaling does not change the information in the data — only the units. The optimal decision boundary in original space is the same line; gradient descent just finds it faster.

Pitfalls.

Scaling after train/test split using test statistics. Fit the scaler on the training set only, then transform both train and test. Using test statistics leaks information.
Not scaling before regularization. L1/L2 penalize weight magnitudes. An unscaled feature with large values gets a small weight and is unfairly penalized less — the opposite of what you want.
Scaling binary features. Binary features (0/1) do not need scaling — they are already in a fixed range. Scaling them destroys their interpretability as indicators.

Recap. Unscaled features = slow, unstable gradient descent + unfair regularization. Standardize (zero mean, unit variance) before training. This applies to both linear and logistic regression. Bridge: With scaled features, a trained logistic regression model, and proper evaluation metrics, we are ready to handle more than two classes. Section 13 extends logistic regression to multi-class problems.

Real-World & Domain Connection. In production ML pipelines (e.g., Uber's demand prediction, Airbnb's pricing models), feature scaling is a mandatory step in the feature engineering pipeline — so universal that platforms like TensorFlow Extended (TFX) and AWS SageMaker automatically apply it. Skipping scaling is one of the top 3 causes of "my model trains but predicts nonsense" in industry.

13. Multi-Class Classification with Logistic Regression

Hook. Logistic regression gives you one probability: "Is this a cat?" But what if you need "Is this a cat, a dog, or a bird?" Logistic regression was not built for three answers — yet with a clever decomposition strategy, you can build a multi-class classifier out of nothing but binary logistic regressions. No new math required.

Intuition — The Tournament vs The Qualifier. Two ways to run a 3-player competition:

One-vs-All (OvR): Each player faces "everyone else" in a qualifier. The player with the best qualifying score wins. Three matches total.
One-vs-One (OvO): Every pair plays a head-to-head match. The player who wins the most matches wins the tournament. Three matches for 3 players, but $(2 K)$ for $K$ players.

Both work. OvR trains on all the data for each classifier (K models). OvO trains on only the relevant two classes per classifier ( $(2 K)$ models, but each on less data).

Strategy 1 — One-vs-All (OvR).

Train $K$ binary logistic regression models. For model $k$ :

Relabel: class $k$ â†’ 1, all other classes â†’ 0.
Train standard binary logistic regression on the full dataset.
Output: $P_{k} = P (class k ∣ x)$ .

Prediction: $y^= ar g max_{k} P_{k}$ .

Pros: Only $K$ models. Each trained on all data. Simple. Cons: Each model sees a heavily imbalanced dataset (1 class vs Kâˆ’1 others). The probabilities $P_{k}$ may not sum to 1 — they come from independent models.

Strategy 2 — One-vs-One (OvO).

Train $(2 K)$ binary models, one per class pair $(i, j)$ . Each model is trained only on examples from classes $i$ and $j$ .

Prediction: Each model votes for one class. The class with the most votes wins.

Pros: Each model trains on a balanced 2-class problem. Simpler decision boundaries. Cons: $(2 K)$ models — $K = 10$ needs 45 models, $K = 100$ needs 4,950. Impractical for many classes.

Strategy	# Models	Data per model	Imbalance issue	Best for
OvR	$K$	All data	Yes (1 vs Kâˆ’1)	Small K (<10)< /td>
OvO	$(2 K)$	Two classes only	No	Medium K, SVMs

Strategy 3 — Softmax (Multinomial Logistic Regression).

The native multi-class extension. One model outputs $K$ probabilities that sum to 1:

$P (y = k ∣ x) = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}, z_{k} = θ_{k}^{⊤} x$

The cost function is the multi-class cross-entropy: $J = - \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{nk} ln y_{nk}$ where $t_{nk}$ is 1 if example $n$ is class $k$ , 0 otherwise. [Verified against Bishop §4.3.4, Eq. 4.104–4.108.]

Key property: For $K = 2$ , softmax reduces exactly to the sigmoid. Proof: $P (y = 1) = e^{z_{1}} / (e^{z_{1}} + e^{z_{2}}) = 1/ (1 + e^{- (z_{1} - z_{2})}) = σ (z_{1} - z_{2})$ . The binary case just compares two scores.

Note: Softmax is not on the core syllabus but is the standard approach in practice. Scikit-learn's LogisticRegression(multi_class='multinomial') uses softmax.

Student Q&A (deduplicated).

Q: Do we need K models for K classes? A: For OvR, yes — K binary models. For OvO, $(2 K)$ models. For softmax, one model with K weight vectors. There is no way around creating multiple binary classifiers in OvR/OvO — each classifier is still standard logistic regression.

Q: What does softmax output? A: A probability distribution over all K classes — a vector like [0.10, 0.30, 0.60] that sums to 1. Binary logistic regression outputs a single number (probability of class 1); softmax outputs K numbers that sum to 1.

Q: If we replace sigmoid, is it still logistic regression? A: Replacing the sigmoid gives a different model. Softmax replaces sigmoid for multi-class. All belong to the Generalized Linear Model (GLM) family. What changes is the link function (identity â†’ linear regression, sigmoid â†’ logistic regression, softmax â†’ multinomial logistic regression). The gradient update rule keeps the same error Ã— input form.

Recap. Multi-class logistic regression = decompose into binary problems (OvR or OvO) or use softmax for a native K-class output. Each binary sub-model is standard logistic regression. Bridge: The GLM framework (Section 14) explains why all these variants share the same gradient descent update rule — they differ only in the link function.

Real-World & Domain Connection. Scikit-learn's LogisticRegression defaults to OvR for multi-class. Most modern deep learning uses softmax as the final layer — every image classifier (ResNet, EfficientNet) ends with a softmax over 1,000 ImageNet classes. The softmax + cross-entropy combination is the single most common output layer in all of deep learning.

14. Generalized Linear Models (GLMs)

Hook. Three models. Three different tasks. Three different output functions. Yet when you write the gradient descent update, the code is identical for all three. This is not a coincidence — it is the signature of a Generalized Linear Model. If you understand one GLM, you understand the skeleton of them all.

The GLM Unifying Pattern.

Every GLM has three components:

Linear predictor: $z = θ^{⊤} x$ . Same weighted sum for every GLM. This is the "linear" in Generalized Linear Model.

Link function $g^{- 1} (z)$ : Maps the linear predictor to the output space:

Linear regression: $g^{- 1} (z) = z$ (identity) — output is any real number.
Logistic regression: $g^{- 1} (z) = σ (z) = \frac{1}{1 + e ^{- z}}$ — output is a probability in (0,1).
Softmax regression: $g^{- 1} (z_{k}) = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$ — output is a probability distribution.
Poisson regression: $g^{- 1} (z) = e^{z}$ — output is a non-negative count.

Error structure — the universal gradient form:

$θ_{j} := θ_{j} - α \cdot (prediction - actual) \cdot x_{j}$

This form holds for ALL GLMs when paired with their natural cost function (MSE for linear, cross-entropy for logistic/softmax). [Bishop §4.3.6 notes: the same simple gradient form arises for the sum-of-squares error with linear models, cross-entropy with logistic sigmoid, and cross-entropy with softmax — it is a general result.]

Why this works: The link function and cost function are canonically paired — the link function's derivative cancels with the cost function's derivative structure, leaving only (prediction âˆ’ actual) Ã— input. This is why we use sigmoid+cross-entropy, not sigmoid+MSE (which breaks the cancellation and gives messier gradients).

The GLM Family Tree.

Model	Link Function	Output Domain	Cost Function	Use Case
Linear Regression	Identity	$(- \infty, \infty)$	MSE	Predict continuous values
Logistic Regression	Sigmoid	$(0, 1)$	Cross-entropy	Binary classification
Softmax Regression	Softmax	Probability simplex	Multi-class cross-entropy	Multi-class classification
Poisson Regression	Exponential	$[0, \infty)$	Poisson log-likelihood	Count data

All share: linear predictor $z = θ^{⊤} x$ and the same gradient descent update skeleton.

Recap. GLMs unify linear regression, logistic regression, and softmax regression under one framework: linear predictor + link function + canonical cost â†’ (prediction âˆ’ actual) Ã— input gradient. Learn one GLM, and you have the template for all of them. Bridge: This concludes the conceptual part of the lecture. Section 15 provides the exam-style problems you must practice.

Real-World & Domain Connection. The GLM framework (Nelder & Wedderburn, 1972) is the statistical foundation of most of classical machine learning. In insurance actuarial science, GLMs are the legally mandated method for pricing auto and home insurance in many jurisdictions. The Poisson GLM models claim frequency; the Gamma GLM models claim severity. The interpretability and statistical guarantees of GLMs make them irreplaceable in regulated industries — even as deep learning advances elsewhere.

15. Problem-Solving Exercises

Exam note: These problems are from previous exam papers. They are not optional. Similar problems appear on the exam. Expect ~2 questions covering linear regression AND logistic regression. The exam is close to book — you must know how to solve gradient descent update steps, compute probabilities from logistic regression, interpret confusion matrices, and calculate evaluation metrics.

Problem 1 — Gradient Descent for Logistic Regression (2 Iterations).

Given:

Dataset with features $x^{(i)}$ and binary targets $y^{(i)}$
Initial weights: $θ_{0}, θ_{1}, θ_{2}$
Learning rate $α$

Required procedure for each iteration:

Forward pass: For each example $i$ , compute $z^{(i)} = θ_{0} + θ_{1} x_{1}^{(i)} + θ_{2} x_{2}^{(i)}$ then $h_{θ}^{(i)} = 1/ (1 + e^{- z^{(i)}})$ .
Error: $e^{(i)} = h_{θ}^{(i)} - y^{(i)}$ .
Gradient: $\frac{\partial J}{\partial θ _{j}} = \frac{1}{m} \sum_{i = 1}^{m} e^{(i)} x_{j}^{(i)}$ (with $x_{0}^{(i)} = 1$ ).
Update: $θ_{j} := θ_{j} - α \cdot \frac{\partial J}{\partial θ _{j}}$ .
Recompute $h_{θ}$ with the new weights before the next iteration.

Critical exam tip: After every iteration, recompute $h_{θ}$ with the new $θ$ . Never reuse old predictions. This is the #1 mistake students make.

What to observe: The weights should move in a direction that reduces the overall error. If the model over-predicts (predicting 1 when actual is 0), the positive error pushes weights down. If it under-predicts, negative error pushes weights up.

Problem 2 — Feature Scaling Explanation.

Given: CGPA âˆˆ [0, 10], IQ âˆˆ [50, 150].

Answer outline:

These features have vastly different ranges (~10 vs ~100).
Without scaling, the cost function $J (θ)$ has elongated elliptical contours.
Gradient descent oscillates across the narrow valley, taking many iterations to converge.
The IQ weight receives ~15Ã— larger gradient updates, dominating the optimization.
Solution: Apply standardization — for each feature, subtract its mean and divide by its standard deviation. Both features then have mean 0, variance 1.
Fit the scaler on training data only, then transform both train and test.

Diagram to draw in your answer: A contour plot with elongated ellipses (unscaled) vs circular contours (scaled), with gradient descent paths overlaid. The unscaled path zigzags; the scaled path goes straight to the minimum.

Exam note: Key formulas you must have memorized.

Formula	When to use
$σ (z) = \frac{1}{1 + e ^{- z}}$	Converting logit to probability
$J (θ) = - \frac{1}{m} \sum [y ln h_{θ} + (1 - y) ln (1 - h_{θ})]$	Cost function definition
$θ_{j} := θ_{j} - \frac{α}{m} \sum (h_{θ} - y) x_{j}$	Gradient descent update
$Odds = \frac{P}{1 - P}$	Converting probability to odds
$e^{θ_{j}}$ = odds ratio for feature $j$	Interpreting feature effects
Accuracy = $\frac{T P + T N}{total}$	Basic evaluation
Precision = $\frac{T P}{T P + F P}$ , Recall = $\frac{T P}{T P + F N}$	Imbalanced data evaluation
F1 = $2 \frac{P \cdot R}{P + R}$	Balanced metric

Common exam traps to avoid:

High accuracy on imbalanced data â†’ meaningless. Always check class distribution.
99% training accuracy â†’ overfitting, not success.
Scikit-learn $C = 1/ λ$ â†’ high C = weak regularization (inverse!).
Sigmoid output is a probability, not a class label — apply threshold (default 0.5) to classify.
Recompute $h_{θ}$ after every gradient descent iteration.

Exam Guidance Summary

Exam note: What to expect.

Topics covered in Lecture 7: Logistic regression (hypothesis, cost function, update rule). Sentiment analysis, odds and log-odds, regularization (L1/L2), confusion matrix. Accuracy, precision, recall, F1, ROC curve and AUC. Class imbalance and SMOTE. Multi-class strategies (OvR, OvO, softmax) and GLMs.

Expected questions: ~2 questions covering linear regression AND logistic regression.

Question types:

Mathematical: Compute gradient descent steps, calculate probabilities from sigmoid, derive evaluation metrics from confusion matrices.
Conceptual: Interpret weights and odds ratios, compare classifiers for given scenarios, choose the right evaluation metric for a context.

Problem sources: The end-of-slide problems (Section 15) are from previous exam papers. Solve them all.

Exam note: Key formulas you must memorize.

Formula	Usage
$σ (z) = \frac{1}{1 + e ^{- z}}$	Logit â†’ probability
$J (θ) = - \frac{1}{m} \sum [y ln h_{θ} + (1 - y) ln (1 - h_{θ})]$	Cost function
$θ_{j} := θ_{j} - \frac{α}{m} \sum (h_{θ} - y) x_{j}$	GD update rule
Odds = $\frac{P}{1 - P}$ , $e^{θ_{j}}$ = odds ratio	Weight interpretation
Acc = $\frac{T P + T N}{total}$ , Prec = $\frac{T P}{T P + F P}$ , Rec = $\frac{T P}{T P + F N}$	Evaluation
F1 = $2 \frac{P \cdot R}{P + R}$	Balanced metric

Exam traps — the professor's explicit warnings.

High accuracy on imbalanced data is meaningless. Always check class distribution before reporting accuracy. The exam will test this with a confusion matrix on imbalanced data.
99% training accuracy = overfitting, not success. A red flag, not a celebration.
Scikit-learn $C = 1/ λ$ . High C = weak regularization (inverse relationship). This is tested every year.
Sigmoid output is a probability, not a class label. Apply threshold (default 0.5) to make the classification decision.
Recompute $h_{θ}$ after every gradient descent iteration. Never reuse old predictions — this is the most common homework/exam mistake.

Exam note: Study strategy.

Practical tip: When solving gradient descent problems, recompute $h_{θ} (x)$ after each iteration with the new weights. The loop is: forward â†’ error â†’ gradient â†’ update â†’ repeat.
Study resources: All materials (slides, problem-solving sessions) are uploaded under the course files. Refer to the problem-solving classes for worked gradient descent examples.
Exam is close to book: The formulas, notation, and problem formats are directly from the lecture slides. Do not rely on external resources that use different notation.

Key Industry Applications

Where logistic regression is used in production.

Application	How Logistic Regression is Used	Key Metric
Credit scoring	Predict default probability from income, debt ratio, payment history	AUC, interpretability
Medical diagnosis	Predict disease presence from symptoms, lab results, imaging features	Recall (don't miss disease)
Fraud detection	Flag suspicious transactions; SMOTE handles severe imbalance	Recall at fixed precision
Spam filtering	Classify emails as spam/ham; false positives are costly	Precision
Sentiment analysis	Classify reviews/tweets as positive/negative from word-count features	F1 or accuracy (balanced)
Customer churn	Predict whether a customer will leave based on usage patterns	AUC, lift curve
Intrusion detection	Detect network attacks from traffic patterns	Recall
Biometric access	Retinal/fingerprint scan authorization for high-security facilities	Precision (no false accepts)

Tools and libraries.

Scikit-learn (Python): LogisticRegression(penalty='l2', C=1.0, class_weight='balanced') — the standard implementation. Provides roc_curve, roc_auc_score, confusion_matrix, classification_report.
Imbalanced-learn: SMOTE, ADASYN, RandomUnderSampler — for handling class imbalance.
Statsmodels (Python): Logit and MNLogit — for statistical inference with p-values and confidence intervals (used in medical/economic research).
Decision trees (next lecture): An alternative classifier that handles imbalanced data and nonlinear boundaries better than logistic regression — but lacks the same interpretability of odds ratios.

When to use logistic regression vs alternatives:

Use logistic regression when you need interpretable coefficients (odds ratios, p-values) — e.g., medical research, credit decisions, regulatory submissions.
Use tree-based models (Random Forest, XGBoost) when you need raw predictive power on tabular data with nonlinear relationships.
Use neural networks when you have unstructured data (images, text, audio) and interpretability is not required.

ML Lecture 7 notes · Logistic Regression — Complete Enriched Lecture Notes

Machine Learning· postgraduate· 2026-06-28

Summary

A comprehensive lecture on logistic regression for binary classification, covering the full pipeline from mathematical foundations (sigmoid function, cross-entropy cost, gradient descent derivation) through practical applications (sentiment analysis with feature engineering) to advanced topics (regularization, confusion matrices, precision-recall-F1 metrics, ROC curves and AUC, class imbalance with SMOTE, multi-class strategies, and the Generalized Linear Model framework). Each concept includes intuitive analogies, fully worked numerical examples, and real-world domain connections spanning credit scoring, medical diagnosis, and fraud detection.

Learning Objectives

1Objective 1: Derive and implement logistic regression — explain the hypothesis function h_θ(x) = σ(θ^T x), the cross-entropy cost function, and the gradient descent update rule identical in form to linear regression.

2Objective 2: Apply logistic regression to text classification — design feature engineering strategies (bag-of-words, word counts, negation markers) and compute a full prediction from trained weights.

3Objective 3: Interpret logistic regression weights through odds — convert between probability and odds, compute odds ratios e^θ, and explain why odds ratios are multiplicative not additive.

4Objective 4: Evaluate binary classifiers comprehensively — build confusion matrices, compute accuracy/precision/recall/F1, construct ROC curves, interpret AUC, and select appropriate metrics for imbalanced data.

5Objective 5: Diagnose and fix common logistic regression problems — apply L1/L2 regularization, use SMOTE for class imbalance, scale features before training, and choose multi-class decomposition strategies (OvR/OvO/softmax).

Sections Breakdown

1Logistic Regression — Core Concepts

Building the logistic regression model: linear score, sigmoid function, log-loss cost function, gradient derivation, and comparison with linear regression.

2Sentiment Analysis Application

Feature engineering for text classification: converting words to numerical features, full prediction walkthrough.

3Gradient Descent for Logistic Regression

Batch gradient descent iteration traced on a job-offer dataset, explaining forward pass, error computation, and weight updates.

4Model Interpretation and Odds

Logit-odds connection, odds ratios for interpreting weights, worked examples of probability and odds conversion.

5Regularization in Logistic Regression

L1 (Lasso) and L2 (Ridge) regularization, the C parameter trap in scikit-learn, preventing overfitting.

6The Confusion Matrix

2x2 confusion matrix structure, true/false positives and negatives, asymmetric cost of errors.

7Accuracy and Its Pitfalls

The accuracy paradox on imbalanced data, when accuracy is acceptable vs misleading.

8Precision, Recall, and F1 Score

Definitions, the harmonic mean penalty, precision-recall trade-off, F-beta extension.

9Comparing Classifiers — Three Use Cases

Cautious vs aggressive vs ideal classifiers mapped to real-world deployment contexts.

10ROC Curve and AUC

Building ROC curves by threshold sweep, AUC probabilistic interpretation, model comparison.

11Class Imbalance — Problems and Solutions

SMOTE algorithm for synthetic minority oversampling, metric selection, class weighting.

12Feature Scaling

Standardization vs min-max normalization, why scaling matters for gradient descent and regularization.

13Multi-Class Classification

One-vs-All (OvR), One-vs-One (OvO), Softmax regression strategies for K-class problems.

14Generalized Linear Models (GLMs)

Unifying framework: linear predictor + link function + canonical cost, the GLM family tree.

15Problem-Solving Exercises

Exam-style problems on gradient descent computation, feature scaling explanation, key formulas.

16Exam Guidance Summary

Expected question types, key formulas to memorize, common exam traps.

17Key Industry Applications

Production use cases of logistic regression across credit scoring, medical diagnosis, fraud detection, and more.

Postgraduate students in Machine Learning (S1-25_AIMLCZG565)

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

Logistic Regression Hypothesis

Must-know: The hypothesis is h_θ(x) = σ(θ^T x) = 1/(1+e^{-θ^T x}), the sigmoid of a linear score. This outputs a probability in (0,1), not a raw score. Always apply the sigmoid — never use the raw logit z as the prediction.

$h_{θ} (x) = σ (θ^{⊤} x) = \frac{1}{1 + e ^{- θ^{⊤} x}}$

⚠️ Top pitfall: Forgetting to apply the sigmoid and treating the raw logit z = θ^T x as the prediction. The logit can be any real number; only the sigmoid converts it to a valid probability.

Self-check: If θ = [0.5, -0.3] and x = [2, 1], what is the predicted probability? First compute z, then apply σ(z).

Connects to: Linear Regression, Gradient Descent, GLMs

Cross-Entropy Cost Function

Must-know: Cross-entropy (log-loss) is convex for logistic regression, unlike MSE which creates a non-convex surface. It derives from maximum likelihood estimation and pairs naturally with the sigmoid.

$J (θ) = - \frac{1}{m} i = 1 \sum m [y^{(i)} ln h_{θ} (x^{(i)}) + (1 - y^{(i)}) ln (1 - h_{θ} (x^{(i)}))]$

⚠️ Top pitfall: Using MSE with sigmoid output. The cost surface becomes non-convex with local minima, breaking gradient descent guarantees.

Self-check: Why does the σ(1-σ) term cancel when taking the derivative of cross-entropy with sigmoid?

Connects to: Maximum Likelihood Estimation, Cost Functions

Gradient Descent Update Rule

Must-know: The gradient ∂J/∂θ_j = (1/m) Σ (h_θ - y) x_j is identical in form to linear regression. The only difference is h_θ: linear regression uses θ^T x, logistic uses σ(θ^T x).

$θ_{j} := θ_{j} - α \cdot \frac{1}{m} i = 1 \sum m (h_{θ} (x^{(i)}) - y^{(i)}) x_{j}^{(i)}$

⚠️ Top pitfall: Not recomputing h_θ after each weight update. Every iteration starts with a fresh forward pass using the new weights.

Self-check: After one gradient descent iteration, θ_0 went from 0.5 to 0.35 and θ_2 from 0.5 to -12.65. Why did θ_2 change so much more?

Connects to: Feature Scaling, Learning Rate, Convergence

Odds and Log-Odds Interpretation

Must-know: The logit z = θ^T x equals the log-odds ln(P/(1-P)). Increasing x_j by 1 multiplies the odds by e^{θ_j} (the odds ratio). Negative θ_j means e^{θ} < 1, so odds decrease.

$ln (\frac{P}{1 - P}) = θ_{0} + θ_{1} x_{1} + \dots + θ_{n} x_{n} and Odds = e^{θ_{0}} \cdot e^{θ_{1} x_{1}} \cdot \dots \cdot e^{θ_{n} x_{n}}$

⚠️ Top pitfall: Interpreting θ_j as additive effect on probability. The effect is additive on log-odds and multiplicative on odds — the effect on probability depends on the current position on the sigmoid curve.

Self-check: If θ_1 = 0.3, what does e^{0.3} ≈ 1.35 mean in plain language?

Connects to: Feature Effects, Model Interpretation

L1 and L2 Regularization

Must-know: L2 (Ridge) shrinks all weights toward zero; L1 (Lasso) drives some to exactly zero for feature selection. In scikit-learn, C = 1/λ — high C = weak regularization.

$J_{reg} (θ) = J (θ) + λ j = 1 \sum n ∣ θ_{j} ∣ (L1) or J (θ) + \frac{λ}{2} j = 1 \sum n θ_{j}^{2} (L2)$

⚠️ Top pitfall: The C-inverse trap: setting C=100 thinking it gives strong regularization. High C = small λ = weak regularization.

Self-check: When data is linearly separable, what happens to unregularized logistic regression weights, and why?

Connects to: Overfitting, Feature Selection, Bias-Variance Trade-off

Confusion Matrix and Evaluation Metrics

Must-know: Accuracy = (TP+TN)/total, but fails on imbalanced data. Precision = TP/(TP+FP), Recall = TP/(TP+FN). F1 = 2PR/(P+R) is the harmonic mean, penalizing imbalance between precision and recall.

$Precision = \frac{T P}{T P + F P}, Recall = \frac{T P}{T P + F N}, F 1 = 2 \cdot \frac{P \cdot R}{P + R}$

⚠️ Top pitfall: Reporting accuracy alone on imbalanced data. A 99% accurate fraud detector that catches zero fraud is useless.

Self-check: In cancer screening, which error is costlier: FP or FN? Which metric should you optimize?

Connects to: Class Imbalance, ROC Curve

ROC Curve and AUC

Must-know: ROC plots TPR vs FPR across all thresholds. AUC = probability a random positive ranks above a random negative. AUC=0.5 is random; AUC=1.0 is perfect. For high imbalance, prefer precision-recall curves.

$TPR = \frac{T P}{T P + F N}, FPR = \frac{F P}{F P + T N}, AUC = P (rank(positive) > rank(negative))$

⚠️ Top pitfall: ROC can be optimistic on imbalanced data because FPR uses a large TN denominator. Always inspect the curve shape, not just the AUC number.

Self-check: What does the point (0,1) on an ROC curve represent, and is it achievable?

Connects to: Threshold Selection, Model Comparison

Class Imbalance and SMOTE

Must-know: SMOTE creates synthetic minority examples by interpolating between existing minority points: x_new = x + r·(x_nn - x) with r ~ Uniform(0,1). Always split train/test before applying SMOTE.

$x_{new} = x + r \cdot (x_{nn} - x), r \sim Uniform (0, 1)$

⚠️ Top pitfall: Applying SMOTE before train/test split — contaminates the test set with synthetic data, giving falsely optimistic evaluation.

Self-check: Why does simple duplication of minority examples fail where SMOTE succeeds?

Connects to: Imbalanced Data, Resampling, Cost-Sensitive Learning

Multi-Class Strategies (OvR, OvO, Softmax)

Must-know: OvR trains K binary models (one per class vs rest). OvO trains C(K,2) pairwise models. Softmax extends sigmoid to K classes with P(y=k|x) = e^{z_k}/Σe^{z_j}. Softmax reduces to sigmoid when K=2.

$P (y = k ∣ x) = \frac{e ^{θ_{k}^{⊤} x}}{\sum _{j = 1}^{K} e ^{θ_{j}^{⊤} x}} (Softmax)$

⚠️ Top pitfall: Assuming OvR probabilities sum to 1. They don't — each model is trained independently. Only softmax produces a proper probability distribution.

Self-check: For K=3 classes, how many binary models does OvR train? How many does OvO train?

Connects to: Binary Classification, Generalized Linear Models

Generalized Linear Models (GLMs)

Must-know: All GLMs share three parts: linear predictor z = θ^T x, a link function, and the universal gradient form θ_j := θ_j - α·(prediction - actual)·x_j paired with the canonical cost.

$θ_{j} := θ_{j} - α \cdot (prediction - actual) \cdot x_{j} (universal GLM gradient)$

⚠️ Top pitfall: Breaking the canonical pairing — using sigmoid with MSE or identity link with cross-entropy destroys the clean gradient cancellation.

Self-check: Name three GLMs and their link functions. What output domain does each map to?

Connects to: Linear Regression, Logistic Regression, Softmax Regression, Poisson Regression

Practice Quiz

Test your understanding of ML Lecture 7 notes. Select an answer for each question — results are instant.

What is the range of the sigmoid function σ(z) = 1/(1+e^{-z})?

Why is cross-entropy (log-loss) used instead of MSE for logistic regression?

In scikit-learn's LogisticRegression, if C=100, what does this mean for regularization?

A model has TP=90, FP=10, FN=30, TN=70. What is its F1 score?

What does an AUC of 0.80 mean in plain language?

Previous LectureLecture 06 · Regularization — Worked Numerical Examples

Next LectureLecture 08 · Decision Trees

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Logistic Regression — Complete Enriched Lecture Notes

1. Logistic Regression — Core Concepts

Symbol Registry (Reference)

Comparison: Linear vs Logistic Regression

2. Sentiment Analysis Application

3. Gradient Descent in Logistic Regression — Worked Iteration

4. Model Interpretation and Odds

5. Regularization in Logistic Regression

6. Evaluating Classifiers — The Confusion Matrix

7. Accuracy and Its Pitfalls

8. Precision, Recall, and F1 Score

9. Comparing Classifiers — Three Use Cases

10. ROC Curve and AUC

11. Class Imbalance — Problems and Solutions

12. Feature Scaling

13. Multi-Class Classification with Logistic Regression

14. Generalized Linear Models (GLMs)

15. Problem-Solving Exercises

Exam Guidance Summary

Key Industry Applications

ML Lecture 7 notes · Logistic Regression — Complete Enriched Lecture Notes

Summary

Learning Objectives

Sections Breakdown

Exam Revision Notes

Logistic Regression Hypothesis

Cross-Entropy Cost Function

Gradient Descent Update Rule

Odds and Log-Odds Interpretation

L1 and L2 Regularization

Confusion Matrix and Evaluation Metrics

ROC Curve and AUC

Class Imbalance and SMOTE

Multi-Class Strategies (OvR, OvO, Softmax)

Generalized Linear Models (GLMs)

Practice Quiz

0 / 5

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat