Purpose. Gradient descent finds the parameter values $\theta =({\theta }_0,{\theta }_1,\dots ,{\theta }_d)$ that minimize the cost function $J(\theta )$. The total prediction error across all training examples. It is an iterative numerical method. Unlike the closed-form normal equations ($\theta =({\Phi }^T\Phi {)}^{-1}{\Phi }^{T}t$), which solve the problem exactly in one shot, gradient descent steps toward the solution gradually. Why use an iterative method when a closed-form exists? Because the normal equations require inverting a $d\times d$ matrix, which costs $O(d^3)$ — impractical when $d$ is large (thousands of features). Gradient descent scales to high dimensions.

Inputs & Outputs.

• Input: Training data $(X,y)$ — $N$ examples, each with $d$ features, plus a bias column $x_0=1$. Hyperparameters: learning rate $\alpha >0$, number of iterations $T$ (or a stopping condition).
• Output: A parameter vector $\theta \in {\mathbb{R}}^{d+1}$ that (approximately) minimizes the cost function. The trained model is $h_{\theta }(x)={\theta }^{\top }x$.

Steps — Batch Gradient Descent Algorithm.

1. Initialize $\theta$ to a small random vector (or zeros). Set iteration counter $t=0$.
2. Repeat until convergence or $t=T$:

• (a) Compute predictions for ALL $N$ examples: $h_{\theta }(x^{(i)})={\theta }^{\top }{x}^{(i)}$ for $i=1,\dots ,N$.
• (b) Compute the gradient of the cost w.r.t. each parameter:

$$\frac{\partial J}{\partial {\theta }_j}=\frac{1}{N}\sum _{i=1}^N\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}$$

• (c) Update all parameters simultaneously:

$${\theta }_j:={\theta }_j-\alpha \cdot \frac{\partial J}{\partial {\theta }_j}$$

• (d) $t:=t+1$.

1. Return $\theta$.

Why simultaneous update matters. Compute ALL the temporary gradients first (using the old $\theta$ values), then update ALL parameters at once. If you update ${\theta }_0$ first and then use the new ${\theta }_0$ to compute the gradient for ${\theta }_1$, you are not following the true gradient. You are following a corrupted direction. Use temporary variables:

$${\text{temp}}_j=\frac{1}{N}\sum _{i=1}^N(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}\text{for all }j$$

Then apply: ${\theta }_j:={\theta }_j-\alpha \cdot {\text{temp}}_j$ for all $j$.

Two Forms of the Hypothesis.

Scalar form — for a single training example. Given one feature vector $x=(1,x_1,x_2,\dots ,x_d{)}^{\top }$ (with $x_0=1$ for the bias):

$$h_{\theta }(x)=\sum _{j=0}^d{\theta }_j{x}_j={\theta }^{\top }x$$

Here ${\theta }^{\top }x$ is a dot product between two vectors — the result is a single scalar (one predicted value).

Matrix/vectorized form — for ALL training examples. Given the full design matrix $X\in {\mathbb{R}}^{N\times (d+1)}$ (one row per example, one column per feature, first column all ones):

$$h_{\theta }(X)=X\theta$$

Here $X\theta$ is a matrix-vector product producing a column vector of predictions — one entry per training example. The result is in ${\mathbb{R}}^N$.

Key distinction: $h_{\theta }(x)$ is a scalar (one number). $h_{\theta }(X)$ is a column vector (N numbers). The professor emphasized. : "${\theta }^{\top }x$ is for one particular training example. This $x$ is for one record. Capital $X\theta$ is for all the examples together."

Three Variants — Formal Definition.

Consider a training set of $N$ examples. All three variants use the same update rule:

$${\theta }_j:={\theta }_j-\alpha \cdot \frac{\partial J}{\partial {\theta }_j}$$

The difference is which examples are used to compute the gradient:

1. Batch Gradient Descent. Uses ALL $N$ examples.

$$\frac{\partial J}{\partial {\theta }_j}=\frac{1}{N}\sum _{i=1}^N\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}$$

One update per full pass through the dataset. Stable, deterministic, but slow for large $N$.

1. Stochastic Gradient Descent (SGD). Uses ONE randomly chosen example $i_k$ per step.

$$\frac{\partial J}{\partial {\theta }_j}\approx \left(h_{\theta }(x^{(i_k)})-y^{(i_k)}\right)\cdot x_j^{(i_k)}$$

$N$ updates per full pass (epoch). Very fast, noisy. The parameter path zigzags toward the minimum. The noise acts as a natural regularizer, sometimes helping generalization.

1. Mini-Batch Gradient Descent. Uses a random subset of $B$ examples ($1<B<N$).

$$\frac{\partial J}{\partial {\theta }_j}\approx \frac{1}{B}\sum _{i\in \text{batch}}\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}$$

$N/B$ updates per epoch. Balances speed and stability. Typical batch sizes: 32, 64, 128, 256. This is the default in modern deep learning frameworks.

Metric Definitions.

Let ${\hat{y}}_i=h_{\theta }(x^{(i)})$ be the predicted value and $y_i$ be the actual value for $i=1,\dots ,N$.

Mean Absolute Error (MAE):

$$\text{MAE}=\frac{1}{N}\sum _{i=1}^N\left|{\hat{y}}_i-y_i\right|$$

• Each error contributes proportionally to its size. A 10-unit error is 10 times worse than a 1-unit error.
• Same units as the target variable.
• Robust to outliers (no squaring).

Mean Squared Error (MSE):

$$\text{MSE}=\frac{1}{N}\sum _{i=1}^N{\left({\hat{y}}_i-y_i\right)}^2$$

• Large errors are penalized quadratically. A 10-unit error contributes 100 to the sum; a 1-unit error contributes 1.
• Units are the square of the target units (e.g., dollars-squared).
• This is the cost function that gradient descent minimizes.

Root Mean Squared Error (RMSE):

$$\text{RMSE}=\sqrt{\text{MSE}}=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left({\hat{y}}_i-y_i\right)}^2}$$

• Same units as the target variable. Easier to interpret than MSE.
• Still sensitive to large errors (the square happens before the root).

R-squared ($R^2$) — Coefficient of Determination:

$$R^2=1-\frac{{\sum }_{i=1}^N(y_i-{\hat{y}}_i{)}^2}{{\sum }_{i=1}^N(y_i-\bar{y}{)}^2}=1-\frac{\text{MSE}}{\text{Variance of }y}$$

• $\bar{y}=\frac{1}{N}\sum y_i$ is the mean of the target values.
• $R^2=1$: perfect fit (predictions match all points exactly).
• $R^2=0$: model is no better than predicting the mean every time.
• $R^2<0$: model is worse than just predicting the mean. (Yes, this can happen with bad models.)
• Dimensionless — always between $-\infty$ and 1 (for linear regression with intercept, between 0 and 1).

Definition. A model is linear when it is linear with respect to the parameters $\theta$, not when it is linear with respect to the input features $x$. The professor emphasized this distinction repeatedly. : "Parameter and feature are different."

General form of a linear model with basis functions:

$$y=\sum _{j=0}^d{\theta }_j{\varphi }_j(x)={\theta }^{\top }\varphi (x)$$

where ${\varphi }_j(x)$ is the $j$-th basis function applied to input $x$, and ${\varphi }_0(x)=1$ (the bias term).

Why this matters: The model is still solved by the same linear regression machinery. The normal equations or gradient descent — regardless of how complex the basis functions are. The cost function remains convex in $\theta$. You get nonlinear predictive power with linear computational cost.

Common basis functions (from Bishop §3.1):

• Polynomial: ${\varphi }_j(x)=x^j$ — standard polynomial regression. $\varphi (x)=(1,x,x^2,\dots ,x^d)$. Fits curves of degree $d$.
• Gaussian (RBF): ${\varphi }_j(x)=\exp \left(-\frac{(x-{\mu }_j{)}^2}{2{\sigma }^2}\right)$ — each basis function is a bell curve centered at ${\mu }_j$. Good for local patterns; the model learns which centers matter.
• Sigmoidal: ${\varphi }_j(x)=\frac{1}{1+e^{-(x-{\mu }_j)/s}}$ — S-shaped functions. Capture threshold effects and saturating relationships.

Linear vs. nonlinear in $x$:

$$\begin{array}{rlrl}y& ={\theta }_0+{\theta }_{1}x& & \text{Linear in both }\theta \text{ and }x\text{(simple line)}\\ y& ={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2& & \text{Linear in }\theta ,\text{ nonlinear in }x\text{(quadratic curve)}\\ y& ={\theta }_0+{\theta }_1{e}^x& & \text{Linear in }\theta ,\text{ nonlinear in }x\text{(exponential growth)}\\ y& ={\theta }_0{e}^{{\theta }_{1}x}& & \text{NOT linear in }\theta \text{(cannot use linear regression directly)}\end{array}$$

Formal Definitions.

Bias. Error from overly simple model assumptions. A high-bias model systematically misses the true pattern (like always fitting a straight line to a curve). It underfits.

Variance. Error from the model being too sensitive to small fluctuations in the training data. A high-variance model treats noise as signal (like fitting every random wiggle). It overfits.

The tradeoff: You cannot simultaneously minimize both. Simpler models have high bias but low variance. Complex models have low bias but high variance. The goal is the sweet spot in the middle.

Visual Intuition — The Bias-Variance Graph:

• X-axis: Model complexity (e.g., polynomial degree).
• Y-axis: Error.
• Training error curve: Starts high on the left (simple model cannot fit training data), decreases monotonically as complexity increases, asymptotically approaches zero.
• Test error curve: U-shaped. High on the left (underfitting), drops to a minimum at optimal complexity, then rises again on the right (overfitting).
• The gap between training and test error widens as overfitting sets in — this gap is the variance.
• Landmark: The optimal model complexity is at the bottom of the test-error U-curve.
• Takeaway: The best model is NOT the one with the lowest training error. It is the one with the lowest test error.

Strategy 1 — Increase Training Data Size.

• With 15 data points, a high-degree polynomial can wiggle through every point and memorize noise.
• With 100+ data points, the same polynomial is forced to approximate the underlying trend because no single curve can exactly pass through 100 noisy points.
• RMSE behavior:
• Simple model (low order): RMSE plateaus at a higher error level as training size grows — it cannot capture the pattern even with infinite data (high bias).
• Complex model (high order): RMSE plateaus at a lower error level — IF the dataset is large enough. More data compensates for high variance.
• Key insight: When using a complex model, you MUST have a large training set. Model complexity and data volume must grow together.

Strategy 2 — Reduce Model Complexity.

• If the true function is quadratic ($y=a{x}^2+bx+c$):
• Order 1 (line): underfits — cannot capture the curve.
• Order 2 (quadratic): correct fit.
• Order 300: massively overfits — 300 degrees of freedom for a degree-2 problem.
• The right choice matches the true complexity of the data-generating process.
• In practice, you try multiple complexities and use validation error to pick the best one (this is called model selection).

Strategy 3 — Early Stopping (for Iterative Models Only).

• Applies to: Models trained iteratively — gradient descent, neural networks, logistic regression with gradient descent.
• Does NOT apply to: Models with closed-form solutions — linear regression (normal equations), KNN, Naive Bayes. These do not "iterate" through data; they compute parameters in one shot.

How it works:

• Plot training error and validation error against epoch number (one epoch = one full pass through the dataset).
• Training error: decreases monotonically (model keeps fitting better).
• Validation error: U-shaped. Decreases initially (model learning patterns), reaches a minimum at the best model point, then increases (model memorizing noise).
• Stop training at the validation error minimum. This is the moment when the model has learned the signal but has not yet started fitting the noise.

Implementation options:

1. Fixed number of iterations (epochs) — set after observing the validation curve.
2. Threshold on error metric — e.g., stop when validation RMSE < 0.5.
3. Patience-based — stop when validation error has not improved for $k$ consecutive epochs (common in deep learning: early_stopping = EarlyStopping(patience=10)).

The Regularized Cost Function — General Form.

The regularized cost adds a penalty term to the standard MSE:

$$J(\theta )=\underset{\text{Data-fit term (MSE)}}{\underbrace{\frac{1}{2N}\sum _{i=1}^N{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2}}+\underset{\text{Penalty term}}{\underbrace{\frac{\lambda }{2}\sum _{j=1}^d{\theta }_j^2}}$$

• First term: Standard mean squared error. Measures how well the model fits the training data.
• Second term: Regularization penalty. Measures how large the coefficients are. The $j=1$ start means ${\theta }_0$ (the bias/intercept) is NOT penalized — we only shrink feature weights, not the baseline offset.
• $\lambda$ (lambda): The regularization hyperparameter. Controls the strength of the penalty. $\lambda =0$ = no penalty (ordinary least squares). $\lambda$ very large = all ${\theta }_j$ shrink to zero (model becomes a constant).

Effect of $\lambda$ — the three regimes:

How to choose $\lambda$: Try multiple values (e.g., 0.001, 0.01, 0.1, 1, 10, 100), train with each, and pick the one with the lowest validation error. This is covered in detail in Section 5.12.

Cost Function — Ridge (L2):

$$J(\theta )=\frac{1}{2N}\sum _{i=1}^N{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2+\frac{\lambda }{2}\sum _{j=1}^d{\theta }_j^2$$

The penalty is the L2 norm (Euclidean norm) of the coefficient vector (excluding bias). Squaring means. : a coefficient of 10 contributes $10^2=100$ to the penalty; a coefficient of 1 contributes $1^2=1$. Large values are penalized disproportionately.

Gradient descent update for Ridge:

$${\theta }_j:={\theta }_j-\alpha \left[\frac{1}{N}\sum _{i=1}^N\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j^{(i)}+\lambda {\theta }_j\right]$$

Rewrite to reveal the shrinkage factor:

$${\theta }_j:={\theta }_j(1-\alpha \lambda )-\alpha \cdot \frac{1}{N}\sum _{i=1}^N\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j^{(i)}$$

The shrinkage factor $(1-\alpha \lambda )$: Since $\alpha >0$ and $\lambda >0$, this is a number slightly less than 1. At every update, the current coefficient is multiplied by $(1-\alpha \lambda )$, pulling it toward zero. The professor emphasized. : "This 1 minus alpha into Lambda is a key part. This is the shrinkage factor. It will try to reduce the coefficient."

Bias term (${\theta }_0$): Updated WITHOUT the shrinkage factor — no penalty on the intercept:

$${\theta }_0:={\theta }_0-\alpha \cdot \frac{1}{N}\sum _{i=1}^N\left(h_{\theta }(x^{(i)})-y^{(i)}\right)$$

Effect of the penalty:

• Large ${\theta }_j$ → huge penalty (squared) → strongly shrunk.
• Small ${\theta }_j$ → tiny penalty → barely affected.
• No coefficient ever reaches exactly zero — all features are retained, just dampened.

Cost Function — Lasso (L1):

$$J(\theta )=\frac{1}{2N}\sum _{i=1}^N{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2+\lambda \sum _{j=1}^d|{\theta }_j|$$

The penalty is the L1 norm. Sum of absolute values. Unlike Ridge's $\frac{\lambda }{2}$ factor, Lasso uses $\lambda$ directly. The penalty grows linearly. : a coefficient of 10 contributes 10; a coefficient of 1 contributes 1.

Gradient descent update for Lasso:

$${\theta }_j:={\theta }_j-\alpha \left[\frac{1}{N}\sum _{i=1}^N\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j^{(i)}+\lambda \cdot \text{sign}({\theta }_j)\right]$$

Where $\text{sign}({\theta }_j)=+1$ if ${\theta }_j>0$, $-1$ if ${\theta }_j<0$, and $0$ if ${\theta }_j=0$.

Key difference from Ridge: Instead of multiplying by $(1-\alpha \lambda )$, Lasso subtracts a constant $\alpha \lambda$ (scaled by sign). A constant subtraction can drive a coefficient to exactly zero. Once it crosses zero, the sign flips and it stays at zero. Ridge's proportional shrinkage can only approach zero asymptotically, never reaching it.

Comparison — Ridge vs. Lasso:

Cost Function — Elastic Net:

$$J(\theta )=\frac{1}{2N}\sum _{i=1}^N{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2+\lambda \left[r\cdot \frac{1}{2}\sum _{j=1}^d{\theta }_j^2+(1-r)\cdot \sum _{j=1}^d|{\theta }_j|\right]$$

Two hyperparameters:

• $\lambda$ — overall penalty strength (same role as in Ridge/Lasso).
• $r$ — the L1 ratio (mix parameter), $0\le r\le 1$:
• $r=1$: Pure Ridge (only L2 penalty).
• $r=0$: Pure Lasso (only L1 penalty).
• $0<r<1$: A blend. Common default: $r=0.5$.

Why Elastic Net exists: Lasso has a known weakness. When features are highly correlated (e.g., "house size in sq ft" and "number of rooms"), Lasso tends to pick one arbitrarily and zero out the other. This is unstable — a small change in data can flip which one survives. Elastic Net's L2 component encourages grouping. : correlated features tend to be shrunk together (all kept or all dropped). The L1 component still enables sparsity.

The Cross-Validation Procedure for $\lambda$:

1. Choose candidate values: A logarithmic grid is typical — e.g., $\lambda \in \{0.001,0.01,0.1,1,10,100\}$. The range should span several orders of magnitude.
2. For each $\lambda$:

• Train the model (Ridge, Lasso, or Elastic Net) on the training set.
• Evaluate on the validation set (data never seen during training).
• Record the validation error.

1. Pick the $\lambda$ that gives the lowest validation error.
2. Optional — K-fold cross-validation: Instead of a single train/validation split, split the training data into $K$ folds. For each $\lambda$, train on $K-1$ folds and validate on the remaining fold. Average the validation error across all $K$ folds. This gives a more robust estimate.

The professor's approach: "We normally try four or five times and then we take the lowest value of Lambda. That is a normal approach. You try multiple values of Lambda in your model and whatever gives you the lowest validation error, you pick that."

Purpose. Implement batch gradient descent for linear regression from scratch using NumPy. This code trains a model to learn the relationship $y=2x+1$ from noisy data, then evaluates it with MAE, MSE, and RMSE.

Inputs & Outputs.

• Input: Synthetic data: $X\in {\mathbb{R}}^{m\times 1}$ (feature), $y\in {\mathbb{R}}^m$ (target). Hyperparameters: $\alpha =0.01$, $T=1000$.
• Output: Learned parameters $\theta =({\theta }_0,{\theta }_1)$; evaluation metrics; plots of regression line and cost convergence.

Steps — Annotated NumPy Implementation.

1. Generate synthetic data:

import numpy as np
np.random.seed(42)
X = 2 * np.random.rand(100, 1)        # 100 points, values in [0, 2]
y = 4 + 3 * X + np.random.randn(100, 1)  # y = 4 + 3x + Gaussian noise

The true relationship is $y=4+3x$. The noise randn adds Gaussian scatter. The model should recover ${\theta }_0\approx 4$ and ${\theta }_1\approx 3$.

2. Add bias column to X:

X_b = np.c_[np.ones((100, 1)), X]     # shape (100, 2) — first column is all 1s

np.c_ concatenates along columns. The first column of 1s allows ${\theta }_0$ to act as the intercept. Without this, the model is forced through the origin.

3. Set hyperparameters and initialize:

alpha = 0.01          # learning rate
n_iterations = 1000   # number of epochs
m = 100               # number of training examples
theta = np.random.randn(2, 1)  # shape (2, 1) — random initialization

4. Training loop:

for iteration in range(n_iterations):
    # (a) Predictions: (100,2) @ (2,1) -> (100,1)
    predictions = X_b.dot(theta)

    # (b) Error vector: (100,1) - (100,1) -> (100,1)
    error = predictions - y

    # (c) Cost (MSE/2): scalar
    cost = (1 / (2 * m)) * np.sum(error ** 2)

    # (d) Gradient: (2,100) @ (100,1) -> (2,1)
    gradients = (1 / m) * X_b.T.dot(error)

    # (e) Update parameters simultaneously
    theta = theta - alpha * gradients

Why X_b.T.dot(error) computes the gradient:

$${\nabla }_{\theta }J=\frac{1}{m}{X}^T(X\theta -y)$$

• $X^T$ is $(d+1)\times m$ — transposed design matrix.
• $(X\theta -y)$ is $m\times 1$ — error vector.
• The product is $(d+1)\times 1$ — one gradient component per parameter.
• This is the vectorized version of $\frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}$ for all $j$ simultaneously.

5. Evaluation:

predictions_final = X_b.dot(theta)
mae  = np.mean(np.abs(predictions_final - y))
mse  = np.mean((predictions_final - y) ** 2)
rmse = np.sqrt(mse)
print(f"Learned theta: {theta.ravel()}")
print(f"MAE: {mae:.4f}, MSE: {mse:.4f}, RMSE: {rmse:.4f}")

Using Scikit-learn's SGDRegressor:

from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler

# Feature scaling is CRITICAL for SGD
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)    # mean=0, std=1

sgd_reg = SGDRegressor(
    max_iter=1000,
    tol=1e-3,              # stop if improvement < 0.001
    loss='squared_error',  # MSE loss = linear regression
    penalty=None,          # no regularization (for comparison)
    random_state=42
)
sgd_reg.fit(X_scaled, y.ravel())      # y must be 1D for scikit-learn
predictions = sgd_reg.predict(X_scaled)

The .fit() → .predict() pattern is universal across scikit-learn. Every model — LinearRegression, Ridge, Lasso, LogisticRegression, SVM — follows this same two-step workflow. Memorize it.

Key parameters:

• loss='squared_error' — ordinary least squares (linear regression).
• penalty=None — no regularization (add penalty='l2' for Ridge, penalty='l1' for Lasso).
• tol=1e-3 — stop when cost improves by less than 0.001.
• random_state — seed for reproducibility.

Classification vs. Regression.

Binary classification = exactly two classes (spam/not spam, fraud/legitimate, disease/healthy). This is the focus of logistic regression.

What logistic regression outputs:

• A probability $P(y=1\mid x)\in (0,1)$.
• Not a hard label — you get a confidence score.
• Example: $P(\text{spam}\mid \text{email})=0.87$ means "87% confident this email is spam."
• To make a decision, apply a threshold (default: 0.5):
• $P\ge 0.5$ → predict class 1 (spam).
• $P<0.5$ → predict class 0 (not spam).

The Sigmoid Function:

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

Where:

• $z={\theta }^{\top }x={\theta }_0+{\theta }_1{x}_1+\dots +{\theta }_d{x}_d$ — the logit or linear score.
• $e\approx 2.71828$ — Euler's number.
• $\sigma (z)\in (0,1)$ — always strictly between 0 and 1 (never exactly 0 or 1, but arbitrarily close).

The logistic regression hypothesis:

$$h_{\theta }(x)=\sigma ({\theta }^{\top }x)=\frac{1}{1+e^{-{\theta }^{\top }x}}$$

Behavior of the sigmoid — key values:

Visual intuition — the S-curve:

• X-axis: $z$ (the logit), ranging from about $-6$ to $+6$.
• Y-axis: $\sigma (z)$, ranging from 0 to 1.
• Shape: a smooth S-curve. Flat near 0 on the left, steepest at $z=0$ (slope = 0.25), flat near 1 on the right.
• Landmarks: $\sigma (0)=0.5$ is the center. The curve is symmetric: $\sigma (-z)=1-\sigma (z)$.
• Takeaway: The sigmoid compresses the entire real line into $(0,1)$, with the steepest response around zero.

Why is the sigmoid S-shaped? For large positive $z$, $e^{-z}$ is tiny, so $\sigma (z)\approx \frac{1}{1+0}=1$. For large negative $z$, $e^{-z}$ is huge, so $\sigma (z)\approx \frac{1}{1+\text{huge}}\approx 0$. The transition between these extremes happens smoothly around $z=0$.

The Linear Discriminant Function:

$$f(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\dots +{\theta }_d{x}_d={\theta }^{\top }x$$

Decision rule:

• If $f(x)\ge 0$, predict class 1 (positive class).
• If $f(x)<0$, predict class 0 (negative class).

Equivalently: if $\sigma ({\theta }^{\top }x)\ge 0.5$, predict class 1; else class 0.

The decision boundary is where $f(x)=0$. This is a hyperplane in ${\mathbb{R}}^d$:

$${\theta }_0+{\theta }_1{x}_1+\dots +{\theta }_d{x}_d=0$$

• In 2D ($d=2$): a straight line.
• In 3D ($d=3$): a flat plane.
• In $d$ dimensions: a $(d-1)$-dimensional hyperplane.

Non-Linear Decision Boundaries.

Linear boundaries fail when classes are not linearly separable (e.g., class 1 points clustered in a circle, surrounded by class 0). The solution. : add polynomial terms to the discriminant, just like linear basis functions for regression.

Example — Circular boundary:

$$f(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_1^2+{\theta }_4{x}_2^2$$

With ${\theta }_0=-1$, ${\theta }_1=0$, ${\theta }_2=0$, ${\theta }_3=1$, ${\theta }_4=1$:

$$f(x)=-1+x_1^2+x_2^2=0\Rightarrow x_1^2+x_2^2=1$$

This is a circle of radius 1 centered at the origin. Points inside the circle ($x_1^2+x_2^2<1$) → class 1. Points outside → class 0.

Risk. Overfitting: Adding too many polynomial terms creates a boundary that wiggles to capture every training point, including noise. A 100-degree polynomial boundary achieves 100% training accuracy but fails on new data. The same bias-variance tradeoff from Section 5.6 applies here.

Why MSE Fails for Logistic Regression.

The logistic regression hypothesis is $h_{\theta }(x)=\sigma ({\theta }^{\top }x)=1/(1+e^{-{\theta }^{\top }x})$. This is a nonlinear function of $\theta$. If you plug it into the squared error cost:

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m(\sigma ({\theta }^{\top }{x}^{(i)})-y^{(i)}{)}^2$$

The resulting cost surface is non-convex — full of local minima. Gradient descent can get stuck in any of them. There is no guarantee of finding the global minimum.

The Log Loss (Cross-Entropy) Cost Function.

For one training example with true label $y\in \{0,1\}$ and predicted probability $\hat{y}=h_{\theta }(x)$:

$$\text{Cost}(\hat{y},y)=\{\begin{array}{ll}-\log (\hat{y})& \text{if }y=1\\ -\log (1-\hat{y})& \text{if }y=0\end{array}$$

Combined form (single formula for all examples):

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\log (h_{\theta }(x^{(i)}))+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))\right]$$

How the combined formula works:

• When $y=1$: the second term $(1-y)\log (1-\hat{y})$ becomes $0\cdot \log (1-\hat{y})=0$. Only $-\log (\hat{y})$ remains.
• When $y=0$: the first term $y\log (\hat{y})$ becomes $0\cdot \log (\hat{y})=0$. Only $-\log (1-\hat{y})$ remains.

Cost behavior table:

Derivation — Why Log Loss Gives a Convex Surface.

The log loss is derived from maximum likelihood estimation. Given $m$ independent examples, the likelihood of observing the labels is:

$$L(\theta )=\prod _{i=1}^m{h}_{\theta }(x^{(i)}{)}^{y^{(i)}}(1-h_{\theta }(x^{(i)}){)}^{1-y^{(i)}}$$

Taking the negative log (and dividing by $m$ for scaling) gives the log loss:

$$J(\theta )=-\frac{1}{m}\log L(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\log (h_{\theta }(x^{(i)}))+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))\right]$$

The log-likelihood for logistic regression is concave in $\theta$ (proved by showing the Hessian is negative semidefinite). Therefore, the negative log-likelihood (log loss) is convex. A bowl shape with a single global minimum. Gradient descent is guaranteed to find it.

Problem Summary.

• Goal: Build a linear regression model to predict hourly bike rental counts.
• Data: Training set with features and target labels provided. Separate test set (no labels) for final evaluation.
• Metric: Root Mean Squared Logarithmic Error (RMSLE):

$$\text{RMSLE}=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left(\log (1+{\hat{y}}_i)-\log (1+y_i)\right)}^2}$$

• The $+1$ inside the log prevents $\log (0)$ issues when $y=0$ (zero rentals in an hour).
• RMSLE penalizes under-prediction (predicting 500 when actual is 1000) more than over-prediction (predicting 1500 when actual is 1000) because the log difference is larger below the true value.
• No built-in scikit-learn function — you must implement it yourself using the formula.

Required workflow:

1. EDA (Exploratory Data Analysis): Check dataset size, missing values, feature types (numeric vs. categorical).
2. Correlation analysis: Find which features are most informative about the target.
3. Feature engineering: Create new features (e.g., "is_weekend" from date, "hour_of_day", interaction terms).
4. Visualization: Scatter plots, heat maps, box plots to understand feature-target relationships.
5. Train/validation split: 80% training, 20% validation.
6. Model experimentation: Try feature scaling, polynomial features with Ridge, polynomial features with Lasso.
7. Comparison table: Summarize all models with their RMSLE scores.

Notebook structure tip from the professor:

• Markdown cell: question you are investigating.
• Code cell: analysis/visualization.
• Markdown cell: interpretation of what you found.