Simple Linear Regression (one input variable):

$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1<br>$$

Multiple Linear Regression (more than one input variable):

$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots +{\theta }_n{x}_n={\theta }_0+\sum _{j=1}^n{\theta }_j{x}_j<br>$$

Vectorized form:

$$h_{\theta }(X)=X\theta <br>$$

Here $X$ is an $m\times (n+1)$ matrix — each row is one training example, column 0 is all 1s (for ${\theta }_0$). $\theta$ is an $(n+1)\times 1$ column vector. This single matrix multiplication computes predictions for all $m$ records at once. For one record, use the scalar form.

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2<br>$$

The goal of training is to find the $\theta$ values that minimize $J(\theta )$.

Why the $\frac{1}{2}$? It cancels the 2 from the derivative of the square, keeping the gradient clean. The $\frac{1}{m}$ averages over examples so the cost does not grow with dataset size. In Bishop's standard text, the cost is written without $1/m$ as $E_D(\mathbf{w})=\frac{1}{2}{\sum }_{n=1}^N(t_n-{\mathbf{w}}^{\top }\varphi ({\mathbf{x}}_n){)}^2$. The $1/m$ convention used here makes the gradient magnitude independent of dataset size.

$$\theta =(X^{\top }X{)}^{-1}{X}^{\top }y<br>$$

This gives the exact minimizer in one step. Cost: $O(n^3)$ for the matrix inverse — slow when the feature count $n$ is large.

Method 2 — Gradient Descent: An iterative method. Start with initial $\theta$ values (often zeros). Repeatedly update all ${\theta }_j$ simultaneously:

$${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )<br>$$

The partial derivative:

$$\frac{\partial }{\partial {\theta }_j}J(\theta )=\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}<br>$$

Full update rule:

$${\theta }_j:={\theta }_j-\alpha \cdot \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}<br>$$

Where $\alpha$ is the learning rate — how big each step is.

An overfitted model has low bias (it fits the training data perfectly) and high variance (small changes in training data would produce very different models).

Example pattern (illustrative):
- ${\theta }_0=2.3$ (modest)
- ${\theta }_1=18.5$ (large — 18.5× the effect per unit of $x_1$)
- ${\theta }_2=102.7$ (huge)
- ${\theta }_3=-250.4$ (huge and negative)

These numbers are illustrative, not from a specific dataset. The pattern is what matters: large coefficients ($\gg 1$ or $\ll -1$) signal that the model is twisting itself to fit noise. A well-fitted model typically has modest coefficients.

The goal: balance bias and variance for the lowest total error on new data.

Regularized linear regression: the best model balances two competing objectives:

1. Low prediction error (same as OLS)
2. Small, simple coefficients — all ${\theta }_1,{\theta }_2,\dots ,{\theta }_n$ should be small in magnitude

The model must reduce prediction error while also keeping coefficients simple. A model with enormous coefficients that fits perfectly is penalized more than a model with modest coefficients that fits almost as well.

$$\text{Cost}=\underset{\text{fit the data}}{\underbrace{\text{Prediction Error}}}+\underset{\text{stay simple}}{\underbrace{\lambda \cdot \text{Penalty for Large Coefficients}}}<br>$$

This is a tax on complexity:

- Small coefficients → small penalty → low tax. This is the desired model.
- Large coefficients → large penalty → high tax. The tax offsets the reduction in error, punishing complexity.

The parameter $\lambda$ (lambda) is the tax rate. $\lambda =0$ means no tax — we are back to OLS. Large $\lambda$ means high tax — the model is forced toward extreme simplicity.

This creates a trade-off: the model can choose large coefficients (pay high tax) or small coefficients (pay low tax). The goal is to find the right balance where the marginal benefit of a larger coefficient equals the marginal tax cost.

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2+\frac{\lambda }{2}\sum _{j=1}^n{\theta }_j^2<br>$$

Where $\lambda$ (lambda) is the regularization parameter — the tax rate. The sum runs from $j=1$ to $n$, excluding ${\theta }_0$.

In Bishop's notation (standard reference): $E(\mathbf{w})=\frac{1}{2}{\sum }_{n=1}^N(t_n-{\mathbf{w}}^{\top }\varphi ({\mathbf{x}}_n){)}^2+\frac{\lambda }{2}{\mathbf{w}}^{\top }\mathbf{w}$. The $\frac{\lambda }{2}$ factor is conventional — it cancels with the 2 from the derivative of the squared penalty.

$$\frac{\partial J}{\partial {\theta }_j}=\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}+\lambda {\theta }_j<br>$$

The derivation: the data term differentiates exactly as in standard linear regression (Section 6.1.4). The penalty term $\frac{\lambda }{2}{\theta }_j^2$ differentiates to $\lambda {\theta }_j$ (the $\frac{1}{2}$ cancels the 2 from the power rule).

Plugging into the gradient descent update ${\theta }_j:={\theta }_j-\alpha \frac{\partial J}{\partial {\theta }_j}$:

$$\begin{array}{rl}<br>{\theta }_j& :={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}+\lambda {\theta }_j\right]\\ <br>& ={\theta }_j-\alpha \lambda {\theta }_j-\alpha \cdot \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}\\ <br>& ={\theta }_j\cdot \left(1-\alpha \lambda \right)-\alpha \cdot \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}<br>\end{array}<br>$$

Two parts of this update:
- Shrinkage term: ${\theta }_j\cdot (1-\alpha \lambda )$ — multiplies the old coefficient by a factor < 1, shrinking it
- Data error term: $-\alpha \cdot \frac{1}{m}\sum (\text{error}\cdot x_j)$ — same gradient step as standard linear regression

> Notation note: Some alternative formulations absorb the $1/m$ into $\lambda$ (writing the penalty as $\frac{\lambda }{2m}\sum {\theta }_j^2$), which would give shrinkage factor $(1-\alpha \lambda /m)$. The professor's worked examples (Section 6.10) use the form derived above with shrinkage factor $(1-\alpha \lambda )$. Both are correct under their respective $\lambda$ conventions; the key is to be consistent.

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2+\lambda \sum _{j=1}^n|{\theta }_j|<br>$$

There are no squared terms in the penalty — it is directly proportional to the absolute coefficient values.

In Bishop's notation: the general regularizer form is $\frac{\lambda }{2}{\sum }_{j=1}^M|w_j{|}^q$. Lasso is $q=1$, Ridge is $q=2$. The factor of $\frac{1}{2}$ in Bishop's form is omitted here (as is common in the Lasso literature) since there is no square to cancel.

Ridge's quadratic penalty makes large coefficients disproportionately expensive. Lasso's linear penalty treats all coefficient increases equally. This is why Ridge shrinks big coefficients aggressively while Lasso is willing to keep some large coefficients if they reduce error enough — and eliminate small ones entirely.

$$\frac{\partial }{\partial {\theta }_j}|{\theta }_j|=\text{sign}({\theta }_j)=\{\begin{array}{ll}+1& \text{if }{\theta }_j>0\\ -1& \text{if }{\theta }_j<0\end{array}<br>$$

At ${\theta }_j=0$, the derivative is not defined (the absolute value has a kink). In practice, implementations use subgradient methods to handle this.

Plugging into gradient descent:

$$\begin{array}{rl}<br>{\theta }_j& :={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}+\lambda \cdot \text{sign}({\theta }_j)\right]\\ <br>& ={\theta }_j-\alpha \cdot \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x_j^{(i)}-\alpha \lambda \cdot \text{sign}({\theta }_j)<br>\end{array}<br>$$

Where $\text{sign}({\theta }_j)$ is the sign of the coefficient — $+1$ if ${\theta }_j>0$, $-1$ if ${\theta }_j<0$. This is NOT the trigonometric sine function ($\sin$). It is simply the plus-or-minus sign.

> Important: $\text{sign}({\theta }_j)$ takes only the sign ($+$ or $-$), NOT the magnitude. For ${\theta }_j=-3$, use $-1$, not $-3$.

Contrast with Ridge:

Because Lasso applies the same constant force regardless of coefficient size, a small coefficient can be pushed all the way through zero. That is how Lasso eliminates features.

Why can Lasso hit zero but Ridge cannot? In Ridge, the update multiplies ${\theta }_j$ by $(1-\alpha \lambda )$. A non-zero number times a positive factor is always non-zero. In Lasso, the update subtracts a constant. If ${\theta }_j=0.02$ and the constant push is $-0.1$, the coefficient goes to $-0.08$, crossing zero. With subgradient methods at zero, the coefficient can stay at exactly zero.

$$\text{Penalty}=\lambda \left[(1-R)\sum _{j=1}^n|{\theta }_j|+\frac{R}{2}\sum _{j=1}^n{\theta }_j^2\right]<br>$$

Where $R$ is the mixing parameter (sometimes called $l_1\text{\_ratio}$ in scikit-learn):

- $R=1$ → Pure Ridge (L2 only)
- $R=0$ → Pure Lasso (L1 only)
- $0<R<1$ → Combination of both (typical: $R=0.5$)

The parameter $\lambda$ controls overall penalty strength; $R$ controls the L1/L2 mix.

One-line rule: If you are unsure whether to use Ridge or Lasso, use Elastic Net and tune $R$ by cross-validation.

1. Try a range of $\lambda$ values on a log scale: $10^{-4},10^{-3},10^{-2},10^{-1},10^0,10^1,10^2$.
2. Use cross-validation (Section 6.14): for each $\lambda$, train on part of the data, evaluate on the held-out part. Pick the $\lambda$ with the lowest average validation error.
3. GridSearchCV in scikit-learn automates this: it tries all combinations and returns the best.

The "one standard error" rule: sometimes you pick a slightly larger $\lambda$ (simpler model) whose error is within one standard error of the minimum — this gives a more parsimonious model with essentially the same performance.

Given parameters:
- $\alpha =0.02$ (learning rate)
- ${\theta }_0=5$ (intercept)
- ${\theta }_1=-0.03$ (coefficient for BMI)
- ${\theta }_2=-0.03$ (coefficient for diastolic pressure)
- $m=3$ (three patients)

Hypothesis function:

$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2<br>$$

Additional given:
- $\lambda =5$ (regularization constant)
- $\alpha =0.02$ (unchanged)
- Apply Ridge update rule for 2 iterations

$$\text{Shrinkage factor}=1-\alpha \lambda =1-(0.02\times 5)=1-0.1=0.9<br>$$

This factor $0.9$ is constant across iterations — it depends only on $\alpha$ and $\lambda$, not on ${\theta }_j$ or the data. Every iteration, each coefficient ${\theta }_j$ (for $j\ge 1$) is multiplied by $0.9$ — a 10% reduction — before the data-driven gradient step.

> Formula reconciliation: The professor's general notation in Section 6.5.2 shows shrinkage as $(1-\alpha \lambda /m)$. However, the worked example here computes $(1-\alpha \lambda )$ without dividing by $m$. This is the correct form given the cost function $J(\theta )=\frac{1}{2m}\sum (\text{error}{)}^2+\frac{\lambda }{2}\sum {\theta }_j^2$, where $\frac{\partial }{\partial {\theta }_j}(\frac{\lambda }{2}{\theta }_j^2)=\lambda {\theta }_j$. The update becomes ${\theta }_j(1-\alpha \lambda )-\alpha \cdot \frac{1}{m}\sum \text{error}\cdot x_j$. The worked example below follows this correct derivation. If an alternative convention is used where the penalty is written as $\frac{\lambda }{2m}\sum {\theta }_j^2$, then the shrinkage would be $(1-\alpha \lambda /m)$. Always verify which convention your code or library uses.

$$\alpha \lambda =0.02\times 5=0.1<br>$$

This constant $0.1$ is the amount added or subtracted in each iteration (the sign depends on $\text{sign}({\theta }_j)$). Unlike Ridge, this does NOT depend on the magnitude of ${\theta }_j$ — it is the same push whether ${\theta }_j=-0.03$ or ${\theta }_j=-212$.

The values are not ideal for any method because the dataset is unscaled. But the computation process is what matters:

- Simple LR: updates from data error gradient only.
- Ridge: multiplicative shrinkage $(1-\alpha \lambda )$ on each coefficient BEFORE the data gradient.
- Lasso: constant additive/subtractive penalty $\alpha \lambda$, with sign determined by $\text{sign}({\theta }_j)$.

$$x_{j,\text{scaled}}=\frac{x_j-{\mu }_j}{{\sigma }_j}<br>$$

Where:
- ${\mu }_j$ = mean of feature $j$ across all training examples
- ${\sigma }_j$ = standard deviation of feature $j$ across all training examples

After scaling, every feature has:
- Mean = 0
- Standard deviation = 1

This puts ALL features on the same scale, so gradient descent can take balanced steps in every direction. The cost function becomes a nice round bowl instead of a stretched valley.

- Scaling: StandardScaler, MinMaxScaler, RobustScaler
- Missing value handling: imputation with mean, median, mode, or model-based
- Encoding: one-hot encoding for nominal categories, label/ordinal encoding for ordered categories
- Feature creation: polynomial features, interaction terms, domain-specific transformations
- Feature selection: choosing which features to include (or letting Lasso do it)

Feature engineering is often more important than model choice. A simple linear regression on well-engineered features can outperform a deep neural network on raw data.

Inputs & Outputs. Input: raw training data $(X_{\text{raw}},y)$. Output: a trained model ready for predictions, with all preprocessing parameters (means, standard deviations, imputation values) baked in.

Steps:

1. Missing value imputation — fill gaps using median, mean, or mode (SimpleImputer)
2. Feature scaling — transform features to mean 0, std 1 (StandardScaler)
3. Model training — fit the chosen model (LinearRegression, Ridge, Lasso)

`
Preprocessor: SimpleImputer(strategy='median') → StandardScaler()
Full Pipeline: Preprocessor → Ridge(alpha=lambda_value)


A single pipeline.fit(X_train, y_train) runs imputation → scaling → model fitting. A single pipeline.predict(X_test)` runs the same transformations then predicts.

`python
import joblib
joblib.dump(pipeline, 'model.joblib') # save
loaded_model = joblib.load('model.joblib') # load later
`

- Load a saved model (using joblib)
- Create input widgets (sliders, text boxes, number inputs)
- Run predictions on user-provided values
- Display results with charts and metrics
- Deploy to Streamlit Cloud or run locally

A Streamlit app can include interactive sliders for hyperparameters like $\lambda$, letting users adjust regularization strength and see the effect on predictions in real time.

1. Fit — Train the model on training data: model.fit(X_train, y_train)
2. Predict — Make predictions on test data: model.predict(X_test)
3. Evaluate — Compare predictions to actual values using metrics (MSE, RMSE, $R^2$)

Never evaluate on training data (that gives an overoptimistic estimate). Never use test data during fit() (that leaks information).

Inputs & Outputs. Input: a model (or pipeline), a parameter grid (dictionary of parameter names → lists of values to try), and the training data. Output: the best parameter combination and the best trained model, ready for prediction.

Steps:

1. Define the parameter grid. Example: {'alpha': [0.001, 0.01, 0.1, 1, 10]} for Ridge.
2. For each parameter combination, perform cross-validation:
- Split training data into $k$ folds
- Train on $k-1$ folds, validate on the held-out fold
- Repeat $k$ times, average the score
3. Select the combination with the highest average validation score.
4. Retrain the final model on ALL training data using the best parameters.

``python
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV

param_grid = {'alpha': [0.001, 0.01, 0.1, 1, 10, 100]}
grid = GridSearchCV(Ridge(), param_grid, cv=5, scoring='neg_mean_squared_error')
grid.fit(X_train, y_train)
print(grid.best_params_) # e.g., {'alpha': 0.1}
``

- Split data into 5 equal parts (5-fold CV)
- Train on folds 1–4, test on fold 5 → score₁
- Train on folds 1–3,5, test on fold 4 → score₂
- …repeat 5 times…
- Final score = average of score₁ through score₅

This gives a more reliable estimate than a single train/test split because every example gets used for both training and validation. The downside: 5× the training time.

Left plot (data space): Scatter of training points $(x,y)$. A regression line drawn through them.

Right plot (parameter space / contour): Contour ellipses of $J({\theta }_0,{\theta }_1)$. A point marks the current $({\theta }_0,{\theta }_1)$.

- Each $({\theta }_0,{\theta }_1)$ point on the contour plot corresponds to a different line on the data plot.
- At the start (iteration 0), the line on the left is far from the data points. The point on the right is far from the contour center.
- Each gradient descent update moves the point on the right closer to the center. Simultaneously, the line on the left shifts and rotates toward the best fit.
- At convergence (innermost contour), the line on the left fits the data optimally.

Axes: The contour plot has ${\theta }_0$ on one axis and ${\theta }_1$ on the other. The data plot has $x$ on the horizontal and $y$ on the vertical.

1. The slope of the data error surface — how much does prediction error change when I adjust ${\theta }_j$?
2. The slope of the penalty surface — how much does the penalty change when I adjust ${\theta }_j$?

The hiker combines both slopes into one step. The penalty surface is a simple bowl centered at the origin (minimum penalty when all ${\theta }_j=0$). The data error surface is centered wherever the OLS solution is. Regularized gradient descent finds a compromise point between these two minima — closer to OLS when $\lambda$ is small, closer to the origin when $\lambda$ is large.

In Bishop's visualization (Figure 3.4), the unregularized error contours (blue ellipses) are overlaid with the constraint region (red circle for Ridge, diamond for Lasso). The solution is where the smallest error contour just touches the constraint region. For Lasso, this touch point often lies on an axis — meaning some ${\theta }_j=0$.

- Ridge penalty for logistic regression: Add $\frac{\lambda }{2}{\sum }_{j=1}^n{\theta }_j^2$ to the log-loss cost.
- Lasso penalty for logistic regression: Add $\lambda {\sum }_{j=1}^n|{\theta }_j|$ to the log-loss cost.
- Elastic Net for logistic regression: Add the mixed L1/L2 penalty.

The intention is unchanged: keep coefficients small and simple. The only difference is the data-error term — log loss instead of MSE. The gradient of the penalty (and thus the shrinkage/constant-push behavior) is identical.

In scikit-learn: LogisticRegression(penalty='l2', C=1/lambda) for Ridge-regularized logistic regression. The parameter C is the inverse of $\lambda$ — $C=1/\lambda$. So small C = strong regularization. This is a common source of confusion.