Machine Learning

Feature Engineering and Linear Regression

📅 Published: 2026-06-27

🎓 Level: postgraduate

👥 Audience: Postgraduate ML students — second lecture covering feature engineering techniques and linear regression fundamentals

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Features, Attributes, Predictors, and Dimensions — covered in Lecture 1 (Introduction to Machine Learning)
Regression (Supervised Learning) — covered in Lecture 1 (Introduction to Machine Learning)
Dimensionality Reduction (Preview) — covered in Lecture 1 (Introduction to Machine Learning)
Nominal and Ordinal Attributes — covered in Lecture 2 (Data Preprocessing for Machine Learning)
Normalization and Standardization — covered in Lecture 2 (Data Preprocessing for Machine Learning)
Data Preprocessing Overview — covered in Lecture 2 (Data Preprocessing for Machine Learning)

Feature Engineering and Linear Regression — Complete Lecture Notes

3.1 Feature Engineering — Overview

Hook: You have a spreadsheet with 100 columns. Your model chokes on it. Which columns matter? Which ones are just noise? And can you invent better columns than the ones you started with?

Intuition + Analogy: Think of feature engineering as preparing ingredients for cooking. Raw data is like whole vegetables — you wash them (cleaning), peel and chop them (transformation), decide which ones actually belong in this dish (selection), and sometimes combine them into a sauce that tastes better than any single ingredient alone (extraction/construction). A chef with great ingredients but no prep skill produces a bad meal. A data scientist with great raw data but no feature engineering produces a bad model. The analogy breaks here: in cooking, you can taste as you go; in ML, you need systematic methods to know if your features are working.

A feature is a property, attribute, or column of structured data. The set of all features is denoted by capital $X$ . Features are also called characteristics or dimensions.

Feature engineering is the process of using domain knowledge to extract, select, create, construct, or transform features so that a machine learning model can learn effectively from them. The goal is to produce a set of features that the model can use efficiently for the task at hand.

Feature engineering includes four major tasks: - Feature extraction — creating a new, smaller set of features by combining original features (dimensionality reduction) - Feature selection — removing irrelevant or redundant features - Feature construction — creating new features from existing ones using domain knowledge - Feature transformation — converting features from one form to another (discretization, encoding, binarization)

Motivating Case Study — Dissertation Eligibility Prediction

Consider a cohort of students enrolled in a postgraduate program. They have completed two semesters with subjects: ISM, MFML, ACI, ML, NLP, and others. The goal: predict, at the end of semester 2, how many students will be eligible for the dissertation semester. Eligibility requires a minimum CGPA of 5.5 or above at the end of semester 3. Since semester 3 results are not yet available, the prediction must use semester 1 and 2 data.

This case study illustrates how different feature engineering tasks apply to a real prediction problem. We will return to it throughout the lecture.

Scope: Feature engineering is not a one-time step. It is iterative — you engineer features, train a model, evaluate, and loop back. The four tasks (extraction, selection, construction, transformation) are not a fixed pipeline; you may apply them in any order and revisit earlier steps.

Visual Intuition: Picture a messy desk covered with papers (raw data). Feature selection throws away irrelevant papers. Feature extraction shreds and recombines papers into summary sheets. Feature construction staples related papers together into new documents. Feature transformation reformats papers into a consistent template. The clean desk that results is your training dataset.

Pitfalls:

1. Skipping domain knowledge. Feature engineering without understanding what the columns mean produces garbage features. The student ID number is a perfect predictor of the target — if you sorted the data by the target before assigning IDs. Always ask: "Does this column mean something, or is it just an artifact?" 2. Doing everything at once. Applying extraction, selection, construction, and transformation blindly creates a tangled mess. Apply each technique with a clear reason. 3. Forgetting to document transformations. The model in production must see data transformed the exact same way. Undocumented feature engineering is a time bomb.

Feature engineering is the craft of turning raw columns into model-ready signals. The four tasks — extraction, selection, construction, transformation — are your toolkit. Domain knowledge is the most important ingredient in every one of them.

Real-World & Domain Connection: In Kaggle competitions, feature engineering is routinely the difference between top-10 and top-100 finishes. In industry, engineered features often outlast the models that use them — a company's feature store (a catalog of curated, documented features) is more valuable than any single trained model. At Uber, features like "average trip duration by hour and neighborhood" are engineered once and reused across dozens of models for pricing, ETA prediction, and fraud detection.

3.2 Feature Extraction

Hook: You have 100 columns, and most of them say roughly the same thing. Can you squeeze all that redundancy into just 5 columns that capture almost everything?

Intuition + Analogy — The Blender: Think of PCA as a blender. You put all your numerical attributes (fruits, vegetables) into the blender. The blender mixes them and produces smoothies. Smoothie 1 (PC1) is a weighted combination of all original attributes. Smoothie 2 (PC2) is a different weighted combination. Smoothie 3 (PC3) is yet another. After blending, you no longer see individual ingredients — you see combined flavors. The analogy breaks here: in a real blender, you lose the original ingredients; in PCA, the transformation is reversible (you can reconstruct the original data from all PCs, though you typically keep only the top few).

Feature extraction means creating a new, smaller set of features by combining the original features. It does not simply select a subset — it creates entirely new features as combinations. This is dimensionality reduction: reducing the number of dimensions while preserving information.

The key idea: instead of using original attributes individually, create combined attributes that may explain the target better than any single original attribute.

3.2.1 Principal Component Analysis (PCA) — Conceptual Overview

PCA is the primary technique for feature extraction. It works only on numerical data, so categorical columns must be converted to numerical form first.

For example, PC1 might be: $PC1 = 0.5 \times ISM + 0.4 \times MFML + 0.3 \times SE + 0.2 \times ML + 0.1 \times NLP$

PC2 and PC3 would be different weighted combinations, each perpendicular to the others.

Key property: If there are 10 original numerical features, PCA creates exactly 10 principal components (PCs). Dimensionality reduction does NOT happen by creating fewer components — it happens by ordering the PCs by how much variance each explains, then selecting only the top few. Variance explained: Each PC is ordered by its ability to explain the variance (spread) in the data. PC1 captures the maximum variance. PC2 captures the maximum remaining variance (and is perpendicular to PC1). PC3 captures the next maximum remaining variance (perpendicular to both PC1 and PC2). After sorting, you might keep only the top 5 PCs out of 10 — that is where dimensionality reduction happens. What is variance? Variance measures how spread out the data is around the mean. Low variance means data points cluster close to the mean. High variance means data points are far from the mean. In PCA, you want components that can explain the maximum spread of the data.

Worked Example — The Camera Analogy

Imagine a group of people standing in a room. You want to position a camera so that one photograph captures all their faces — maximum spread. If you place the camera at the wrong angle, some faces are hidden behind others. PCA finds the camera angle (the new axis) that captures the maximum variance.

Projection intuition: Data points that lie roughly along a straight line are poorly explained by the original X and Y axes. PCA finds a new axis along that line. Projecting all points onto this new axis gives a better representation than either original axis alone. When PCA helps: If you have 100 highly correlated features, PCA can combine them into perhaps 5 new principal components that explain most of the data's variance. The first two principal components capture as much of the variation in the data as is possible with two orthogonal attributes that are linear combinations of the original attributes.

Assumptions & Scope:

- Assumption: PCA assumes linear relationships among features. If the data lies on a curved manifold, PCA will not find the best low-dimensional representation. - Assumption: PCA is sensitive to the scale of features. Always standardize (zero mean, unit variance) before applying PCA. A feature measured in millimeters will dominate a feature measured in kilometers otherwise. - Scope: PCA works only on numerical data. Categorical features must be encoded first. - When it breaks: If features are truly independent (no correlation), PCA provides no benefit — every PC explains roughly equal variance, and you cannot drop any without losing information.

Visual Intuition: Plot the original data as a scatter cloud in 2D. PCA draws a new axis (PC1) through the longest direction of the cloud — the direction of maximum spread. PC2 is drawn perpendicular to PC1, through the next-longest direction. The takeaway: PCA rotates the coordinate system to align with the data's natural axes of variation.

Pitfalls:

1. Forgetting to standardize. PCA on unstandardized data gives the feature with the largest numeric range all the weight. Always scale first. 2. Keeping too many PCs. The goal is dimensionality reduction. If you keep 9 out of 10 PCs, you have saved almost nothing. Use a scree plot (variance explained vs. PC number) and pick the elbow. 3. Interpreting PCs as "meaningful." PC1 = 0.5×ISM + 0.4×MFML is a mathematical construct, not a real-world quantity. Don't name it "Academic Ability" without strong evidence.

Q: Can we do two levels of reduction — apply PCA twice? A: Once PCA is applied, the data is transformed. A second application would work on already-transformed data. This is beyond the current scope. Q: Is PCA supervised or unsupervised? A: PCA is unsupervised — it does not use the target variable. It only looks at the spread of the input features. The key point is that feature extraction (dimensionality reduction) is different from feature selection. In extraction, you create combinations of attributes. If there are 10 attributes, there are 10 PCs, but you do not use all 10 — that is where dimensionality reduction happens.

PCA is a blender for your numerical columns: it mixes them into new "smoothie" features ordered by how much data spread each one captures. Keep only the top smoothies and you have dimensionality reduction.

Real-World & Domain Connection: PCA is used extensively in genetics (population structure analysis from thousands of gene markers reduced to 2-3 PCs that often map to geographic ancestry), finance (yield curve modeling — hundreds of bond yields reduced to 3 factors: level, slope, curvature), and computer vision (eigenfaces for face recognition). In the dissertation eligibility case study, if ISM, MFML, ML, and NLP scores are all highly correlated, PCA could combine them into a single "quantitative aptitude" component.

3.3 Feature Selection

Hook: Your model has 100 features. Training takes hours. Half the features are noise, and a quarter are copies of each other. How do you find the keepers without trying all

2^{100}

possible combinations?

Intuition + Analogy — Packing for a Trip: You have a closet full of clothes but a small suitcase. You don't bring everything — you pick what you need. Some items are irrelevant (a winter coat for a beach trip). Some are redundant (two nearly identical blue shirts — bring one). Feature selection is packing: keep the useful, drop the useless and the duplicate. The analogy breaks here: in packing, you know the destination; in feature selection, you may not know which features matter until you test them.

Feature selection means identifying and keeping only the relevant features while removing irrelevant and redundant ones. Unlike feature extraction, feature selection does NOT create new features — it drops existing ones. What makes a feature irrelevant? Domain knowledge tells you. For predicting dissertation eligibility, the student's name, gender, age, date of birth, and organization are irrelevant. The scores in ISM, MFML, ACI, ML, NLP and the student's previous degree are relevant. What makes a feature redundant? Two features are redundant if they are highly correlated — knowing one tells you the other. If ISM score and MFML score are highly correlated (when ISM increases, MFML also increases), you need only one of them to predict CGPA. Handling missing values: If a column has a large number of missing values (e.g., more than 50 out of 100 records are missing ISM scores), that column may be dropped entirely.

3.3.1 Three Methods of Feature Selection

3.3.1.1 Filter Method

Uses statistical measures to evaluate features before any ML model is involved. Example: calculate the Pearson correlation between every pair of features. If two features are highly correlated, drop one. This is fast and model-agnostic.

3.3.1.2 Wrapper Method

Uses an ML algorithm itself to evaluate which features to keep. The model is run repeatedly with different feature subsets.

- Forward selection: Start with 1 feature, run the model, note accuracy. Add a second feature, run again. If accuracy improves, keep it. Add a third, and so on. Stop when adding features no longer improves accuracy. - Backward elimination: Start with ALL features (say 100), run the model, note accuracy. Remove one feature (99 features), run again. If accuracy improves, keep the reduced set. Continue removing features. Stop when removal hurts accuracy.

Wrapper methods are very expensive because the model must be trained many times. If using decision trees, you retrain the model for every feature subset tested.

3.3.1.3 Embedded Method

Feature selection happens as part of the model training process itself. You do not pre-select features and then train — the algorithm selects features while it trains. Examples: decision trees and random forests perform feature selection inherently during tree construction. LASSO regression (L1 regularization) drives irrelevant feature coefficients to exactly zero, performing selection during training.

Worked Example — Dissertation Eligibility with Wrapper Forward Selection

Start with 1 feature: ISM score. Train a model → accuracy = 65%. Add MFML score (2 features). Train → accuracy = 72%. Keep it. Add ACI score (3 features). Train → accuracy = 73%. Keep it. Add student age (4 features). Train → accuracy = 72%. Drop it — adding age hurt performance. Add NLP score (4 features, having dropped age). Train → accuracy = 75%. Keep it. Stop when no new feature improves accuracy.

Sense-check: The selected features (ISM, MFML, ACI, NLP) are all academic scores — they make sense for predicting CGPA. Age was correctly discarded as irrelevant.

Assumptions & Scope:

- Assumption (Filter): Statistical measures like correlation capture only linear relationships. Two features could be nonlinearly related and a filter method would miss it. - Assumption (Wrapper): The model used for evaluation is the same model you will deploy. A feature subset optimal for decision trees may not be optimal for SVM. - Scope: Feature selection reduces overfitting risk by removing noise features. But aggressive selection can discard weakly predictive features that together would be strong.

Visual Intuition: Imagine a Venn diagram. The left circle is "features correlated with the target" (relevant). The right circle is "features correlated with each other" (redundant). You want features in the left circle but not in the right. Filter methods draw these circles using statistics. Wrapper methods draw them by actually training models.

Pitfalls:

1. Dropping features based on low individual correlation. A feature may have near-zero correlation with the target alone but be key in combination with another feature (interaction effect). Filter methods miss this. 2. Wrapper overfitting to the validation set. If you try 1000 feature subsets and pick the best, you have effectively trained on the validation set. Use a separate holdout set for final evaluation. 3. Ignoring domain knowledge. A doctor tells you "this lab value matters." A filter method says it has low correlation. Trust the doctor — the correlation may be nonlinear or context-dependent.

Q: In what order should we do feature selection and feature extraction? A: There is no fixed order. You can do feature selection first, then feature extraction. The explanation here is not in any prescribed sequence — it is just explaining the terms.

Feature selection is packing for your model: keep what helps, drop what doesn't, and never bring two copies of the same thing. Filter is fast; wrapper is thorough; embedded is automatic.

Real-World & Domain Connection: In genomics, feature selection is critical — you may have 20,000 gene expression measurements but only 200 patient samples. Without feature selection, any model will overfit catastrophically. In credit scoring, regulations like the Equal Credit Opportunity Act require that you can explain which features drive decisions — feature selection produces simpler, auditable models. The dissertation eligibility case study is a classic feature selection scenario: from dozens of student attributes, select the handful that actually predict CGPA ≥ 5.5.

3.4 Feature Construction

Hook: Your dataset has total_orders and customer_since_date. Neither predicts churn well alone. But orders_per_month — a column you don't have — might be the single best predictor. Can you invent it?

Intuition + Analogy — Building with LEGO: You have individual LEGO bricks (original features). A red brick alone tells you little. A blue brick alone tells you little. But snap them together and you get something new — a shape neither brick could represent alone. Feature construction is snapping existing columns together to build columns that capture relationships the original columns miss. The analogy breaks here: LEGO combinations are physical; feature combinations are mathematical, and the right math (multiply? divide? square?) depends on domain knowledge.

Feature construction means creating entirely new features from existing ones. Domain knowledge is the most important ingredient. Examples:

- In e-commerce: from total_orders and customer_since_date, create average_orders_per_month. - In real estate: from total_rooms and total_population, create rooms_per_person.

3.4.1 Polynomial Expansion

When the relationship between a feature and the target is not a straight line but a curve (e.g., a U-shape or parabola), a straight-line model cannot capture it. Polynomial expansion creates new features by raising a numerical feature to a power.

Example: In housing, both very small and very large houses may command high prices per square foot, while medium-sized houses have lower prices per square foot. This U-shaped relationship cannot be captured by a linear term alone. Adding $x^{2}$ (square of the size) as a new feature allows the model to fit a curve.

The model goes from: $h_{θ} (x) = θ_{0} + θ_{1} x$ to: $h_{θ} (x) = θ_{0} + θ_{1} x + θ_{2} x^{2}$

This is still a linear model — linear in the parameters $θ_{j}$ . The nonlinearity is in the features, not the parameters.

3.4.2 Feature Crossing

Sometimes individual features contribute little to predicting the target, but their combination is highly predictive. Feature crossing combines two or more features into one.

Example — Latitude and Longitude: Latitude alone may not predict something well. Longitude alone may not either. But latitude × longitude together (a specific location) can be very predictive. Example — City and Job Type: Consider two cities (New York, London) and two job types (Finance, Technical). To predict salary:

- "Finance" alone does not tell you much. - "New York" alone does not tell you much. - But "Finance in New York" vs "Finance in London" may predict very different salaries.

The combined feature (city × job type) captures the interaction effect that neither feature captures alone.

Example — Medical diagnosis: Individual readings like diabetes test results and blood pressure readings may not predict a disease well on their own. But the combination of diabetes result AND BP result together may have a strong effect on disease prediction.

Worked Example — Dissertation Eligibility: Choosing the Right Representation

You could represent student performance as: - Total marks per semester → requires further calculation to get GPA; adds an unnecessary step for the model - GPA per semester → easier for the model to use; already normalized - CGPA (cumulative GPA across all semesters) → directly maps to the ≥5.5 eligibility threshold; the most direct predictor - Is Complete (a simple yes/no status) → loses all granularity; a student at 5.4 and a student at 2.0 look identical

Best choice: CGPA gives the most direct prediction for the eligibility criterion. The choice of representation affects model performance dramatically — a model predicting from "Is Complete" can never distinguish the borderline 5.4 student from the failing 2.0 student.

Assumptions & Scope:

- Assumption: Polynomial expansion assumes the relationship is polynomial (quadratic, cubic, etc.). If the true relationship is exponential or logarithmic, polynomial terms may fit poorly or require very high degrees. - Assumption: Feature crossing assumes the interaction is multiplicative. Some interactions are better captured by other operations (ratio, difference, logical AND). - Scope: Feature construction can explode the feature count. Crossing every pair of 100 features produces ~5000 new features. Use domain knowledge to guide which crosses to create.

Visual Intuition: Plot house price vs. size. The points form a U-shaped smile. A straight line cuts through the middle, missing both ends. Add

x^{2}

as a new axis (a new dimension). In this 3D space (size, size², price), the points lie on a plane — and a plane is something linear regression can fit perfectly.

Pitfalls:

1. Creating features without domain logic. age × income might make sense; age × shoe_size probably doesn't. Every constructed feature should have a story. 2. Polynomial degree too high. $x, x^{2}, x^{3}, x^{4}, x^{5}$ will fit the training data perfectly — and fail catastrophically on new data. Start with degree 2, only go higher with strong evidence. 3. Data leakage through construction. If you compute average_orders_per_month using future data (orders from months that haven't happened yet at prediction time), your model will look great in training and fail in production.

Q: By adding new features, aren't we giving more weight to those features? Isn't that wrong? A: No. By adding new features, you are capturing the combined effect of individual features. You keep the original features in the dataset, but for the model you use the combined feature. The combined feature represents the interaction, not a duplication. The model learns weights for each feature independently — if the combined feature is useful, it gets a non-zero weight; if not, the model can set its weight to zero.

Feature construction is inventing better columns from the ones you have. Polynomial expansion captures curves; feature crossing captures interactions. Domain knowledge tells you which inventions are worth trying.

Real-World & Domain Connection: At Airbnb, a constructed feature nights_booked / nights_available (occupancy rate) is far more predictive of listing quality than either raw count alone. In fraud detection, transaction_amount / average_user_transaction_amount flags anomalous spending instantly. In the dissertation case study, constructing CGPA from individual semester GPAs is itself feature construction — and it is the single most important feature for predicting eligibility.

3.5 Feature Transformation — Discretization (Binning)

Hook: You have ages: 21, 22, 23, 24, 25... Does a 21-year-old really behave differently from a 22-year-old for your problem? Or do they both just belong to "Young Adult"?

Intuition + Analogy — Sorting Mail by Zip Code: The post office doesn't deliver to every individual house number from a central hub. They bucket mail by zip code first, then by street, then by house. Discretization is creating zip codes for your numbers — grouping nearby values into meaningful buckets. The analogy breaks here: zip codes are geographic and fixed; discretization boundaries are chosen by you and can make or break your model.

Discretization (also called binning or bucketing) is the process of converting a continuous numerical attribute into a discrete categorical attribute. Example — Age groups: Instead of analyzing every individual age value (21, 22, 23, ...), group them into categories: Young, Adult, Middle Age, Senior. Small variations within a group (21 vs 22 vs 23) are treated as the same category. Why discretize? Many algorithms prefer or require discrete features: Bayesian algorithms, decision trees, ensemble learning, random forest, minimum distance classifiers, K-Nearest Neighbors (KNN). Discretization helps handle outliers and noise: Small, insignificant variations (e.g., 70, 70.5, 71, 72.5) can be grouped together, making the model more stable. Types of discretized labels:

- Interval labels: 0–10, 11–20, 21–30 (numerical ranges) - Conceptual labels: Youth, Adult, Senior (meaningful categories)

Important: Discretized categories need not be ordinal. The age example coincidentally produced ordinal groups, but discretization does not always preserve order.

3.5.1 Equi-Width Binning

All bins have the same numerical width.

Procedure:

1. Sort the data in ascending order. 2. Decide the number of bins $K$ . 3. Calculate bin width: $width = \frac{m a x - m i n}{K}$ 4. Create bins of equal width.

Worked Example — Equi-Width Binning

Data: 30, 2, 9, 8, 21, 22, 20, 29, 10

Step 1 — Sort: 0, 2, 8, 10, 20, 21, 29, 29, 30

Step 2 — $K = 3$ bins.

Step 3 — Width = $\frac{30 - 0}{3} = 10$

Step 4 — Bins (using the standard convention [lower, upper) for all bins except the last which is [lower, upper]): - Bin 1: [0, 10) → values {0, 2, 8} - Bin 2: [10, 20) → values {10} - Bin 3: [20, 30] → values {20, 21, 29, 29, 30}

Note on boundary placement: The original example placed 10 in Bin 2 and 20 in Bin 2 as well, which is inconsistent with a strict width-of-10 partitioning from 0. The standard convention resolves this: each bin is [lower, upper) — includes lower bound, excludes upper bound — except the final bin which is [lower, upper] to capture the maximum value. Under this convention, 10 falls in Bin 2 and 20 falls in Bin 3. The important lesson is not the exact bin assignment but the method: equal-width bins can distribute data very unevenly when the data is not uniform. Sense-check: Bin 1 has 3 values, Bin 2 has 1 value, Bin 3 has 5 values. The bins are equal width (10 each) but very unequal depth. This is the key weakness of equi-width binning.

3.5.2 Equi-Depth Binning (Equi-Frequency Binning)

All bins have about the same number of data points.

Procedure:

1. Sort the data in ascending order. 2. Decide how many values per bin ( $K$ ). 3. Divide the sorted data into groups of $K$ values each. 4. Any remainder forms an extra bin.

Worked Example — Equi-Depth Binning

Same sorted data: 0, 2, 8, 10, 20, 21, 29, 29, 30

$K = 3$ values per bin.

- Bin 1: {0, 2, 8} - Bin 2: {10, 20, 21} - Bin 3: {29, 29, 30}

If there were a 10th value (e.g., 31), it would form Bin 4: {31}.

Sense-check: Each bin has exactly 3 values. The bins have very different widths (Bin 1 spans 0–8 = width 8; Bin 2 spans 10–21 = width 11; Bin 3 spans 29–30 = width 1), but equal representation. This handles skewed data well.

3.5.3 When to Use Which

Situation	Use
-----------	-----
Data is heavily skewed (values clustered on one side)	Equi-depth
Data has natural clusters	Equi-depth
Data is evenly spread	Equi-width
You want quartile binning (Q1, Q2, Q3, Q4)	Equi-depth

Why avoid equi-width for skewed data? If data is skewed right, the left-side bins will be nearly empty. If data clusters in the 21–30 range, the 0–10 and 11–20 bins will have very few values — because bin widths are computed from max and min, not from data distribution. Quartile binning belongs to equi-depth: With 100 data points and 4 bins, you want 25% of data in each bin — same number of observations per bin, regardless of numerical width. The 0–25th percentile goes in bin 1, 25th–50th in bin 2, 50th–75th in bin 3, 75th–100th in bin 4.

Visual Intuition: Draw a number line from 0 to 30. For equi-width, draw vertical lines at 10 and 20 — equal spacing, but the points cluster on the right. For equi-depth, draw vertical lines at 8 and 21 — unequal spacing, but each region contains exactly 3 points. The takeaway: equi-width cares about the ruler; equi-depth cares about the dots.

Pitfalls:

1. Using equi-width on skewed data. You get empty or near-empty bins, which are useless for learning. Always check the histogram of your data before choosing a binning strategy. 2. Too many bins. 100 bins on 100 data points gives ~1 point per bin — you have just replaced a continuous variable with a unique-ID variable. The whole point of discretization is to group similar values. 3. Too few bins. 2 bins throws away almost all information. The number of bins is a tradeoff: more bins preserve more information but risk empty bins; fewer bins are stable but coarse. 4. Arbitrary boundaries. Binning ages as 0–20, 21–40, 41–60, 61+ is arbitrary. Why 20 and not 18? Domain knowledge should drive boundary choices when possible (e.g., legal drinking age, retirement age).

Discretization groups continuous numbers into buckets. Equi-width makes equal-sized buckets on the number line; equi-depth puts equal numbers of data points in each bucket. Use equi-depth when your data is skewed; use equi-width when it is uniform.

Real-World & Domain Connection: Credit scoring models discretize income into brackets because the relationship between income and creditworthiness is not linear — going from $10K to $20K matters more than going from $200K to $210K. Medical diagnosis systems discretize lab values into "low," "normal," and "high" based on clinical reference ranges — domain knowledge, not equal-width or equal-depth, drives the boundaries.

3.6 Feature Transformation — Encoding

Hook: Your model speaks numbers. Your data says "red," "green," "blue." How do you translate without inventing a fake ordering (red > green > blue)?

Intuition + Analogy — Multiple-Choice vs. Fill-in-the-Blank: Label encoding is like numbering answers 1, 2, 3 — it implies an order (3 > 2 > 1). One-hot encoding is like giving each answer its own checkbox — checked or unchecked, no ordering implied. Use checkboxes (one-hot) when the categories are just names. Use numbering (label) only when the categories have a natural order. The analogy breaks here: in a test, you pick exactly one answer; in one-hot encoding, exactly one column is 1 and the rest are 0 — this is the same idea mathematically.

Encoding is the process of converting categorical features into numerical format. Most ML algorithms (linear regression, neural networks, SVM) cannot process text labels — they need numbers.

Encoding is the reverse of discretization: discretization goes numerical → categorical; encoding goes categorical → numerical.

3.6.1 One-Hot Encoding

The most common and safest encoding method. For each distinct category value, create a new binary column (a "dummy column").

When to use one-hot encoding:

- The categories are nominal (no natural order among values — red, green, blue have no inherent ranking). - The number of distinct categories is small. If there are too many distinct values, creating that many columns becomes an overhead.

Worked Example — One-Hot Encoding

Original column color with values: red, green, blue, red.

Three distinct values → three new columns: color_red, color_green, color_blue.

Original	color_red	color_green	color_blue
----------	-----------	-------------	------------
red	1	0	0
green	0	1	0
blue	0	0	1
red	1	0	0

A 1 indicates the presence of that category; 0 indicates absence.

Sense-check: Each row has exactly one 1 — the categories are mutually exclusive. No false ordering is implied. The original column can be perfectly reconstructed from the three encoded columns.

3.6.2 Label Encoding

Assign a unique integer to each distinct category value.

When to use label encoding:

- The categorical attribute is ordinal (has a natural order). Examples: ratings (poor, average, good, excellent), survey responses (not satisfied, neutral, satisfied). The integers 0, 1, 2, 3 then carry meaningful order information.

When NOT to use label encoding: For nominal attributes like color, label encoding is inappropriate because the model may interpret 2 > 1 > 0 as a meaningful ordering when none exists. One-hot encoding is correct for nominal data.

Worked Example — Label Encoding

Same color column: red, green, blue, red.

Assign: blue → 0, green → 1, red → 2 (assignment is arbitrary).

Original	encoded_color
----------	---------------
red	2
green	1
blue	0
red	2

Sense-check: The model sees numbers 0, 1, 2 and may learn that "higher number = better" — which is nonsense for colors. This is why label encoding is dangerous for nominal data. For ordinal data like "poor"=0, "average"=1, "good"=2, "excellent"=3, the ordering is real and label encoding is appropriate.

Assumptions & Scope:

- Assumption (One-hot): Categories are mutually exclusive — each observation belongs to exactly one category. - Assumption (Label): The integer assignment preserves the true ordering of categories. 0 < 1 < 2 must match poor < average < good. - Scope: One-hot encoding explodes feature count. A column with 1000 categories creates 1000 new columns. For high-cardinality categorical features, consider target encoding or embedding methods instead.

Visual Intuition: Picture a spreadsheet. One-hot encoding turns one "Color" column into three columns, each with 0s and 1s — like turning a single multiple-choice question into three true/false questions. Label encoding turns "red," "green," "blue" into 2, 1, 0 — like assigning each color a podium position in a race they never ran.

Pitfalls:

1. Label encoding nominal data. The model learns "red > green > blue" and makes decisions based on this fake ordering. Always ask: does the order of my categories mean anything? 2. The dummy variable trap. If you create one-hot columns for all $k$ categories, the columns sum to 1 for every row — perfect multicollinearity. Drop one column (the "reference category") to break the trap. For linear regression, this is essential; for tree-based models, it matters less. 3. High-cardinality one-hot. A "zip code" column with 500 unique values creates 500 new columns. Your dataset becomes 99.8% zeros (sparse). Consider grouping rare categories into "Other" before encoding.

Encoding translates categories into numbers. One-hot is the safe default for nominal data (no fake ordering). Label encoding is for ordinal data (real ordering). Never label-encode nominal categories.

Real-World & Domain Connection: In natural language processing, one-hot encoding of words produces vectors of dimension equal to vocabulary size (often 10,000+). This sparsity motivated word embeddings (Word2Vec, GloVe) — dense vectors that capture semantic meaning. In the dissertation eligibility case study, "previous degree" (BSc, BA, BCom, BTech) is nominal — one-hot encode it. "Letter grade" (A, B, C, D, F) is ordinal — label encode it.

3.7 Feature Transformation — Binarization

Hook: You have test scores: 85, 42, 91, 30, 50. You don't care about the exact score — you just need to know: pass or fail?

Intuition + Analogy — Light Switch: A dimmer lets you set any brightness from 0 to 100. A light switch has exactly two states: ON or OFF. Binarization is replacing the dimmer with a switch — you pick a threshold, and everything above it becomes ON (1), everything below becomes OFF (0). The analogy breaks here: a real dimmer and switch control the same bulb; binarization throws away information permanently — you cannot recover the original 85 from a 1.

Binarization converts data into exactly two categories (0 or 1). It is distinct from both discretization and encoding.

3.7.1 Binarization of Categorical Features

When a categorical feature already has exactly two categories (yes/no, male/female, true/false), simply assign 1 to one category and 0 to the other.

Example: gender with values male, female → male = 1, female = 0.

3.7.2 Binarization of Numerical Features

When a numerical feature needs to be converted to a binary outcome, define a threshold from domain knowledge. Values above the threshold become 1; values below become 0.

Worked Example — Binarizing Test Scores

Test scores: 85, 42, 91, 30, 50

Threshold: score > 50 means Pass (1); score ≤ 50 means Fail (0).

Original Score	Binarized
----------------	-----------
85	1
42	0
91	1
30	0
50	0

The threshold logic must be documented in code with an explicit comment explaining the chosen cutoff.

Sense-check: The binarized column answers exactly one question: "Did this student pass?" The original scores are lost. This is appropriate when the downstream task only needs the pass/fail distinction — for example, predicting dissertation eligibility (CGPA ≥ 5.5 → 1, CGPA < 5.5 → 0).

3.7.3 Distinguishing Binarization from One-Hot Encoding

One-hot encoding creates as many columns as there are distinct categories, with values of 0 or 1 in each. Binarization always produces exactly two categories (a single binary column). One-hot encoding can be seen as a form of binarization (since its values are 0/1), but it is best to keep the concepts distinct: binarization = divide into two; one-hot = one column per category with binary indicators.

Assumptions & Scope:

- Assumption: The threshold is meaningful. "Score > 50 = Pass" is meaningful because 50 is the passing mark. "Age > 37 = 1" needs justification — why 37? - Scope: Binarization is the most extreme form of discretization (exactly 2 bins). Use it when the target concept is inherently binary (pass/fail, eligible/ineligible, fraud/not-fraud).

Visual Intuition: Draw a number line. Place a single vertical line at the threshold (e.g., 50). Everything to the left is 0. Everything to the right is 1. That is the entire transformation — one line, two regions.

Pitfalls:

1. Arbitrary threshold. "I'll just use the median" is not domain knowledge. The threshold should come from the problem: a legal age, a passing grade, a medical cutoff. 2. Information loss. Binarizing age throws away the difference between a 60-year-old and an 80-year-old. Only binarize when the binary distinction is truly what matters. 3. Threshold at the wrong place. If you binarize CGPA at 5.5 for eligibility prediction, a student at 5.4 and a student at 2.0 both become 0 — but the 5.4 student is far closer to being eligible. Consider whether you need the granularity.

Q: When you binarize, how do you set the threshold? A: You must define the threshold from domain knowledge and explain the logic in your code. The threshold determines the yes/no assignment. For the dissertation eligibility case study, the threshold is 5.5 CGPA — this comes from the university's own eligibility rule, not from data analysis.

Binarization reduces any feature to a yes/no question. The threshold is everything — it must come from domain knowledge, not from convenience.

Real-World & Domain Connection: Medical screening uses binarization extensively: blood pressure above 140/90 → "hypertensive" (1), below → "normal" (0). The thresholds come from clinical guidelines based on population studies. In spam detection, the final output is inherently binary (spam/not-spam). In the dissertation case study, the target variable itself — "eligible" (CGPA ≥ 5.5) — is a binarized version of CGPA.

3.8 Data Transformation — Summary of Techniques Covered

The data transformation techniques discussed form a toolkit: 1. Discretization (Binning): Continuous → discrete. Equi-width (equal ranges) or equi-depth (equal counts). 2. Encoding: Categorical → numerical. One-hot (nominal, few categories) or label (ordinal). 3. Binarization: Anything → exactly two categories (0/1). Threshold from domain knowledge.

Practice assignment: A dataset (automobile dataset from Kaggle) is provided. Required tasks: data quality and cleaning, feature construction and transformation, scaling and normalization. A Python notebook demonstrating binning and discretization on this dataset was shared in the previous session.

3.9 Foundational ML Concepts — Hypothesis, Inductive Learning, Task Types

Hook: You show a child 10 pictures of cats. She points to a dog and says "not a cat." You never showed her a dog. How did she know?

Intuition + Analogy — Learning from Examples: A child sees examples of cats. From these specific examples, she forms a general idea of "cat-ness" — pointy ears, whiskers, tail, size. When shown a new animal, she compares it to this general idea. Machine learning works the same way: from specific training examples, the algorithm forms a general rule (the hypothesis), then applies it to new cases. The analogy breaks here: a child learns from a handful of examples; ML models often need thousands or millions.

3.9.1 The Hypothesis in Machine Learning

A hypothesis is a model or function that the learning algorithm learns from training data. The hypothesis is the model itself.

Fundamental assumption of ML: A hypothesis that performs well on a sufficiently large set of training examples will also perform well on unseen examples. This is the entire reason machine learning is useful. Analogy: Show a person enough pictures of cats. If they can correctly identify them, we assume they have learned the concept of "cat." When shown a new cat picture they have never seen, they will still identify it correctly. The person is the hypothesis/model. We learn a general rule from specific examples.

3.9.2 Inductive Learning

Inductive learning is the process of reasoning from specific observations to general rules. Given examples of a function

(X, f (X))

— input features

X

and target

f (X)

— the goal is to predict

f (X)

for new examples of

X

- $X$ : input features (the training data) - $f (X)$ : the target function — the real-world rule you are trying to learn - The model learns from $(X, f (X))$ pairs and predicts $f (X)$ for new $X$

3.9.3 Classification, Regression, and Probability Estimation

Task Type	What $f (X)$ predicts	Example
-----------	----------------------	---------
Classification	A category or label (discrete)	Spam vs. non-spam; cat vs. dog vs. bird; will customer buy or not?
Regression	A numerical value (continuous)	Tomorrow's temperature; selling price of a house; days until an event
Probability Estimation	A number between 0 and 1 (likelihood)	Probability user clicks an ad; chance of rain today

Worked Example — Task Identification

Given: sky, air temperature, altitude, wind, water, and forecast — predict humidity.

Humidity is a numerical value (e.g., 65%). This is a regression task.

Sense-check: The output is a continuous number, not a category and not a probability. Regression is the correct framing.

Assumptions & Scope:

- Assumption (Inductive Learning): The future will resemble the past. If the training data was collected in summer and predictions are needed for winter, the hypothesis may fail — the underlying distribution changed. - Assumption: The training examples are representative of the population you will predict on. A cat detector trained only on orange tabbies may fail on black cats.

Visual Intuition: Imagine a scatter plot. Classification draws a boundary line separating regions (spam on one side, not-spam on the other). Regression draws a line through the middle of the points. Probability estimation shades the plot — darker where the model is more confident.

Pitfalls:

1. Confusing classification and regression. "Predict the price" is regression. "Predict whether price will go up or down" is classification. The same underlying data can frame either task — choose based on what decision you need to make. 2. Ignoring the inductive learning assumption. Training on data from one distribution and testing on another is the most common cause of ML project failure. Always ask: "Will my training data look like my production data?"

A hypothesis is what the model learns — a general rule from specific examples. Inductive learning is the process of going from examples to rules. Classification predicts categories; regression predicts numbers; probability estimation predicts likelihoods.

Real-World & Domain Connection: The inductive learning framework underpins all of supervised machine learning. In the dissertation eligibility case study:

X

= student scores from semesters 1 and 2;

f (X)

= the true eligibility rule (CGPA ≥ 5.5); the model learns an approximation of

f

from historical student data and predicts eligibility for current students.

3.10 Linear Regression — Core Concepts

Hook: You have data points scattered across a page. You need one straight line that summarizes them all — and predicts where the next point will land. Which line do you draw?

Intuition + Analogy — Best-Fit Line Through a Scatter of Points: Imagine throwing darts at a dartboard. The darts scatter. If someone asked you to summarize where the darts landed with a single straight line, you would draw a line through the middle of the cluster — not through the outliers, not hugging any single dart, but balancing all of them. Linear regression is the mathematical way to find that "best summary line." The analogy breaks here: darts have no input-output relationship; in regression, the x-axis is the input (house size) and the y-axis is the output (price) — the line captures how y changes with x.

Linear regression is a method to find a straight-line relationship between input features and a target variable. Given data points $(X_{1}, Y_{1}), (X_{2}, Y_{2}), \dots$ , learn a function $f$ to predict $Y$ given $X$ , where $Y$ is real-valued.

Classic example — House price prediction: Given the size of a house (square feet), predict its price. Collected data points might be (500 sq ft, $200K), (600 sq ft, $200K), etc. The goal: draw a single straight line that best fits (best summarizes) these points. Then, for a new size (e.g., 1800 sq ft), use the line to predict the price (~$280K).

3.10.1 The Hypothesis Function for Linear Regression

From school mathematics, the equation of a straight line is: $y = m x + b$ where $m$ is the slope and $b$ is the y-intercept (the value of $y$ when $x = 0$ , where the line crosses the y-axis).

In machine learning notation, this is written as: $h_{θ} (x) = θ_{0} + θ_{1} x$

Symbol registry — Simple Linear Regression:

- $h_{θ} (x)$ — the hypothesis function (the model's prediction, also written as $y^$ ) - $x$ — the input feature (e.g., size of house in square feet) - $θ_{0}$ — the y-intercept (bias term). Conceptually: the base price of a house in a given locality — the price when size = 0. - $θ_{1}$ — the slope (weight). Conceptually: for every one-unit increase in $x$ , how much does the predicted $y$ go up or down?

The model makes predictions by calculating a weighted sum of the input features plus a bias.

Worked Example — House Price Prediction

Model: $h_{θ} (x) = 50000 + 150 \times x$

- $θ_{0} = 50000$ : A zero-square-foot house "costs" $50,000 (the land value, or the baseline). - $θ_{1} = 150$ : Each extra square foot adds $150 to the price.

Prediction for 1800 sq ft: $h_{θ} (1800) = 50000 + 150 \times 1800 = 50000 + 270000 = 320000$

Predicted price: $320,000. Sense-check: $150/sq ft × 1800 sq ft = $270,000 contribution from size, plus $50,000 base = $320,000. The numbers are internally consistent.

3.10.2 The Goal of Linear Regression

Find the best possible values of $θ_{0}$ and $θ_{1}$ that make the line fit the data as closely as possible. Among all possible lines (different slopes, different intercepts), which one best explains the data?

Training flow:

1. Start with a training set: pairs of (size, price). 2. Feed the training set into a learning algorithm. 3. The learning algorithm outputs a model: $h_{θ} (x) = θ_{0} + θ_{1} x$ . 4. This model can then predict prices for new house sizes.

Assumptions & Scope:

- Assumption (Linearity): The relationship between $x$ and $y$ is roughly linear. If the true relationship is a curve, a straight line will systematically underpredict in some regions and overpredict in others. - Assumption (IID): Training examples are independent and identically distributed — each house sale is independent of others, and all come from the same underlying price distribution. - Scope: Simple linear regression uses exactly one feature. For multiple features, use multiple linear regression. For curves, use polynomial regression (still linear in parameters).

Visual Intuition: Plot house size on the x-axis (0 to 3000 sq ft) and price on the y-axis ($0 to $500K). The data points form a loose upward-sloping cloud. The regression line cuts through the middle of this cloud. The vertical distance from each point to the line is the error for that house. The best line minimizes the sum of these squared vertical distances.

Pitfalls:

1. Interpreting $θ_{0}$ literally. A house with 0 sq ft doesn't exist. The intercept is often just a mathematical artifact that positions the line correctly within the range of observed data. Don't say "a zero-square-foot house costs $50,000" as a meaningful statement. 2. Extrapolating beyond the data. Your model was trained on houses from 500–3000 sq ft. Predicting the price of a 10,000 sq ft mansion assumes the same $150/sq ft rate holds — it probably doesn't. 3. Ignoring the units. If $x$ is in square feet and $y$ is in dollars, $θ_{1}$ is in dollars per square foot. Always track units — they catch dimensional errors.

Linear regression finds the straight line $h_{θ} (x) = θ_{0} + θ_{1} x$ that best summarizes the relationship between input $x$ and output $y$ . The parameters $θ_{0}$ (intercept) and $θ_{1}$ (slope) are what the model learns from data.

Real-World & Domain Connection: Linear regression is the most widely used statistical method in industry. It is used for demand forecasting (predict sales from advertising spend), real estate valuation (Zillow's Zestimate started as a linear regression), and medical dosage (predict drug concentration from patient weight). Its simplicity, interpretability, and mathematical tractability make it the first model tried in most regression problems — and often the benchmark that more complex models must beat.

3.11 Simple, Multiple, and Polynomial Regression

Hook: A house's price depends on more than just its size. It depends on bedrooms, bathrooms, age, location, school district... How do you fit a straight line through a cloud of points in 10 dimensions?

Intuition + Analogy — From Ruler to Measuring Tape to Contour Map: Simple linear regression is a ruler — one dimension, one measurement. Multiple linear regression is a measuring tape that accounts for length, width, and depth. Polynomial regression is a contour map — it captures hills and valleys (curves) while still being built from the same measuring tools. All three are "linear" in the same sense: the parameters appear as simple multipliers, never inside exponentials or denominators.

3.11.1 Simple Linear Regression

Uses exactly one input feature: $h_{θ} (x) = θ_{0} + θ_{1} x$

3.11.2 Multiple Linear Regression

Uses more than one input feature. For a house, features might include: size, number of bedrooms, age of house.

$h_{θ} (x) = θ_{0} + θ_{1} x_{1} + θ_{2} x_{2} + \dots + θ_{d} x_{d}$

In summation notation: $h_{θ} (x) = j = 0 \sum d θ_{j} x_{j}$

Convention:

x_{0} = 1

(there is no actual feature

x_{0}

; this is a notational trick so that

θ_{0} x_{0} = θ_{0}

, absorbing the intercept into the summation).

3.11.3 Vectorized Form

The summation can be written compactly as: $h_{θ} (x) = θ^{⊤} x$

Where: - $θ = [θ_{0}, θ_{1}, θ_{2}, \dots, θ_{d}]^{⊤}$ — a column vector of all parameters (size $d + 1 \times 1$ ) - $x = [x_{0}, x_{1}, x_{2}, \dots, x_{d}]^{⊤}$ — a column vector of all features, with $x_{0} = 1$ (size $d + 1 \times 1$ ) - $θ^{⊤}$ — the transpose of $θ$ (size $1 \times (d + 1)$ ) - The product $θ^{⊤} x$ is a $1 \times 1$ scalar — a single number, the weighted sum

This is the vectorized form of the hypothesis. The model is a weighted sum of input features.

3.11.4 Polynomial Regression

When data follows a curve rather than a straight line, polynomial regression adds powers of $x$ as features: $h_{θ} (x) = θ_{0} + θ_{1} x + θ_{2} x^{2} + θ_{3} x^{3} + \dots$

Polynomial regression is still considered a linear regression because the equation is linear in the parameters $θ_{j}$ (each $θ_{j}$ appears with power 1, just multiplied by a transformed feature).

3.11.5 Nonlinear Regression

In true nonlinear regression, the model equation is not linear in its parameters. Example: $y = \frac{θ _{0}}{1 + e ^{- θ_{1} x}}$ This form appears in logistic regression. Here, the parameters $θ_{0}, θ_{1}$ do not appear as simple multipliers — the equation is nonlinear in the parameters.

Key distinction: "Nonlinear" visually may suggest a curve, but the formal definition is about linearity in parameters. Polynomial regression produces curves but is linear in parameters, so it is a type of linear regression.

Worked Example — Multiple Linear Regression for House Price

Features: $x_{1}$ = size (sq ft), $x_{2}$ = bedrooms, $x_{3}$ = age (years)

Model: $h_{θ} (x) = 50000 + 120 x_{1} + 8000 x_{2} - 1000 x_{3}$

Prediction for a 2000 sq ft, 3-bedroom, 10-year-old house: $h_{θ} (x) = 50000 + 120 (2000) + 8000 (3) - 1000 (10)$ $h_{θ} (x) = 50000 + 240000 + 24000 - 10000 = 304000$

Predicted price: $304,000. Sense-check: Size contributes $240K, bedrooms add $24K, age subtracts $10K, base is $50K. Each feature's contribution is interpretable — this is a key advantage of linear models.

Assumptions & Scope:

- Assumption (Multiple): Features are not perfectly collinear. If $x_{2} = 2 \times x_{1}$ always, the model cannot uniquely determine $θ_{1}$ and $θ_{2}$ . This is called multicollinearity. - Assumption (Polynomial): The curve is well-approximated by a polynomial of the chosen degree. Degree too low → underfit. Degree too high → overfit (the curve wiggles to hit every training point). - Scope: The vectorized form $h_{θ} (x) = θ^{⊤} x$ is the foundation for all linear models, including logistic regression and SVM.

Visual Intuition: Simple regression is a line on a 2D plot. Multiple regression is a plane (or hyperplane) in (d+1)-dimensional space — impossible to visualize directly for d > 2, but mathematically the same idea: a flat surface cutting through a cloud of points. Polynomial regression is a curve on a 2D plot that is actually a straight line in a higher-dimensional space where

x^{2}, x^{3}

are treated as separate dimensions.

Pitfalls:

1. Thinking polynomial = nonlinear model. Polynomial regression is linear in parameters. The "linear" in linear regression refers to parameters, not the shape of the curve. 2. Too many polynomial degrees. $x^{10}$ terms explode numerically. A house of 2000 sq ft raised to the 10th power is $1.024 \times 1 0^{33}$ — a number that causes floating-point overflow. Always scale features before polynomial expansion. 3. Interpreting coefficients in polynomial regression. In $h_{θ} (x) = θ_{0} + θ_{1} x + θ_{2} x^{2}$ , you cannot say "a one-unit increase in $x$ increases $y$ by $θ_{1}$ " because $x$ also appears in the $x^{2}$ term. The effect of $x$ depends on the current value of $x$ .

Simple regression uses one feature; multiple regression uses many; polynomial regression uses powers of features. All are linear in parameters — the "linear" means parameters appear as multipliers, not that the graph is a straight line.

Real-World & Domain Connection: Multiple linear regression is the workhorse of econometrics — predicting GDP from interest rates, unemployment, and inflation. Polynomial regression is used in physics and engineering where known laws produce polynomial relationships (e.g., distance traveled under constant acceleration:

d = v_{0} t + \frac{1}{2} a t^{2}

— quadratic in time, linear in parameters

v_{0}

and

a

3.12 Cost Function — Mean Squared Error

Hook: You draw a line through your data. Your colleague draws a different line. Who drew the better line? You need a number — a score — that tells you how wrong each line is.

Intuition + Analogy — The Archery Score: An archer shoots arrows at a target. The score is based on how far each arrow lands from the bullseye. You square the distances (so left-misses and right-misses both count as positive), sum them up, and average. Lower score = better archer. The cost function is the archery score for your regression line: each data point is an arrow, the line's prediction is the bullseye, and the cost is the average squared miss distance. The analogy breaks here: in archery, the bullseye is fixed; in regression, you can move the line (change

θ_{0}, θ_{1}

) to get a better score.

3.12.1 Why a Cost Function?

When you draw a line through data, you need a way to quantify how good or bad that line is. The cost function (also called loss function or error function) measures how wrong the model's predictions are. You want the cost to be as low as possible.

3.12.2 Motivating Example — Average Model vs. Regression Model

Setup: Predict tip amount for meals. Six meals with actual tip amounts shown as red dots. Baseline — Average Model: Predict $10 (the mean tip) for every meal, regardless of any feature.

Meal	Actual Tip	Predicted Tip	Residual (Actual − Predicted)	Residual²
------	-----------	---------------	-------------------------------	-----------
1	5	10	−5	25
2	17	10	+7	49
3	11	10	+1	1
4	8	10	−2	4
5	14	10	+4	16
6	5	10	−5	25

Sum of squared residuals = 120. With 6 meals: MSE = 120 / 6 = 20.

Linear Regression Model: Using total_bill as the input feature, the model is:

$tip = 0.1462 \times total_bill - 8.81$

(These coefficients are presented as given — they come from fitting a regression to a standard restaurant tips dataset. The negative intercept reflects the mathematical fit; in practice, tips are never negative for real bills.)

Plugging in the total_bill values produces predicted tips. Computing residuals, squaring them, summing, and dividing by 6 gives MSE ≈ 5.01.

Comparison: The regression model's MSE (≈5) is roughly 4× better (lower) than the average model's MSE (20). A model with lower cost is better. The regression model predicts tip amounts much more accurately. Sense-check: The average model ignores the bill amount entirely — it predicts $10 whether the bill is $5 or $50. The regression model uses the bill amount and gets 4× closer to the true tips. Using relevant features reduces error.

3.12.3 The Mean Squared Error Cost Function

The cost function for linear regression is:

$J (θ) = \frac{1}{2 n} i = 1 \sum n (h_{θ} (x^{(i)}) - y^{(i)})^{2}$

Symbol registry — Cost Function:

- $J (θ)$ — the cost function (a function of the parameters $θ$ ) - $n$ — total number of training examples - $h_{θ} (x^{(i)})$ — the predicted value for the $i$ -th example (also written $y^^{(i)}$ ) - $y^{(i)}$ — the actual (true) value for the $i$ -th example - $h_{θ} (x^{(i)}) - y^{(i)}$ — the residual (error) for the $i$ -th example - The square $(\cdot)^{2}$ — ensures positive and negative residuals do not cancel each other out - $\frac{1}{2 n}$ — averages the squared errors; the extra factor of $\frac{1}{2}$ is a mathematical convenience that cancels with the 2 from the derivative of the square during gradient descent

Why square the residuals? Without squaring, a −5 error and a +5 error would sum to 0, making the model appear perfect when it is not. Squaring makes all errors positive. Why the extra ½? When you take the derivative of

x^{2}

, you get

2 x

. The ½ cancels with this 2, making gradient descent calculations cleaner. It does not change the location of the minimum. Note on notation: The standard form in many textbooks (Bishop, PRML) writes the sum-of-squares error as

E_{D} (w) = \frac{1}{2} \sum_{n = 1}^{N} {t_{n} - w^{⊤} ϕ (x_{n})}^{2}

. The professor uses

J (θ)

with

\frac{1}{2 n}

— the extra

\frac{1}{n}

normalizes by the number of examples, making the cost interpretable as an average per-example error. Both forms have the same minimum; only the scale of the cost value differs.

3.12.4 Comparison — MSE vs. MAE

Property	MSE (Mean Squared Error)	MAE (Mean Absolute Error)
----------	--------------------------	---------------------------
Formula	$\frac{1}{n} \sum (y_{i} - y^_{i})^{2}$	$\frac{1}{n} \sum < t d > y_{i} - y^_{i} < / t d >< t d >$
Shape of cost curve	Smooth parabola (U-shaped)	V-shaped
Derivative at minimum	0 (well-defined)	Undefined (kink)
Penalty for large errors	Quadratic (error of 10 costs 100)	Linear (error of 10 costs 10)
Best for	When large errors are especially bad	When all errors are equally bad

Q: Instead of squaring, can we use absolute value (modulus) to handle negative residuals? That would also prevent cancellation. A: Yes, that is called Mean Absolute Error (MAE). However, MSE is preferred for several reasons:

1. Smoothness: Squared error creates a smooth parabola when plotted. Absolute error creates a V-shape. At the bottom of the V, the slope is undefined; at the bottom of the parabola, the slope is 0 (well-defined). This matters for gradient-based optimization — gradient descent needs a well-defined slope everywhere. 2. Penalizing large errors: Squaring makes large errors much more costly. A residual of 3 gives squared error 9; a residual of 10 gives squared error 100. With absolute error, they are 3 and 10. Squaring tells the model: "An error of 10 is not just 3× worse than an error of 3 — it is about 11× worse." This heavily penalizes outliers and large mistakes.

MSE is the most commonly used cost function for regression. Minimizing the SSE implicitly assumes that the random noise follows a normal distribution; minimizing MAE assumes a Laplacian distribution.

Q: If

J (θ) = 0

, what does that mean? A: It means perfect predictions — every predicted value equals the actual value. The line passes exactly through every data point. This almost never happens with real data. If it does, suspect overfitting (the model has memorized the training data) or data leakage (the target leaked into the features).

Assumptions & Scope:

- Assumption (Gaussian noise): Minimizing MSE is equivalent to maximum likelihood estimation under the assumption that errors are normally distributed with constant variance. If errors have heavy tails (many extreme outliers), MSE over-penalizes them and the model fits the outliers at the expense of the typical data. - Scope: MSE is the default cost function for regression. For classification, cross-entropy loss is used instead.

Visual Intuition: Plot the residual (error) on the x-axis and the cost on the y-axis. MSE is a parabola centered at 0 — symmetric, smooth, curving upward. MAE is a V centered at 0 — symmetric but with a sharp point at the bottom. The parabola's smooth bottom means gradient descent can slide all the way to the minimum; the V's sharp bottom means gradient descent might oscillate around it.

Pitfalls:

1. MSE is sensitive to outliers. A single data point with a huge error can dominate the cost and pull the entire regression line toward it. Always check for outliers before using MSE. If outliers are present, consider MAE or Huber loss. 2. Confusing MSE and RMSE. RMSE = $MSE$ . RMSE is in the same units as $y$ (dollars, not dollars²), making it more interpretable. But the minimum is at the same parameter values — optimizing MSE and RMSE gives the same model. 3. Forgetting the ½ in derivations. The ½ is there purely for mathematical convenience. If you drop it, the minimum doesn't move, but your gradient descent update rule will be off by a factor of 2.

The cost function $J (θ) = \frac{1}{2 n} \sum (h_{θ} (x^{(i)}) - y^{(i)})^{2}$ measures how wrong your line is. Squaring prevents cancellation and heavily penalizes large errors. The ½ is there to make the derivative clean. Lower cost = better line.

Real-World & Domain Connection: MSE is the default loss function in virtually every regression library (scikit-learn's LinearRegression, statsmodels OLS, TensorFlow/Keras mean_squared_error). In finance, however, the asymmetry of risk (losing money is worse than missing an opportunity) leads to custom loss functions that penalize underpredictions and overpredictions differently — MSE is symmetric and cannot capture this.

3.13 The Cost Function Curve — Convexity and Global Minimum

Hook: You are standing on a hillside in thick fog. You want to reach the lowest point in the valley. You can't see the whole landscape — you can only feel the slope under your feet. How do you know you have reached the true bottom and not just a small dip?

Intuition + Analogy — The Bowl: The cost function for linear regression is shaped like a bowl. Drop a marble anywhere on the inner surface of the bowl. It rolls down and settles at the single lowest point — the global minimum. There are no fake bottoms (local minima), no ridges that trap the marble halfway. This bowl shape is called convexity, and it is the property that makes linear regression mathematically guaranteed to find the best parameters. The analogy breaks here: a real bowl has friction that can stop the marble early; gradient descent has a learning rate that, if set wrong, can overshoot or oscillate.

3.13.1 Visualizing $J (h e t a)$ for a Simplified Model

To understand the shape of the cost function, simplify by setting $θ_{0} = 0$ . The model becomes: $h_{θ} (x) = θ_{1} x$

With three training examples: $(1, 1), (2, 2), (3, 3)$ .

When $θ_{1} = 1$ :

h_{θ} (x) = x

. Predictions: 1, 2, 3. All residuals = 0. So

J (θ_{1}) = 0

. When $θ_{1} = 0.5$ : Predictions: 0.5, 1.0, 1.5.

$J (0.5) = \frac{1}{2 \times 3} [(0.5 - 1)^{2} + (1.0 - 2)^{2} + (1.5 - 3)^{2}] = \frac{1}{6} [0.25 + 1 + 2.25] = \frac{3.5}{6} \approx 0.58$

When $θ_{1} = 0$ : Predictions: 0, 0, 0.

$J (0) = \frac{1}{6} [(0 - 1)^{2} + (0 - 2)^{2} + (0 - 3)^{2}] = \frac{1}{6} [1 + 4 + 9] = \frac{14}{6} \approx 2.33$

When $θ_{1} = - 0.5$ : Predictions: −0.5, −1.0, −1.5.

$J (- 0.5) = \frac{1}{6} [(- 0.5 - 1)^{2} + (- 1.0 - 2)^{2} + (- 1.5 - 3)^{2}] = \frac{1}{6} [2.25 + 9 + 20.25] = \frac{31.5}{6} \approx 5.25$

Plotting $θ_{1}$ on the x-axis and $J (θ_{1})$ on the y-axis produces a U-shaped parabola. This is the cost function curve.

3.13.2 Convexity

The cost function curve for linear regression is convex. A function is convex if, when you take any two points on the curve and draw a straight line connecting them, the curve lies entirely below that line.

Why convexity matters: A convex function has exactly one minimum — a single global minimum. There are no local minima (dips that are low but not the lowest). This means optimization will always find the true best parameters if it reaches the bottom. The bowl analogy: The cost function surface is like a bowl. If you drop a ball anywhere on the inner surface of the bowl, it will roll down and settle at the single lowest point. This is the intuition behind gradient descent — starting from any initial parameter values, following the slope downward will eventually reach the global minimum.

3.13.3 The Full 3D Cost Function Surface

When both $θ_{0}$ and $θ_{1}$ are considered, the cost function $J (θ_{0}, θ_{1})$ forms a 3D bowl-shaped surface: - x-axis: $θ_{1}$ (slope) - y-axis: $θ_{0}$ (intercept) - z-axis: $J (θ_{0}, θ_{1})$ (cost)

There is a single lowest point — the global minimum.

3.13.4 Visualizing from Above — Contour Plot

Looking at the 3D bowl from directly above (like viewing a mountain from a helicopter) produces a contour plot: concentric ellipses. Each ellipse represents a constant value of $J (θ)$ . The smallest, innermost ellipse is the global minimum — the point with the lowest cost. As ellipses grow larger, the cost increases.

The goal of linear regression: Find the exact coordinates

(θ_{0}, θ_{1})

corresponding to the center of the smallest ellipse — the bottom of the bowl.

Worked Example — Reading a Contour Plot

Imagine a contour plot with $θ_{0}$ on the horizontal axis (range: −2 to 4) and $θ_{1}$ on the vertical axis (range: −1 to 3). Concentric ellipses are centered at about $(θ_{0}, θ_{1}) = (1, 1.5)$ .

- The outermost ellipse might be labeled $J = 50$ — any $(θ_{0}, θ_{1})$ on this ellipse gives cost 50. - The next ellipse inward: $J = 20$ . - The next: $J = 5$ . - The innermost dot: $J \approx 0.58$ — the global minimum.

Takeaway: The center of the smallest ellipse is where you want to be. Gradient descent starts at some random

(θ_{0}, θ_{1})

on the outer ellipse and takes steps perpendicular to the contour lines, moving inward toward the center. Sense-check: The contour lines are ellipses, not circles, because

θ_{0}

and

θ_{1}

may have different scales. If the features are standardized, the contours become more circular.

Assumptions & Scope:

- Assumption (Convexity): The MSE cost function for linear regression is always convex. This is a mathematical guarantee — it follows from the fact that MSE is a quadratic function of the parameters. For other models (neural networks), the cost function is non-convex with many local minima. - Scope: Convexity guarantees that gradient descent will find the global minimum — if it converges. The learning rate must be chosen appropriately; too large and it diverges, too small and it takes forever.

Visual Intuition: The 3D bowl: hold your hands together, palms up, forming a bowl shape. The deepest point of your palms is the global minimum. Now flatten your hands and look at them from above — the creases form concentric patterns. That is the contour plot. The takeaway: one unique bottom, reachable from any starting point by following the downward slope.

Q: How do you visualize the 3D cost surface? A: Think of it as a bowl. Then flatten it — imagine spreading the bowl flat on a table. From above, it looks like concentric ellipses (a contour map). The smallest ellipse at the center is the minimum cost point. This is the same as viewing a mountain from a helicopter — the peak (or in this case, the valley bottom) appears as the innermost contour ring.

Pitfalls:

1. Assuming all cost functions are convex. Only some are. Neural network loss surfaces are notoriously non-convex with many local minima. The convexity of linear regression's MSE is a special, valuable property. 2. Confusing the cost function curve with the regression line. The regression line is $y$ vs. $x$ . The cost function curve is $J (θ)$ vs. $θ$ . They are completely different plots. Students frequently mix them up. 3. Thinking the global minimum means zero error. The global minimum is the lowest possible cost given your model and data — it is rarely zero. Zero cost means the line passes through every point exactly, which only happens if the data is perfectly linear with no noise.

The MSE cost function is a convex bowl with exactly one global minimum and no local minima. The 3D bowl becomes concentric ellipses when viewed from above (contour plot). The goal of training is to find the $(θ_{0}, θ_{1})$ at the center of the smallest ellipse — the bottom of the bowl.

Real-World & Domain Connection: Convex optimization is a entire subfield of applied mathematics. The convexity of linear regression and logistic regression is why these models are so reliable — you always get the same answer from the same data. In contrast, training a neural network three times from different random initializations can give three different models with three different accuracies. When reliability and reproducibility matter (medical diagnosis, credit decisions), convex models are preferred.

3.14 Summary — What Must Be Clear Before the Next Session

The following concepts must be thoroughly understood before proceeding to the closed-form solution and gradient descent:

1. What is linear regression? — A method to find a straight-line relationship between features and target. 2. What is the hypothesis function? — $h_{θ} (x) = θ_{0} + θ_{1} x$ (simple) or $h_{θ} (x) = θ^{⊤} x$ (multiple). 3. What is the cost function? — $J (θ) = \frac{1}{2 n} \sum_{i = 1}^{n} (h_{θ} (x^{(i)}) - y^{(i)})^{2}$ , the mean squared error. 4. What are the parameters? — $θ_{0}, θ_{1}, \dots, θ_{d}$ , the weights the model learns. 5. Why is the cost function curve convex? — It is a parabola/bowl shape with a single global minimum and no local minima. 6. How to visualize the cost function in 3D and 2D (contour)? — Bowl shape in 3D; concentric ellipses in 2D contour view. The smallest ellipse is the minimum.

The closed-form solution (normal equations) and gradient descent will be covered in the next session. These foundational concepts will not be repeated — students must review and come prepared.

3.15 Exam Guidance Summary

Exam note: The closed-form solution for linear regression is described as "very important." Gradient descent will be covered in a subsequent (buffer/makeup) session. The concepts covered in this session (hypothesis function, cost function, convexity, global minimum) are foundational — the next session will not repeat them. Students must review and come prepared. Exam note: A practice assignment on data preprocessing (using the automobile dataset from Kaggle) is assigned. Required tasks: data quality and cleaning, feature construction and transformation, scaling and normalization. Lab sessions are available with lab sheets on data preprocessing. Students should practice in the virtual machine.

3.16 Key Industry Applications

- E-commerce: Feature construction — creating average_orders_per_month from total_orders and customer_since_date. Churn prediction models rely heavily on such engineered features. - Real estate: Feature construction — creating rooms_per_person from total_rooms and total_population. House price prediction is the classic linear regression example, used by Zillow, Redfin, and every real estate platform. - Medical diagnosis: Feature crossing — combining diabetes test results and blood pressure readings for disease prediction. Binarization of lab values against clinical thresholds is standard practice. - Salary prediction: Feature crossing — combining city and job type to predict compensation (Finance in New York vs. Finance in London). Glassdoor and LinkedIn Salary use such crossed features. - Restaurant tips: The tips dataset (total_bill → tip) is a standard pedagogical example for linear regression and cost function comparison, built into seaborn and scikit-learn. - Algorithms preferring discrete features: Bayesian algorithms, decision trees, ensemble learning, random forest, minimum distance classifiers, KNN — all benefit from discretization of continuous features. - Credit scoring: Discretization of income into brackets; binarization of "defaulted" as the target variable. Linear regression coefficients are directly interpretable as "impact on credit score." - Genomics: PCA on thousands of gene expressions to find population structure; feature selection to identify disease-associated genes from high-dimensional microarray data.

ML Lecture 3 notes · Feature Engineering and Linear Regression

Machine Learning· postgraduate· 2026-06-27

Summary

Comprehensive lecture notes covering the complete feature engineering toolkit: extraction (PCA), selection (filter, wrapper, embedded methods), construction (polynomial expansion, feature crossing), and transformation (discretization, encoding, binarization). Then builds the foundations of linear regression: the hypothesis function, simple/multiple/polynomial forms, the mean squared error cost function, and the convexity property that guarantees a unique global minimum. Every concept includes fully worked examples, domain connections, and exam-focused guidance.

Learning Objectives

1Understand the four tasks of feature engineering: extraction, selection, construction, transformation

2Apply PCA for dimensionality reduction and explain why standardization is essential

3Differentiate filter, wrapper, and embedded feature selection methods

4Construct new features via polynomial expansion and feature crossing

5Apply equi-width and equi-depth discretization appropriately based on data distribution

6Choose between one-hot and label encoding based on whether categories are nominal or ordinal

7Apply binarization with domain-knowledge thresholds

8Define the hypothesis function for simple, multiple, and polynomial linear regression

9Compute and interpret the mean squared error cost function

10Explain why the MSE cost function is convex and why convexity matters for optimization

Sections Breakdown

13.1 Feature Engineering — Overview

Covers 3.1. feature engineering — overview.

23.2 Feature Extraction

Covers 3.2. feature extraction.

33.3 Feature Selection

Covers 3.3. feature selection.

43.4 Feature Construction

Covers 3.4. feature construction.

53.5 Feature Transformation — Discretization (Binning)

Covers 3.5. feature transformation — discretization (binning).

63.6 Feature Transformation — Encoding

Covers 3.6. feature transformation — encoding.

73.7 Feature Transformation — Binarization

Covers 3.7. feature transformation — binarization.

83.8 Data Transformation — Summary of Techniques Covered

Covers 3.8. data transformation — summary of techniques covered.

93.9 Foundational ML Concepts — Hypothesis, Inductive Learning, Task Types

Covers 3.9. foundational ml concepts — hypothesis, inductive learning, task types.

103.10 Linear Regression — Core Concepts

Covers 3.10. linear regression — core concepts.

113.11 Simple, Multiple, and Polynomial Regression

Covers 3.11. simple, multiple, and polynomial regression.

123.12 Cost Function — Mean Squared Error

Covers 3.12. cost function — mean squared error.

133.13 The Cost Function Curve — Convexity and Global Minimum

Covers 3.13. the cost function curve — convexity and global minimum.

143.14 Summary — What Must Be Clear Before the Next Session

Covers 3.14. summary — what must be clear before the next session.

153.15 Exam Guidance Summary

Covers 3.15. exam guidance summary.

163.16 Key Industry Applications

Covers 3.16. key industry applications.

Postgraduate ML students — second lecture covering feature engineering techniques and linear regression fundamentals

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

3.1 Feature Engineering — Overview

Must-know: Feature engineering includes four tasks: extraction (dimensionality reduction), selection (removing irrelevant/redundant features), construction (creating new features from existing ones), and transformation (discretization, encoding, binarization). Domain knowledge is the most important ingredient in every task.

$No single formula — feature engineering is a process, not an equation.$

⚠️ Top pitfall: Applying all four tasks blindly without domain knowledge. Each technique must be applied with a clear reason tied to the data's meaning.

Self-check: What are the four tasks of feature engineering and how does each differ?

Connects to: 3.2 Feature Extraction — PCA, 3.3 Feature Selection — Filter, 3.3 Feature Selection — Filter, Wrapper, Embedded, 3.4 Feature Construction — Polynomial Expansion & Feature Crossing, 3.5 Discretization — Equi-Width vs. Equi-Depth Binning, 3.6 Encoding — One-Hot vs. Label Encoding, 3.7 Binarization

3.2 Feature Extraction — PCA

Must-know: PCA creates new features (principal components) as weighted combinations of original numerical features. PCs are ordered by variance explained. Dimensionality reduction happens by keeping only the top k PCs. Always standardize before PCA.

$PC_{1} = w_{1} x_{1} + w_{2} x_{2} + \dots + w_{d} x_{d}$

⚠️ Top pitfall: Forgetting to standardize before PCA — a feature with larger numeric range dominates the components.

Self-check: If you have 10 original features, how many principal components does PCA create? Where does dimensionality reduction actually happen?

Connects to: 3.1 Feature Engineering — Overview, 3.3 Feature Selection — Filter, 3.3 Feature Selection — Filter, Wrapper, Embedded, 3.13 Cost Function Convexity & Global Minimum

3.3 Feature Selection — Filter, Wrapper, Embedded

Must-know: Filter methods use statistical measures (e.g., correlation) before modeling. Wrapper methods use the ML model itself to evaluate subsets (forward selection, backward elimination). Embedded methods perform selection during training (e.g., LASSO, decision trees).

$Forward selection: start with 1 feature, add if accuracy improves. Backward elimination: start with all, remove if accuracy improves.$

⚠️ Top pitfall: Dropping features based on low individual correlation — a feature may be key in combination with another (interaction effect). Filter methods miss this.

Self-check: What is the key difference between filter, wrapper, and embedded feature selection methods?

Connects to: 3.1 Feature Engineering — Overview, 3.2 Feature Extraction — PCA, 3.10 Linear Regression — Hypothesis Function

3.4 Feature Construction — Polynomial Expansion & Feature Crossing

Must-know: Polynomial expansion adds powers of a feature (x², x³) to capture curves while keeping the model linear in parameters. Feature crossing combines two or more features to capture interaction effects (e.g., city × job type).

$h_{θ} (x) = θ_{0} + θ_{1} x + θ_{2} x^{2}$

⚠️ Top pitfall: Polynomial degree too high — x⁵ terms fit training data perfectly but fail catastrophically on new data. Start with degree 2.

Self-check: Why is polynomial regression still considered 'linear' regression?

Connects to: 3.1 Feature Engineering — Overview, 3.10 Linear Regression — Hypothesis Function, 3.11 Simple, 3.11 Simple, Multiple & Polynomial Regression

3.5 Discretization — Equi-Width vs. Equi-Depth Binning

Must-know: Equi-width: all bins have same numerical width (max−min)/K. Equi-depth: all bins have about the same number of data points. Use equi-depth for skewed data; equi-width for uniform data.

$Equi-width: width = \frac{max - min}{K}$

⚠️ Top pitfall: Using equi-width on skewed data produces empty or near-empty bins. Always check the histogram before choosing a binning strategy.

Self-check: When would you choose equi-depth binning over equi-width binning?

Connects to: 3.6 Encoding — One-Hot vs. Label Encoding, 3.7 Binarization

3.6 Encoding — One-Hot vs. Label Encoding

Must-know: One-hot encoding: create k binary columns for k categories (use for nominal data). Label encoding: assign integers 0,1,2,... (use ONLY for ordinal data with natural order). Never label-encode nominal categories.

$One-hot: color \in {red, green, blue} \to color_red, color_green, color_blue \in {0, 1}$

⚠️ Top pitfall: Label encoding nominal data — the model learns 'red > green > blue' as a meaningful ordering when none exists.

Self-check: For a 'city' column with values {Mumbai, Delhi, Bangalore}, which encoding should you use and why?

Connects to: 3.5 Discretization — Equi-Width vs. Equi-Depth Binning, 3.7 Binarization

3.7 Binarization

Must-know: Binarization converts any feature to exactly two categories (0/1) using a domain-knowledge threshold. It is the most extreme form of discretization. The threshold must come from the problem domain, not from convenience.

$binarized (x) = {10 if x > threshold otherwise$

⚠️ Top pitfall: Arbitrary threshold — 'I'll just use the median' is not domain knowledge. The threshold should come from the problem: a legal age, a passing grade, a medical cutoff.

Self-check: How is binarization different from one-hot encoding?

Connects to: 3.5 Discretization — Equi-Width vs. Equi-Depth Binning, 3.6 Encoding — One-Hot vs. Label Encoding

3.9 Hypothesis, Inductive Learning & Task Types

Must-know: A hypothesis is the model/function learned from training data. Inductive learning reasons from specific examples to general rules. Classification predicts categories; regression predicts continuous values; probability estimation predicts likelihoods.

$h_{θ} (x) \approx f (x) (hypothesis approximates the true target function)$

⚠️ Top pitfall: Confusing classification and regression — 'predict the price' is regression; 'predict whether price will go up or down' is classification.

Self-check: Given features (sky, temperature, altitude, wind) to predict humidity — is this classification, regression, or probability estimation?

Connects to: 3.10 Linear Regression — Hypothesis Function, 3.12 Cost Function — Mean Squared Error

3.10 Linear Regression — Hypothesis Function

Must-know: The hypothesis for simple linear regression is h_θ(x) = θ₀ + θ₁x. θ₀ is the intercept (bias), θ₁ is the slope (weight). The model predicts by computing a weighted sum of input features plus a bias.

$h_{θ} (x) = θ_{0} + θ_{1} x$

⚠️ Top pitfall: Interpreting θ₀ literally — a house with 0 sq ft doesn't exist. The intercept is often just a mathematical artifact positioning the line correctly within the observed data range.

Self-check: In h_θ(x) = 50000 + 150x for house price prediction, what does 150 represent?

Connects to: 3.11 Simple, 3.11 Simple, Multiple & Polynomial Regression, 3.12 Cost Function — Mean Squared Error

3.11 Simple, Multiple & Polynomial Regression

Must-know: Simple: one feature. Multiple: h_θ(x) = θᵀx (vectorized form with x₀=1). Polynomial: adds powers of x as features but remains linear in parameters. 'Linear' refers to linearity in parameters, not the shape of the curve.

$h_{θ} (x) = θ^{⊤} x = j = 0 \sum d θ_{j} x_{j} (x_{0} = 1)$

⚠️ Top pitfall: Thinking polynomial regression is a 'nonlinear' model — it is linear in parameters. True nonlinear regression has parameters inside exponentials or denominators.

Self-check: Why is h_θ(x) = θ₀ + θ₁x + θ₂x² considered linear regression?

Connects to: 3.10 Linear Regression — Hypothesis Function, 3.4 Feature Construction — Polynomial Expansion & Feature Crossing, 3.12 Cost Function — Mean Squared Error

3.12 Cost Function — Mean Squared Error

Must-know: J(θ) = (1/2n) Σ(h_θ(x⁽ⁱ⁾) − y⁽ⁱ⁾)² measures how wrong predictions are. Squaring prevents positive/negative cancellation and heavily penalizes large errors. The ½ factor cancels with the derivative's 2 during gradient descent.

$J (θ) = \frac{1}{2 n} i = 1 \sum n (h_{θ} (x^{(i)}) - y^{(i)})^{2}$

⚠️ Top pitfall: MSE is sensitive to outliers — a single data point with huge error can dominate the cost and pull the entire regression line toward it.

Self-check: Why do we square the residuals instead of using absolute values? Give two reasons.

Connects to: 3.10 Linear Regression — Hypothesis Function, 3.13 Cost Function Convexity & Global Minimum

3.13 Cost Function Convexity & Global Minimum

Must-know: The MSE cost function is convex (bowl-shaped) with exactly one global minimum and no local minima. This guarantees gradient descent will find the optimal parameters if it converges. The 3D bowl becomes concentric ellipses in a 2D contour plot.

$Convex: J (λa + (1 - λ) b) \leq λ J (a) + (1 - λ) J (b)$

⚠️ Top pitfall: Confusing the cost function curve (J(θ) vs θ) with the regression line (y vs x). They are completely different plots.

Self-check: Why is convexity of the cost function important for optimization?

Connects to: 3.12 Cost Function — Mean Squared Error, 3.10 Linear Regression — Hypothesis Function

Practice Quiz

Test your understanding of ML Lecture 3 notes. Select an answer for each question — results are instant.

Which feature engineering task creates entirely new features as weighted combinations of original features?

Why must you standardize data before applying PCA?

In the linear regression hypothesis h_θ(x) = θ₀ + θ₁x, what does θ₁ represent?

Why is the MSE cost function for linear regression convex?

When should you use label encoding instead of one-hot encoding?

Previous LectureLecture 02 · Data Preprocessing for Machine Learning — Lecture 2

Next LectureLecture 04 · Linear Regression — Complete Lecture Notes

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Feature Engineering and Linear Regression — Complete Lecture Notes

3.1 Feature Engineering — Overview

3.2 Feature Extraction

3.2.1 Principal Component Analysis (PCA) — Conceptual Overview

3.3 Feature Selection

3.3.1 Three Methods of Feature Selection

3.3.1.1 Filter Method

3.3.1.2 Wrapper Method

3.3.1.3 Embedded Method

3.4 Feature Construction

3.4.1 Polynomial Expansion

3.4.2 Feature Crossing

3.5 Feature Transformation — Discretization (Binning)

3.5.1 Equi-Width Binning

3.5.2 Equi-Depth Binning (Equi-Frequency Binning)

3.5.3 When to Use Which

3.6 Feature Transformation — Encoding

3.6.1 One-Hot Encoding

3.6.2 Label Encoding

3.7 Feature Transformation — Binarization

3.7.1 Binarization of Categorical Features

3.7.2 Binarization of Numerical Features

3.7.3 Distinguishing Binarization from One-Hot Encoding

3.8 Data Transformation — Summary of Techniques Covered

3.9 Foundational ML Concepts — Hypothesis, Inductive Learning, Task Types

3.9.1 The Hypothesis in Machine Learning

3.9.2 Inductive Learning

3.9.3 Classification, Regression, and Probability Estimation

3.10 Linear Regression — Core Concepts

3.10.1 The Hypothesis Function for Linear Regression

3.10.2 The Goal of Linear Regression

3.11 Simple, Multiple, and Polynomial Regression

3.11.1 Simple Linear Regression

3.11.2 Multiple Linear Regression

3.11.3 Vectorized Form

3.11.4 Polynomial Regression

3.11.5 Nonlinear Regression

3.12 Cost Function — Mean Squared Error

3.12.1 Why a Cost Function?

3.12.2 Motivating Example — Average Model vs. Regression Model

3.12.3 The Mean Squared Error Cost Function

3.12.4 Comparison — MSE vs. MAE

3.13 The Cost Function Curve — Convexity and Global Minimum

3.13.1 Visualizing J(heta) for a Simplified Model

3.13.2 Convexity

3.13.3 The Full 3D Cost Function Surface

3.13.4 Visualizing from Above — Contour Plot

3.14 Summary — What Must Be Clear Before the Next Session

3.15 Exam Guidance Summary

3.16 Key Industry Applications

ML Lecture 3 notes · Feature Engineering and Linear Regression

Summary

Learning Objectives

Sections Breakdown

Exam Revision Notes

3.1 Feature Engineering — Overview

3.2 Feature Extraction — PCA

3.3 Feature Selection — Filter, Wrapper, Embedded

3.4 Feature Construction — Polynomial Expansion & Feature Crossing

3.5 Discretization — Equi-Width vs. Equi-Depth Binning

3.6 Encoding — One-Hot vs. Label Encoding

3.7 Binarization

3.9 Hypothesis, Inductive Learning & Task Types

3.10 Linear Regression — Hypothesis Function

3.11 Simple, Multiple & Polynomial Regression

3.12 Cost Function — Mean Squared Error

3.13 Cost Function Convexity & Global Minimum

Practice Quiz

0 / 5

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat

3.13.1 Visualizing $J (h e t a)$ for a Simplified Model