—

Data Mining

Data Preprocessing: Noise, Integration, Transformation, and Reduction

Published: 2026-08-05

Level: Postgraduate

Audience: Postgraduate students in Data Mining

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Data Preprocessing and Data Quality — covered in Lecture 3 (the six quality parameters and why quality is subjective)
From Physical Entity to Digital Form — covered in Lectures 2 and 3 (how real-world objects become datasets)
Data Quality Problems: Noise, Outliers, Missing Values, Duplicates — covered in Lecture 3
Handling Missing Values — covered in Lecture 3 (global constants, local constants, central tendency, conditional probability)
Where Noise Comes From and Handling Noisy Data — covered in Lecture 3

4.1 Data Preprocessing: Objectives and Data Quality

Hook — a puzzle to start with. You spend hours cleaning a spreadsheet by hand, then build a model on it, and the results are still wrong. Why? Often it is not the algorithm that failed — it is the data that went in. This session is about the step that sits between raw data and mining: preprocessing. Get it right and everything downstream gets easier.

4.1.1 The Two Objectives

The two objectives of data preprocessing. Data preprocessing exists to serve two goals:

Improve the quality of the data — remove or reduce the errors, gaps, and inconsistencies that real-world data carries.
Modify the data so that it fits a specific data mining technique better — reshape values (scaling, transforming, aggregating) so an algorithm can consume them effectively.

A raw dataset, as it comes from the real world, is rarely in a shape that a mining algorithm can consume directly. Preprocessing is the bridge: raw data goes in, prepared data goes out, and only then does the mining stage begin.

Data quality itself is very hard to quantify. There is no single number that tells you a dataset is "good enough." Still, we have a set of measures we can look at. These six words are the vocabulary of data quality assessment:

Measure	Plain meaning
Accuracy	How correctly the recorded values match the true values.
Completeness	Whether all needed fields and records are present, or some are missing.
Consistency	Whether the same fact is recorded the same way everywhere (no contradictory entries).
Timeliness	Whether the data is current enough for the question being asked.
Believability	How much the users trust that the data reflects reality.
Interpretability	How easy it is to understand what each attribute actually means.

4.1.2 What Makes Data "Good"

The bottom line is simple: data has quality if it satisfies the requirement of the intended user. A single dataset can be excellent for one person and useless for another.

Suppose you have dataset A. It might be very useful for one person who is asking the right kind of question, and completely irrelevant for another person whose question does not match the information stored in the dataset. Quality is not an absolute property of the data; it depends on who is using it and what they are asking.

That is why the set of questions you plan to ask matters as much as the data itself. If you feed bad data into a mining algorithm, the algorithm will most likely give you poor results.

Garbage in, garbage out. The chain of consequences is worth stating directly: if you pass bad data to a data mining algorithm, you get bad results. No clever model can rescue data that was wrong from the start — the mining stage only works as well as the preprocessing that prepared its input.

4.1.3 The Four Data Quality Problems

Working with data quality means two phases. First you detect the problems in the dataset, then you solve them. The main problems fall into four families:

Noise — a modification in the original data; more precisely, a random error sitting inside your measured variable.
Outliers — legitimate data objects that are considerably different from the other objects in the dataset.
Missing values — fields in your table that are simply absent.
Duplicate data — redundant data appearing multiple times in the same dataset.

Notice the careful wording: noise is a random corruption of what was measured, while an outlier is a genuine object that simply stands out. The distinction matters, and we will return to it when we study outlier analysis.

4.1.4 From Physical Entity to Digital Form

This is very relevant in practice. Whenever we convert a physical entity into a digital form so that we can run a mining algorithm on it, it is extremely hard to map the physical entity to the digital form correctly.

Think about a bank that wants to model customers. The customer is a physical entity; the bank's table is the digital form. The table can carry only a chosen list of attributes, and a few of them may be impossible to gather — so the table is incomplete. Data can carry errors. Data can carry inconsistencies. All of this has to be fixed before mining, which is why we preprocess the data: to convert the raw data and prepare it for the next stages.

There are five methods of data preprocessing:

Data cleaning
Data integration
Data transformation
Data reduction
Data discretization

Data cleaning was started in the previous session, and it works on three fronts: filling in incomplete data, identifying and handling noisy data, and fixing inconsistent data. For missing values specifically, the fixes discussed earlier are:

Fill with a global constant (the same value for every gap in the column),
Fill with a local constant (a constant chosen per group or per attribute),
Fill with a measure of central tendency of the attribute (like the mean or median),
Fill based on conditional probability — the value most likely given the other attributes of the tuple.

Recap and bridge. Preprocessing has two objectives — better quality and better fit for the algorithm — and quality is measured relative to the user's question, not in the abstract. The four quality problems (noise, outliers, missing values, duplicates) are what the rest of this session attacks, one technique at a time. Next we look at the first problem in detail: where noise comes from.

Where this matters in practice. Nearly every industrial data pipeline runs on this idea: a hospital's patient records table, a bank's transaction log, an e-commerce product catalog — all are digital forms of physical entities, all incomplete, all noisy. The preprocessing choices made here decide whether the downstream model (fraud detection, diagnosis support, recommendation) is trustworthy or silently wrong.

4.2 Where Noise Comes From

Hook. Your thermometer reads 37.5 °C for a healthy patient. The true temperature was 36.9. No one lied, nothing broke catastrophically — yet the reading is wrong. This small, random corruption of a measured value has a name: noise. Knowing where it enters the data tells you how to fight it.

4.2.1 What Counts as Noise

Noise is the random error that sits in your measured variable. The word "random" matters: the error is not a pattern, and it is not a legitimate extreme value — it is just corruption in what was recorded. Understanding how noise enters the data helps you decide how to handle it, so let's go through the main sources.

4.2.2 Five Sources of Noise

A faulty instrument or sensor. The sensor you deployed to gather the data was itself faulty. It collected wrong information and reported that wrong information to the server, so the data points in your dataset were wrong from the start.

Data entry problems. Someone copied the data by hand or typed it manually and made a mistake. A human at a computer entering records is one of the most common noise sources in real datasets — a fat-fingered salary, a transposed date, a dropped digit.

Data transmission problems. Noise can be added while data travels. If you are talking on a phone in bad weather, noise enters the signal and there are jitters on the receiving end and on the sending end. The same physics applies when data moves from a device to a server over a lossy channel — the bits get corrupted in transit.

Professor's analogy — technology limits add noise without any mistake. GPS can measure position with precision of maybe centimeters today, which is fine for most needs. But suppose the technology was older and GPS could only measure with a precision in the range of 10 kilometres — the smallest distance it can resolve is 10 kilometres. If your requirement is centimeters, you hit a hard limit because of the technology. The same story holds for any sensor: a sensor measuring today's water level with a precision of meters cannot report anything finer — it cannot go below its own resolution. If you need centimeters or millimeters, you need a better sensor. Because of such technological limitations, noise enters your data even though nobody made a mistake.

Naming inconsistencies when merging. When you merge datasets, fields can be named or represented differently in each source, which quietly corrupts the merged result. (We will study this carefully in the data integration section.)

4.2.3 Two Strategies for Handling Noise

There are two broad strategies.

Identify the noise and fix it — find the bad points and remove or correct them. This is the most direct approach, and it is covered later in this session (outlier analysis and automated checks).
Accept that noise exists but reduce its impact — smooth it. With smoothing you do not remove the noisy points; you reduce the effect they have on the overall dataset so the overall impact of the noise comes down.

The main smoothing techniques we will look at are binning and linear regression.

Recap and bridge. Noise is random corruption of a measured value, and it enters data through five doors: faulty sensors, manual data entry, transmission loss, technology resolution limits, and sloppy merging. Against it you can fight directly (identify and fix) or absorb it (smooth). Next we learn the first smoothing tool — binning.

Where this matters in practice. Any system that reads the physical world produces noise: weather stations, fitness trackers, industrial pressure gauges, and GPS modules in delivery fleets. Engineers decide for each signal whether to fix (flag and reject bad readings) or smooth (average them away), and the choice drives the design of the whole data pipeline.

4.3 Smoothing by Binning

Hook. A retail mart records item prices by hand, and a few of them are far off — a packet that usually costs 15 rupees shows up as 8. Which value is the noisy one? Rather than guessing, binning avoids the question: it smooths the whole neighborhood of sorted values, so no single point has to be declared guilty.

4.3.1 The Three-Step Procedure

Binning is the classic smoothing technique, and it follows a fixed procedure.

Step 1: sort the data. Suppose you have the prices of different items in a retail mart, and the values were entered by hand so some of them are suspect. First you sort this array into ascending order, from the lowest value to the highest.

Step 2: partition the sorted data into equal-frequency bins. Equal frequency means you fix how many data points each bin will hold. For example, you might decide each bin has three data points. Bin one takes the first three points, bin two the next three, bin three the last three, and so on.

Step 3: smooth each bin by consulting its neighborhood. The idea is that if one point inside a bin is noisy, you use its neighbors in the same bin to reduce its effect. The smoothing itself can be done in a few ways: with the bin's mean, with the bin's median, or with the bin's boundary values.

The intuition: local smoothing. Because every replacement is decided using only the values inside the same bin, binning performs local smoothing — the outcome for a point depends on the handful of values around it, not on the whole dataset. Notice what we are not doing: we are not searching for which specific point inside a bin is the noisy one. We are only reducing the impact of whatever noise exists there. That is what makes binning fast and easy to apply.

4.3.2 Worked Example: Smoothing by Bin Means

Here is the lecture's dataset: the sorted prices of nine items, entered by hand so some values are suspect:

$4, 8, 15, 21, 21, 24, 25, 28, 34$

We fix equal-frequency bins of size 3, so the sorted values split as:

Bin B1 = {4, 8, 15}
Bin B2 = {21, 21, 24}
Bin B3 = {25, 28, 34}

Smoothing by bin means. Bin B1 contains 4, 8, 15. The mean of the bin is:

$mean (B 1) = \frac{4 + 8 + 15}{3} = \frac{27}{3} = 9$

Now we replace each value in the bin with the bin's mean. So bin B1 becomes 9, 9, 9 — the 9 takes the place of 4, the place of 8, and the place of 15.

Repeat for bin B2. Its mean is:

$mean (B 2) = \frac{21 + 21 + 24}{3} = \frac{66}{3} = 22$

So every value in B2 is replaced by 22 — the lecture stated this mean of 22, and the values 21, 21, 24 are exactly what the reference treatment uses for this example. Bin B2 becomes 22, 22, 22.

Bin B3 follows the same procedure:

$mean (B 3) = \frac{25 + 28 + 34}{3} = \frac{87}{3} = 29$

So bin B3 becomes 29, 29, 29.

The smoothed data is: 9, 9, 9, 22, 22, 22, 29, 29, 29.

Sense-check: every smoothed value is the center of its own bin, and the three blocks are ordered exactly as the original sorted data was — the smoothing reduced the spread inside each bin without disturbing the overall ordering.

Smoothing by bin medians works the same way, except each value is replaced by the bin's median instead of its mean. For B1 = {4, 8, 15} the median is 8, so bin B1 would become 8, 8, 8. Which one you pick — mean or median — is the same kind of choice we discuss below.

4.3.3 Worked Example: Smoothing by Bin Boundaries

Boundary smoothing works on the same sorted bins, but instead of replacing with a mean, we replace each value with its closest boundary — the minimum or the maximum of the bin.

Boundary smoothing for all three bins. For each value, compare the distance to the bin's minimum with the distance to the bin's maximum, and snap to the nearer edge.

Bin B1 = {4, 8, 15}: the minimum is 4 and the maximum is 15.

- 4: the distances are	4 − 4	= 0 to the minimum and	4 − 15	= 11 to the maximum. It is closest to the minimum, and since 4 is already the minimum, it stays unchanged.
- 8: the distances are	8 − 4	= 4 to the minimum and	8 − 15	= 7 to the maximum. Since 4 < 7, the value is closer to 4, so 8 is replaced by 4.

15: it is the maximum itself, so it stays unchanged.

Result: 4, 4, 15.

Bin B2 = {21, 21, 24}: the minimum is 21 and the maximum is 24. The first two values are already the minimum, and 24 is the maximum, so every value is on a boundary — nothing changes. Result: 21, 21, 24.

Bin B3 = {25, 28, 34}: the minimum is 25 and the maximum is 34.

25: already the minimum — unchanged.
28: the distances are |28 − 25| = 3 to the minimum and |28 − 34| = 6 to the maximum. Since 3 < 6, 28 is replaced by 25.
34: already the maximum — unchanged.

Result: 25, 25, 34.

Sense-check: the only values that moved are interior values, and each moved to the edge it was physically nearer to — exactly the "nearest boundary" rule. The smoothed sequence is 4, 4, 15, 21, 21, 24, 25, 25, 34.

That is boundary smoothing: every interior value snaps to whichever of the two edges of its bin it is nearer to.

4.3.4 Choosing Between Mean, Median, and Boundary

Q: When do we choose mean, median, or boundary? A: It depends totally on your dataset. You look into each bin, see which part looks noisy, and pick the smoothing that reduces its impact best. This choice is very domain specific — there are no statistical thumb rules that say, for example, that the mean is always better than the median. For certain cases the mean works well, for others the boundary works well, and the mode can also be used if it suits the data. What matters is that you know these tools exist, so that when you handle noise in your own data you can reach for the right one.

The same logic extends beyond binning: there are many other data cleaning techniques in the literature, and they are very subjective to the domain you are working in. If you know the toolkit, you can build better models; if you do not know it, it would be a disaster when you face a real problem. Gather as many data cleaning techniques as you can, then apply them based on domain knowledge and requirements.

Scope: when does binning apply, and where does it bend?

The procedure assumes you can choose a sensible bin size (here, 3). The wider the bin, the stronger the smoothing — a bin of size 5 changes the data far more than a bin of size 2. That power is also a risk: too-wide bins can swallow real structure in the data.
Bins need not be equal-frequency. Equal-width bins (fixed value ranges, e.g., 0–10, 10–20, 20–30) are an alternative, but the lecture's procedure — and the standard reference — uses equal-frequency bins.
Binning is not only a cleaning tool: it also acts as discretization — it replaces many distinct values with a few representative ones, which is itself a form of data reduction that helps methods like decision trees.

Pitfalls — the traps beginners fall into.

Hunting for the guilty point. Binning never tells you which value was noisy — it changes every value in the bin. Do not use it when you need to know exactly which record to fix.
Ignoring the bin-size choice. The result is completely different for bin size 2 versus 5; the choice is yours, and it should be driven by domain knowledge, not habit.
Forgetting that smoothing alters good values too. The genuine values in a bin are replaced just like the noisy ones — you are trading precision for robustness.
Mixing up mean and boundary behavior. Mean smoothing drags everything to the center of the bin; boundary smoothing pushes interior values to an edge. They answer different questions — choose deliberately.

Visual intuition. Plot the sorted prices as a bar chart: nine bars of heights 4, 8, 15, 21, 21, 24, 25, 28, 34. After mean smoothing, the chart shows three flat plateaus at 9, 22, and 29 — each bin collapsed to a single level. After boundary smoothing, the same chart shows the left and right edges of each bin preserved while the interior bars lean toward one edge. One glance at the two charts shows why boundary smoothing keeps more of the original spread.

Recap and bridge. Binning is a three-step local smoother: sort, cut into equal-frequency bins, then replace each bin's values by its mean, median, or nearest boundary. It reduces the impact of noise without ever identifying the noisy point. Next we meet the second smoothing strategy: linear regression, which uses a fitted line instead of bins.

Exam note: there is no fixed rule for choosing mean, median, or boundary — the choice depends on the dataset and the domain. Be ready to justify your choice in words, and to compute all three variants on a small sorted list.

Where this matters in practice. Binning is a standard step in retail and sensor analytics: hand-entered item prices, supermarket scan data, and IoT sensor logs all get binned before aggregation. Because it needs no labeled "noise" examples and costs almost nothing, it is often the first cleaner applied in production pipelines.

4.4 Smoothing by Linear Regression

Hook. Plot age on the x-axis and salary on the y-axis for a group of employees: the points drift upward from left to right — older people generally earn more. If a handful of points sit slightly off the pattern, are those people's salaries wrong? Probably not — but their recorded values carry small noise. A single line can capture the pattern and pull those noisy points back onto it.

4.4.1 The Idea Behind Regression Smoothing

The second smoothing technique is linear regression. Suppose you plot a two-dimensional dataset with an x axis and a y axis, and the points scatter upward from left to right: as the value of $x$ increases, the value of $y$ increases in roughly the same fashion. A simple real case is age and salary — in general terms, as your age increases, your salary also tends to increase. It is not always true, but it holds for the majority of people.

Whenever one variable increases while the other increases (or decreases) proportionally, you can write an equation that represents the data points. For this scatter, a line like:

$y = x + 1$

can represent all the data points. In this line, $x$ is the first attribute (age), $y$ is the second attribute (salary), the number multiplying $x$ (here 1) is the slope — how many units $y$ rises per unit of $x$ — and the "+1" is the intercept, the value of $y$ when $x = 0$ .

What linear regression does. You draw the line that overlaps most of the data points — the one with minimum error, running as close as possible to every point. Formally, linear regression finds the "best" line to fit two attributes so that one attribute can be used to predict the other; "best" means the line that minimizes the total error between the points and the line. The logic in full: if two variables are related, we can write a mathematical equation that captures the connection between them, and if we can, we can draw the line.

4.4.2 Worked Example: Age and Salary

Once the line is drawn, only a minor variation remains between each data point and the line. That minor variation is very likely coming from noise. So we project each point onto the line — map the point to the point on the line directly below or above it — and the projected value replaces the noisy measured value.

Projecting noisy points onto $y = x + 1$ . Suppose five employees give these (age, salary) pairs, where salary is in thousands:

Employee	Measured (age, salary)	Salary on the line $y = x + 1$
1	(20, 21.5)	$y = 20 + 1 = 21$
2	(25, 25.8)	$y = 25 + 1 = 26$
3	(30, 31.2)	$y = 30 + 1 = 31$
4	(35, 35.9)	$y = 35 + 1 = 36$
5	(40, 40.6)	$y = 40 + 1 = 41$

Take employee 2: the measured salary 25.8 sits above the line, with a residual of $25.8 - 26 = - 0.2$ . That 0.2-unit gap is the minor variation we attribute to noise. Projection replaces the measured value with the line's value: 26. Employee 4's salary 35.9 is projected down to 36, employee 3's 31.2 projects down to 31, and so on.

The smoothed dataset is: (20, 21), (25, 26), (30, 31), (35, 36), (40, 41) — every point now lies exactly on the line.

Sense-check: each projected point is the vertical shadow of the measured point on the line, so the whole set collapses onto one straight line and every residual becomes zero. The pattern (salary grows with age) is preserved; only the wobble around it is removed.

A point that sits exactly on the line has no noise; the rest of the points carry minor noise, and by mapping each of them onto the line that represents the true relationship between the two attributes, we reduce the noise in each point. So linear regression can be used to smooth the effect of noise, without ever identifying which point is noisy — the same "don't hunt the culprit" philosophy as binning.

Scope: when does regression smoothing apply?

The relationship between the two attributes must be roughly linear — one variable should increase (or decrease) proportionally with the other. A curved relationship needs a curve, not a line.
Regression smooths only the second attribute (the y-value). It assumes $x$ is measured reliably and the noise lives in $y$ ; if both variables carry noise, a single line fit is only an approximation.
For more than two attributes, linear regression generalizes to multiple linear regression, which fits a multidimensional surface instead of a line — outside this session's scope, but the same idea.

Pitfalls — the traps beginners fall into.

Drawing the line by eye through two points. The fitted line must be the one with minimum error over all points, not just the extremes. Extreme points that happen to lie far out can drag the line if they are outliers.
Projecting without checking the scale. If salary is recorded in rupees and age in years, the line's slope is meaningful only in those units — projection must use the same units as the fit.
Extrapolating beyond the data. The line $y = x + 1$ was fitted for ages 20–40; claiming salaries for age 100 from the same line is unsupported.
Forgetting the residual logic. The method only works when points hug the line. If the scatter is a wide cloud with no clear trend, the "noise" you remove is actually real signal.

Visual intuition. Picture a scatter plot with axes age (years, x-axis) and salary (thousands, y-axis). The points form a loose diagonal band rising from lower-left to upper-right. A straight line runs through the middle of the band — the minimum-error fit — and small vertical dashes connect each point to the line. Those dashes are the residuals, the minor variations attributed to noise. The takeaway: the band collapses onto the line, and the dashes vanish.

Recap and bridge. Linear regression smoothing fits a minimum-error line to two related attributes and replaces each point with its projection onto that line — reducing noise without identifying noisy points. Binning smoothed with neighbors; regression smooths with a global pattern. Next we flip the strategy: instead of smoothing noise away, outlier analysis tries to identify suspicious points and decide whether they are genuine.

Where this matters in practice. This is the classic tool of compensation analytics (age–salary benchmarking), economics (demand versus price), and experimental science (calibration curves). Any two attributes with an approximately linear trend can be de-noised this way before further modeling.

4.5 Outlier Analysis

Hook. A credit-card transaction of ₹4,90,000 at 2 a.m. in a city where the cardholder lives — suspicious or not? Outlier analysis is how data mining answers this: it finds the few points that look nothing like the rest, then asks whether they are genuine.

4.5.1 Outliers Versus Noise

The smoothing strategies so far did not bother about finding noise and removing it. Outlier analysis is different: here we identify the suspicious points and then handle them, maybe by removing them.

The setup: in your dataset there are many points, but a few points are very different from the rest. We focus on those few and ask whether they are legitimate or not. If a point is legitimate, we do not call it noise. If it is not legitimate, most likely we consider it noise.

The crucial asymmetry. Noise often shows up as an outlier, but the reverse is not true. Some points are outliers and yet are perfectly genuine data. The outlier label is about difference; the noise label is about corruption.

Example: a billionaire's net worth in a table of typical incomes is a genuine, legitimate data object — it is an outlier by difference, but it is not noise. A salary of ₹10,000 where ₹1,00,000 was intended is noise that looks like an outlier. The two labels answer different questions, and mixing them up leads you to delete real data.

4.5.2 The Cluster-First Procedure

The trick is to avoid scanning the entire dataset for noise. Instead:

The cluster-first procedure.

Cluster. Run cluster analysis on the data to form clusters — groups of points that look alike.
Collect candidates. A few points land outside any cluster. These are the candidates.
Verify. Focus only on these outliers and check whether each one is genuine or bad.
Act. If a point is bad — a noisy point — drop it instantly.

Trace on a tiny example. Suppose the dataset is a map of 50 customer locations in a city, plotted by (x = longitude, y = latitude). Clustering groups the customers into three dense neighborhoods — say a northern cluster, an eastern cluster, and a southern cluster. Two points lie far from all three neighborhoods: one on the city's edge (the sales team's office, genuine) and one in the sea (recorded with a faulty GPS, noise). Step 3 inspects only these two points — not the other 48 — and step 4 drops the sea point, keeping the office.

Sense-check: we examined 2 points instead of 50, and the genuine outlier survived while the noisy one was removed. That is the whole point of the procedure — effort spent only on the suspect set.

So instead of looking at the complete dataset to find noise, you use clusters to isolate a small suspect set, then spend your effort only on those points. This is one standard way of handling noise: given a dataset, do outlier analysis to find outliers, analyze them to see whether they are genuine, and remove the ones that are not.

Visual intuition. Imagine a scatter plot of customer locations in a city (axes: longitude and latitude). Three dense blobs mark the clusters; a handful of isolated dots float in the empty space between and around them. The eye goes straight to the isolated dots — that is the advantage of clustering: the human (or the algorithm) can focus on the sparse regions instead of scanning the whole plane.

Scope: when does cluster-first work?

It assumes the data genuinely forms clusters — tightly grouped, separated regions. If the data is one long smear with no structure, "outside every cluster" is ill-defined and the method loses its grip.
Clustering quality limits the result: the method is only as good as the clusters it starts from, and different clustering methods give different suspect sets.
A point "outside all clusters" still needs human or domain judgment to be called noise — clustering only shortlists candidates, it does not decide guilt.

Pitfalls — the traps beginners fall into.

Deleting every outlier. Legitimate outliers exist (billionaires, rare diseases, one-in-a-million records) — dropping them silently destroys real information.
Using outlier detection as noise detection. Noise is corruption; outliers are difference. An outlier needs inspection; only bad ones get deleted.
Scanning everything by eye. The whole benefit of cluster-first is focusing on the few; inspecting all points manually is expensive and error-prone.
Forgetting to record what was removed. Downstream analysis needs to know which points were deleted and why.

Recap and bridge. Outlier analysis flips the strategy from smoothing to identification: cluster first, shortlist the points outside the clusters, verify each, and drop only the bad ones — remembering that outliers can be perfectly genuine. Next we look at how noise and inconsistencies are caught in practice: by hand, and by automated checks.

Where this matters in practice. Outlier analysis is the engine behind credit-card fraud detection (unusual transactions), network intrusion detection (unusual traffic volumes), manufacturing quality control (defective parts that deviate from the norm), and medical screening (patients whose readings deviate from the healthy range).

4.6 Manual and Automated Noise Detection

Hook. Somewhere in your 50,000-row table sits an age of 250. Two ways to find it: stare at the table for hours, or tell the computer "age must be under 100." Which one would you bet your evening on?

4.6.1 Manual Inspection

Another way to find noise or inconsistent data is to do it by hand. You have a big table, you start looking through it manually, and you skim fast to spot anything weird. This is not exotic — people do this all the time, giving the data a quick eyeball pass to find something bad, then focusing on the bad part to fix it.

The catch: manual inspection is expensive and tedious. Looking through noisy points manually and fixing them during corrections costs time and attention. Manual inspection works for small data and spot checks, but it does not scale — the bigger the table, the more noise hides in it, and the more it costs to find.

4.6.2 Automated Checks with Domain Constraints

The alternative is to write modules or checks that find inconsistencies automatically.

The domain-constraint pattern. Every data point is described by attributes. Based on your domain knowledge, you can state that a particular attribute may range within certain values. If a value goes beyond the range, raise the alarm, then inspect that data point. These checks are called global constraints — a rule that every tuple in the table must obey.

Worked example: the age check. Think about a table where you are gathering the age of human beings. We know the typical age of a human is from 0 to 100 — you will not find a person who is 200 years old. So you put a check on the age attribute:

$age < 100 for every tuple$

If a tuple violates the rule — say age = 250 — raise the alarm. Then you focus on that particular tuple to find out whether the point is genuine or was recorded noisily.

Sense-check: one short line of logic screened an entire column, and only the violating rows need human attention — the same "shortlist, then inspect" economy as outlier analysis, applied at the row level.

4.6.3 Functional Dependencies

You can also write functional dependencies, where one attribute is expected to depend on another.

Cross-attribute validation. A functional dependency lets one attribute validate another: one feature depends on another feature, and you check whether the dependency holds to find noise in the data.

Worked example: the salary dependency. Example: typically, if a child is less than 15 years old, they will not be earning — the salary should be zero. So you write a check across two attributes:

$if age < 15 then salary = 0$

Apply it to a small table:

Tuple	Age	Salary	Check result
1	12	0	Passes — child, no income
2	45	24,000	Passes — age is not below 15
3	10	8,500	Flagged — child with nonzero salary

Tuple 3 is flagged as a possible noisy record: either the age is wrong (a 10-year-old does not earn 8,500), or the salary is wrong (a real employee's age was mistyped). Either way, this one tuple is pulled out for inspection while the rest of the table is trusted.

Sense-check: the check needs only two columns and one rule, yet it catches exactly the kind of inconsistent record that skips past a single-column range check.

4.6.4 Combining Both

In real life we combine human inspection and computer inspection to find noise in the data and then remove it. The computer applies range checks and dependency rules at scale; the human reviews the flagged tuples, decides which are genuine and which are noisy, and fixes or removes them.

Recap and bridge. Noise detection comes in two speeds: slow manual eyeballing for small tables, and automated checks — global constraints on single attributes and functional dependencies across attributes — that shortlist suspect tuples for human review. Real pipelines run both together. This closes the noise-handling story; next we move to the second preprocessing method: data integration.

Where this matters in practice. Every serious ETL (extract, transform, load) pipeline in banking, insurance, and healthcare runs exactly these checks: age ranges, zip-code formats, "salary must be zero when unemployed" rules. Automated constraint checking is the cheapest insurance a data team can buy — it catches errors before they reach the models that make decisions.

4.7 Data Integration

Hook. Your company's data lives in three places: sales in one database, customers in another, products in a spreadsheet. To build one model you must merge them. The catch: merge naively and the same customer appears twice, height appears in two units, and one table's cust ID silently becomes a second column. Integration is where good datasets go to die — or get born.

4.7.1 Why Integration Matters

The second part of data preprocessing is data integration, and it matters because in real life data does not come from a single source. Data always comes from multiple sources, multiple datasets. If you have several tables and you want to run or build a mining algorithm on them, you must first merge all of them into a bigger dataset.

Merging touches two things at once: tuples (the rows) and attributes or features (the columns and the schema). Both must be handled carefully so that you avoid or reduce inconsistencies and redundancy in the merged dataset.

The recurring warning: don't merge blindly. When we talk about data integration, three problems deserve attention: the entity identification problem, tuple duplication, and data value conflicts. Every one of them is a way that a careless merge quietly corrupts the result.

4.7.2 Entity Identification Problem

The entity identification problem is about attributes that mean the same thing but have different names. You have two tables. In both tables there is a field that records the customer ID — but in table A the field is called cust ID and in table B it is called customer ID.

If you merge directly without looking at what each attribute signifies, the check is just on names. The names do not match, so both columns get included, and the resulting table now has two different attributes that both represent customer ID. That is a bad merge.

Worked example: the same concept, two names.

Table A: | cust ID | Name | |---|---| | 101 | Priya R. | | 102 | Arjun S. |

Table B: | customer ID | City | |---|---| | 101 | Pune | | 102 | Mumbai |

A blind merge concatenates the columns and produces: | cust ID | customer ID | Name | City | |---|---|---|---| | 101 | 101 | Priya R. | Pune | | 102 | 102 | Arjun S. | Mumbai |

Now the table carries two columns that mean the same thing — redundancy that confuses any algorithm and doubles the memory cost.

The fix: based on analysis of the data and the metadata, identify what each attribute signifies. Understanding that cust ID in one table stores the same thing as customer ID in the other, you merge them in a different fashion — a single customer_id column — instead of blindly keeping both. You must understand what each attribute signifies in each dataset before merging.

Sense-check: after the fix, the merged table has one ID column, and the same customer can never be split across two inconsistent ID columns.

4.7.3 Data Value Conflicts

Data value conflicts appear when the attribute names match but the way the values are represented differs between the two tables.

Worked example: same name, different representation.

A particular entry for a person's name in table A was written as "JD Smith," and the same person appears in table B with the name written differently ("J.D. Smith" or "John Smith"). There is no entity identification problem here — the attribute name matches — but the representation differs. If you merge blindly, you create two entries for the same person.
Another example: in table A, height is recorded in feet; in table B, height is recorded in centimeters.

Tuple	Table A (feet)	Table B (cm)	Blind merge
1	5.9	180	5.9
2	—	175	175

A blind merge leaves some height values in centimeters and others in feet, which is wrong. Each tuple should use a single unit; a tuple cannot carry multiple units of the same measure.

Sense-check: every row of the merged table must use one unit of measure. Converting table A's feet to centimeters (5.9 ft × 30.48 ≈ 180 cm) before merging makes the column uniform.

So whenever you merge, you must look at both the attributes and the tuples before combining two or more tables.

4.7.4 Tuple Duplication and File Formats

Even after you merge carefully, you should check whether the merged table contains duplicacy. One way to detect redundancy is correlation analysis, covered next. It can be done over columns and over rows.

File formats are another integration trap. Not all datasets come in the same format. Suppose one dataset is stored as CSV (text) and another as Excel; the Excel file carries some extra information. You have to map the two in a certain fashion so that there is uniformity once the datasets are merged — you cannot just blindly concatenate them.

Q: Data cubes means merging multiple datasets into one N-dimensional structure — how do we analyze that? A: Yes. You take one dataset, another dataset, another, and merge them all, and you end up with a data cube that is N-dimensional. How you analyze that is data cube analysis. And on file formats: if one dataset is text, another an image, another something else, you have to be careful while merging. First normalize them to a uniform representation — for example, convert everything to the same tabular form — then merge. Tools like scikit-learn provide functions to merge datasets, and the tutorial will merge two datasets live into a single data frame to show how the analysis works.

Pitfalls — the traps beginners fall into.

Concatenating CSV and Excel files without mapping. The extra metadata Excel carries gets lost or garbled; normalize every source to one representation first.
Ignoring duplicate rows after the merge. Two sources can hold the same real-world object; without a check (e.g., a key column), the merged table silently double-counts.
Trusting column names. Names are the last thing to trust — cust ID and customer ID prove it. Confirm meaning from metadata and data analysis.
Mixing units or date formats. Feet and centimeters, dd/mm vs mm/dd, currency symbols — every representation difference is a future bug.

Recap and bridge. Integration merges tuples and attributes from multiple sources, and it fails three classic ways: same concept under different names (entity identification), same name with different representations (value conflicts), and duplicated tuples. Correlation analysis — our next topic — is the tool that detects when merged columns (or rows) are redundant because they move together.

Where this matters in practice. Integration is the daily reality of data engineering: hospital systems merging patient records from clinics and labs, retailers combining online and in-store sales, banks joining transaction logs with customer profiles. A uniform, consistent, deduplicated merged table is the foundation every downstream model sits on.

4.8 Correlation Analysis

Hook. Is "years of experience" a separate piece of information from "age"? Maybe not — the two rise together. When attributes move together, the table carries redundant information. Correlation analysis is the tool that measures exactly how much two attributes move together.

4.8.1 Positive, Negative, and Zero Correlation

Correlation analysis is used to find whether attributes (or tuples) are redundant because they move together. The idea: suppose there are two attributes, A and B, in your dataset. If attribute A increases and B also increases proportionally, that is positive correlation.

The correlation value ranges from −1 to 1, with 0 in the middle:

+1 (positive correlation): increase the value of one attribute, and you will see a definite, similar proportional increase in the second attribute.
−1 (negative correlation): increase the value of one attribute, and the second attribute decreases in the same proportion.
0 (zero correlation): the two attributes are not related at all.

In the ideal case, many attributes should not be dependent on each other. This correlation idea is also exploited by classification algorithms later in the course — we will come back to it when we do classification.

The formal definition (Pearson's correlation coefficient). For numeric attributes, the correlation between A and B is measured by Pearson's product-moment correlation coefficient:

$r_{A, B} = \frac{\sum _{i = 1}^{n} ( a _{i} - A ˉ ) ( b _{i} - B ˉ )}{n \cdot σ _{A} \cdot σ _{B}}$

where $n$ is the number of tuples, $a_{i}$ and $b_{i}$ are the values of A and B in tuple $i$ , $A ˉ$ and $B ˉ$ are the mean values of the two attributes, and $σ_{A}$ , $σ_{B}$ are their standard deviations. The numerator sums, over every tuple, the product of the two deviations from the means; dividing by $n σ_{A} σ_{B}$ scales the result into the range $- 1 \leq r_{A, B} \leq + 1$ . If $r_{A, B} > 0$ , the attributes are positively correlated (A rising with B); if $r_{A, B} < 0$ , negatively; if $r_{A, B} = 0$ , there is no linear correlation. The higher the absolute value, the stronger the correlation — and the more one attribute implies the other, which may mean one of them is redundant and can be removed. (For nominal data, the analogous test is the chi-square $χ^{2}$ test — mentioned here for completeness; the hands-on computation is covered in the tutorial.)

Visual intuition. Picture three scatter plots, each with attribute A on the x-axis and attribute B on the y-axis. In the first, the points form a tight diagonal band rising left to right — strongly positive correlation, $r \approx + 1$ . In the second, the band falls left to right — negative correlation, $r \approx - 1$ . In the third, the points fill a round cloud with no direction — zero correlation, $r \approx 0$ . One glance at the shape of the cloud tells you the sign and strength of the relationship.

4.8.2 Correlation as an Exploratory Tool

Correlation is not always bad. It is not saying that positive or negative correlation is good or bad; it is saying that correlation gives you better insight into your data. That is what makes it an exploratory tool. Contrast that with the earlier problems: entity identification and data value conflicts are genuinely bad and should not happen at all, but correlation analysis is a way to understand your data better — it tells you which attributes carry the same information, where redundancy lives, and which attributes might be safely dropped.

In scikit-learn (the Python library), there are APIs to compute correlation between different features in a dataset. You can find the correlation and study the relationships among features. Correlation analysis will also be covered hands-on in the tutorial.

Scope and pitfalls.

Correlation does not imply causality. If A and B are correlated, that does not mean A causes B. A classic example: the number of hospitals in a region and the number of car thefts are positively correlated — but hospitals do not cause thefts. Both are caused by a third attribute: population. Never write "A causes B" from a correlation number alone.
r = 0 does not mean independent. Pearson's r only measures linear relationships. Two attributes can be strongly related in a curved (nonlinear) way and still give r ≈ 0.
One outlier can hijack r. A single extreme point can swing the coefficient far from its true value — check the scatter plot before trusting the number.
Range matters. Correlation is about co-movement, not magnitude: two attributes moving together in perfect lockstep give r = ±1 regardless of how different their scales are.

Recap and bridge. Correlation measures how two attributes move together, on a scale from −1 through 0 to +1, and it doubles as a redundancy detector after integration and as a general exploratory tool. Next we move to the third preprocessing method — data transformation — and the question of why raw attribute values often need rescaling before an algorithm sees them.

Exam note: the formal definition of correlation is easy to find — know that r lies in [−1, +1] and what the extremes mean. The tutorial covers computing correlation in scikit-learn, so be comfortable reading a correlation matrix.

Where this matters in practice. Correlation analysis drives feature selection in every industry: banks drop one of two nearly identical credit-risk attributes, marketers discover that engagement and retention move together, and data teams use correlation matrices to spot redundancies before training anything. The same idea powers PCA, which we meet later in this session's transformation toolkit.

4.9 Data Transformation: Why Normalize

Hook. Two neighbors, same age, but one earns 5,00,000 and the other 1,00,000. In any distance calculation, the age difference between them is invisible next to the salary difference — the big numbers simply swallow the small ones. Normalization exists to stop that.

4.9.1 The Goal of Transformation

Data transformation consolidates data and features into a form that is appropriate for data mining. You transform some tuples or attributes so that they have much more impact — a better effect — on the data mining algorithm when you build the model. Typically, transformation is done in terms of columns: you take one or more attributes and transform them so that the transformation aids your data mining process.

4.9.2 The Dominance Problem of Large Values

Why normalize at all? Two reasons. First, normalization speeds up the learning process, the data mining process itself. Second — and this is the important one — it avoids a variable with large values dominating the calculations.

The dominance mechanism. You have numerical values, and you are building functions over them, computing distances, doing multiplication, division, and other mathematical operations. If one attribute has a big range of values — say a value ranging from 1,00,000 to 50,00,000 — and another attribute has small values, say 1 to 200, then whenever you calculate distances or do any mathematical calculation, the larger values will have more impact than the smaller values.

Worked example: age versus salary. A table with two attributes, age and salary. Age takes 20, 21, 22 — small numbers. Salary takes values like 1,00,000, 50,00,000, 10,00,000, 15,00,000 — huge numbers. If you project this into a two-dimensional space, the points are dominated toward the salary axis because that range is high.

Take two concrete points:

Point	Age	Salary
P1	25	5,00,000
P2	30	5,01,000

The Euclidean distance between them is:

$d (P 1, P 2) = (30 - 25)^{2} + (5, 01, 000 - 5, 00, 000)^{2} = 5^{2} + 100 0^{2} \approx 1000.01$

The age difference contributes $5^{2} = 25$ out of a total of $25 + 10, 00, 000$ — about 0.0025% of the distance. Purely because its absolute values are much higher, the salary attribute owns the entire calculation. Any distance calculation will be dominated by the salary attribute, and the model will behave as if age did not exist.

Sense-check: if the two attributes are normalized to the same scale, age contributes its fair share of the distance, and both attributes influence the model.

This will be shown with a real-life example in kNN (k-nearest neighbors) when we get there. For now the goal is clear: normalize one or both attributes so that both come to almost the same scale.

4.9.3 The Toolkit of Techniques

The normalization techniques we will cover: min-max normalization, z-score normalization, and decimal scaling. Beyond normalization there is also attribute construction — building new attributes from existing ones, with PCA and t-SNE as examples — and aggregation, covered later in this session.

Recap and bridge. Transformation reshapes attributes so they help the algorithm: it speeds up learning and — critically — stops large-range attributes from dominating distance and other calculations. The three rescaling tools come next, starting with min-max normalization.

Where this matters in practice. Every distance-based or gradient-based model — kNN, k-means clustering, SVMs, neural networks — is sensitive to scale. In industry, features arrive in wildly different units (income in rupees, age in years, clicks in thousands), and normalization before training is standard practice in every ML pipeline, from credit scoring to recommendation systems.

4.10 Min-Max Normalization

Hook. The simplest way to fix the dominance problem: squeeze the whole column into a range you choose, like 0 to 1. One linear formula, and every value lands exactly where you want it.

4.10.1 The Formula

Min-max normalization is a linear transformation. Suppose a particular attribute has values $v_{1}, v_{2}, \dots, v_{n}$ . Find the maximum and minimum of that attribute, then transform every value linearly onto a new range of your choice.

The formula.

$v^{'} = \frac{v - min _{A}}{max _{A} - min _{A}} \cdot (new_max - new_min) + new_min$

where:

$v$ is the old value,
$v^{'}$ is the new value,
$min_{A}$ and $max_{A}$ are the minimum and maximum of the attribute in the original data,
$new_min$ and $new_max$ define the target range.

Where the new minimum and new maximum come from is a domain decision — the formula does not tell you that. You decide the target range based on your requirement, and then all points are defined by it.

How to read the formula in words. The fraction $\frac{v - m i n _{A}}{m a x _{A} - m i n _{A}}$ measures where $v$ sits inside the original range: 0 at the minimum, 1 at the maximum, 0.5 exactly halfway. Multiplying by $(new_max - new_min)$ stretches that position onto the new range's width, and adding $new_min$ shifts it into place. So a value that was the smallest in the column becomes $new_min$ , the largest becomes $new_max$ , and everything in between is placed proportionally.

4.10.2 Worked Example: Age 30 in Range 10 to 80

Normalizing the value 30 into [0, 1]. Take the age attribute from the earlier table, and suppose we want to normalize the value 30. The original range of the attribute is 10 to 80, and we choose to normalize into the range 0 to 1.

Substitute into the formula:

$v^{'} = \frac{30 - 10}{80 - 10} \cdot (1 - 0) + 0 = \frac{20}{70} = \frac{2}{7} \approx 0.28$

So the value 30 is replaced by about 0.28 in the normalized attribute.

Check the extremes to build confidence. The minimum 10 maps to $\frac{10 - 10}{70} \cdot 1 + 0 = 0$ — the smallest value becomes the new minimum 0. The maximum 80 maps to $\frac{80 - 10}{70} \cdot 1 + 0 = 1$ — the largest becomes 1. And 30, sitting two-sevenths of the way between 10 and 80, lands at 2/7 ≈ 0.28 — exactly where the sense-check predicts.

We do the same exercise for every tuple in the attribute: the whole column, which used to run from 10 to 80, now runs from 0 to 1.

A second walk-through (income, from the reference). Suppose income ranges from ₹12,000 to ₹98,000 and we map it to [0, 1]. A salary of ₹73,600 transforms to:

$v^{'} = \frac{73 , 600 - 12 , 000}{98 , 000 - 12 , 000} \cdot (1 - 0) + 0 = \frac{61 , 600}{86 , 000} \approx 0.716$

The high salary lands high in the new range — the ordering is kept, only the numbers change.

4.10.3 The Relationship-Preserving Property

Min-max normalization has one notable advantage: it preserves the relationship between the original data values.

Why the relationship survives. Suppose the original values $v_{1}, v_{2}, v_{3}, v_{4}$ hold certain relationships among themselves — say $v_{1}$ and $v_{2}$ differ by some proportion, $v_{2}$ to $v_{3}$ by another. After the transformation, the transformed values $v_{1}^{'}, v_{2}^{'}, v_{3}^{'}, v_{4}^{'}$ hold the same relationships.

The reason is that the transformation is linear in the old value: the difference between any two transformed values is a fixed multiple of the original difference,

$v_{i}^{'} - v_{j}^{'} = \frac{new_max - new_min}{max _{A} - min _{A}} \cdot (v_{i} - v_{j})$

so the spacing between values is only scaled by one constant — never distorted. Relative spacing is unchanged, so the relationship between the values of a particular attribute does not get destroyed by min-max normalization.

Keep this property in mind — it becomes the homework question for z-score normalization next.

Scope and assumptions.

Out-of-bounds future values break the range. If a future input case falls outside the original range of A (a new employee aged 82 when the table only reached 80), the formula pushes it outside [new_min, new_max] — an "out-of-bounds" error. The technique assumes new data arrives inside the range the minimum and maximum were computed from.
Min and max are the whole story. The two extremes are sensitive to outliers: one absurdly high value inflates $max_{A}$ and compresses everyone else near the bottom.
The target range is your choice. [0, 1] is common, but [−1, 1] or any other range is legitimate — the choice is a requirement decision.
A constant attribute breaks the formula. If every value is identical, $max_{A} - min_{A} = 0$ and the fraction divides by zero — you cannot min-max normalize a column with no variation.

Pitfalls — the traps beginners fall into.

Forgetting to save the parameters. The $min_{A}$ and $max_{A}$ used for training must be reused for future data — recomputing them on new data silently changes the mapping.
Applying the formula column-wise but comparing across columns. Normalizing each attribute separately is the standard use; mixing the ranges afterwards undoes the work.
Ignoring outliers before choosing min and max. One extreme value can crush the whole column into a narrow band.
Confusing min-max with z-score. Min-max maps into a fixed interval using only min and max; z-score (next) centers on the mean using mean and standard deviation — different formulas, different properties.

Recap and bridge. Min-max normalization maps every value of an attribute linearly onto a chosen target range, and because it is linear it preserves the relative spacing — the relationships among the values survive. Next we see z-score normalization, which uses the mean and standard deviation instead — and the homework question asks whether its transformed values keep the relationships too.

Exam note: min-max normalization preserves the relationship between the original data values — be able to state why (the transformation is linear) and to compute a single value by hand.

Where this matters in practice. Min-max is the default rescaling in image processing (pixel intensities to [0, 1]), in neural network inputs, and anywhere the domain guarantees bounded values — like test scores or percentages. When the data is bounded and clean, min-max is simple and effective.

4.11 Z-Score Normalization

Hook. Min-max needs a fixed minimum and maximum — but what if you do not know the true bounds of your data? Z-score normalization sidesteps the question: it centers the data on the mean and scales it by the spread, no min or max required.

4.11.1 The Formula

Z-score normalization normalizes the values of an attribute based on two statistics of that attribute: the mean and the standard deviation.

The formula.

$v^{'} = \frac{v - μ _{A}}{σ _{A}}$

where:

$v$ is the original value of the attribute,
$v^{'}$ is the transformed value,
$μ_{A}$ is the mean of the attribute,
$σ_{A}$ is the standard deviation of the attribute.

In words: subtract the mean, then divide by the standard deviation. So the transformed value for an attribute depends on two things only — the mean and the standard deviation of that attribute.

How to read the formula in words. The numerator $v - μ_{A}$ measures how far the value sits from the center of the column — its deviation. Dividing by $σ_{A}$ converts that deviation into "how many spread-units from the center": a value exactly at the mean becomes 0, a value one standard deviation above the mean becomes +1, one standard deviation below becomes −1. Z-score is also called zero-mean normalization because the transformed column always has mean 0 and standard deviation 1.

4.11.2 Worked Example: Marks 35, 65, and 90

Three students A, B, C have marks 35, 65, and 90 in an attribute. We transform the values.

The full transform, corrected. First find the mean of the column:

$μ = \frac{35 + 65 + 90}{3} = \frac{190}{3} \approx 63.33$

The correction in the class. The mean stated in the class was 60, which is not correct for these marks — the correct mean is 63.33. This slip was caught by a student in the chat during the class (see the Q&A below), and the corrected arithmetic was promised in the slides. We work with the corrected mean here.

Next, the deviation scores — subtract the mean from each mark. Using the corrected mean of 63.33:

$35 - 63.33 = - 28.33, 65 - 63.33 = + 1.67, 90 - 63.33 = + 26.67$

(The spoken deviations during the class were "minus 25, 5 and 10", computed against the slipped mean of 60; the corrected deviations above are the ones the slides re-solve with.)

Then each deviation score is divided by the standard deviation. The standard deviation of the marks is:

$σ = \frac{( - 28.33 ) ^{2} + ( 1.67 ) ^{2} + ( 26.67 ) ^{2}}{3} = \frac{802.8 + 2.8 + 711.1}{3} = \frac{1516.7}{3} \approx 505.6 \approx 22.48$

So the z-scores are:

$v_{A}^{'} = \frac{- 28.33}{22.48} \approx - 1.26, v_{B}^{'} = \frac{+ 1.67}{22.48} \approx 0.07, v_{C}^{'} = \frac{+ 26.67}{22.48} \approx 1.19$

Student A sits about 1.26 standard deviations below the class average, student C about 1.19 above it, and student B is essentially at the average.

Sense-check: z-scores always add to zero ( $- 1.26 + 0.07 + 1.19 \approx 0$ ) because the positive and negative deviations cancel — a quick arithmetic check that the transform was applied correctly.

Q: In the chat: the mean is 63.3, not 60, for the marks 35, 65, and 90. A: Correct — the arithmetic slip is acknowledged: 35 + 65 + 90 = 190, and 190 divided by 3 is 63.3, not 60. The example will be re-solved with the corrected values and uploaded in the slides.

The takeaway stands regardless of the slip: the transformed value in z-score normalization depends on the mean and the standard deviation of the attribute, and we get it by subtracting the value from the mean and dividing by the standard deviation.

4.11.3 The Absolute-Deviation Variant

There is a variation of z-score normalization: instead of using the standard deviation in the denominator, use the absolute standard deviation — the mean absolute deviation of the attribute.

The mean absolute deviation. The lecture's "absolute standard deviation" is the mean absolute deviation of the attribute, written $s_{A}$ :

$s_{A} = \frac{1}{n} i = 1 \sum n ∣ v_{i} - μ_{A} ∣$

Instead of squaring the deviations (as the standard deviation does), it takes their absolute values and averages them. The z-score-like transform becomes:

$v^{'} = \frac{v - μ _{A}}{s _{A}}$

Why it exists. The mean absolute deviation is more robust to outliers than the standard deviation: because deviations are not squared, a single extreme value cannot inflate $s_{A}$ as much as it inflates $σ_{A}$ . When your column contains wild values, dividing by $s_{A}$ keeps the transformed values calmer.

Same marks, absolute-deviation variant. With the corrected mean 63.33:

$s_{A} = \frac{∣35 - 63.33∣ + ∣65 - 63.33∣ + ∣90 - 63.33∣}{3} = \frac{28.33 + 1.67 + 26.67}{3} = \frac{56.67}{3} \approx 18.89$

The transformed values are:

$v_{A}^{'} = \frac{- 28.33}{18.89} \approx - 1.50, v_{B}^{'} = \frac{+ 1.67}{18.89} \approx 0.09, v_{C}^{'} = \frac{+ 26.67}{18.89} \approx 1.41$

Sense-check: the order of the students is unchanged and the values still center on 0, but the scale is wider than the standard-deviation version because 18.89 < 22.48 — the absolute deviation variant spreads the z-scores further apart for this small, clean dataset.

4.11.4 Homework: Does Z-Score Preserve Relationships?

Recall that min-max normalization preserves the relationship between the original data points — the transformed values keep the same relative spacing. Now that you have seen the z-score formula, ask yourself the homework question: does the same relationship hold between the z-score transformed values? Are the relationships among the original data points preserved after z-score normalization, or not? Think it through — it is a question about what the formula actually does to the spacing between values.

There is a second homework question paired with it: when do we use min-max and when do we use z-score? The relationship property is one reason, but there are others.

Recap and bridge. Z-score normalization centers an attribute on its mean and rescales by its standard deviation — or by the more outlier-robust mean absolute deviation. Unlike min-max, it needs no assumed minimum and maximum, and the transformed column always has mean 0. The homework questions — does z-score preserve spacing, and when to prefer it over min-max — are the bridge to the third technique, decimal scaling.

Exam note: the z-score worked example is being re-solved in the slides because the mean is 63.33, not 60 — use the corrected arithmetic. And know both homework directions: the relationship-preservation question and the min-max-versus-z-score choice.

Where this matters in practice. Z-score is the default when the data has unknown bounds or contains outliers: feature scaling for distance-based models like kNN and k-means, standardizing features in PCA, and preparing inputs for many regression and classification models. It is the standard scaler in most machine-learning toolkits.

4.12 Decimal Scaling

Hook. The class called it the "dumb way" to normalize — just move the decimal point. Move it far enough, and every value in the column lands between −1 and 1. It is simple, it is in the literature, and sometimes simple is exactly what you need.

4.12.1 The Formula and the Meaning of J

Decimal scaling is another normalization technique. It is a simple approach, but it is in the literature and you will see it used.

The transformation.

$v^{'} = \frac{v}{1 0 ^{J}}$

where $J$ is the smallest number such that the absolute maximum of the attribute becomes ≤ 1 after division. In practice, $J$ is found by looking at the absolute maximum of the attribute: $J$ is essentially the number of digits in the absolute maximum value. Then every value of the attribute is divided by $1 0^{J}$ .

How to read the formula in words. Dividing by $1 0^{J}$ moves the decimal point $J$ places to the left. Choose $J$ so that the biggest value in the column — after the division — is at most 1; then every smaller value is also inside [−1, 1] automatically, and the whole column fits the target band. (Standard treatments write the condition strictly as $max ∣ v^{'} ∣ < 1$ ; the lecture's "≤ 1" version is the same in practice, since the boundary case only matters when the maximum is an exact power of ten.)

4.12.2 Worked Example: Range −986 to 917

Moving the decimal point. Suppose an attribute's recorded values range from −986 to 917. The absolute maximum of the attribute is 986 — that is, $max ∣ v ∣ = 986$ . This number has three digits, so:

$J = 3, 1 0^{J} = 1000$

Now transform the values by dividing by 1000:

$\frac{917}{1000} = 0.917 and \frac{- 986}{1000} = - 0.986$

The value 917 becomes 0.917, −986 becomes −0.986, and every other value in the attribute is divided by the same 1000, so the entire column lands between −1 and 1.

Sense-check: the largest absolute value in the column is exactly the one that set $J$ — 986/1000 = 0.986 ≤ 1 — so every other value, being smaller in absolute terms, also satisfies the bound. The sign of every value is preserved: positives stay positive, negatives stay negative.

It is a very simple way of doing it — the class called it a "dumb way" — but it is legitimate and appears in practice, so know it.

Scope and pitfalls.

One attribute-wide decision, applied to every value. $J$ is fixed once, from the absolute maximum of the whole column; do not recompute it per value.
The digit rule can mislead on boundary cases. A maximum of exactly 1000 has four digits (J = 4 needed under the strict form, since 1000/1000 = 1 is not < 1). When in doubt, test the division directly rather than counting digits by habit.
Small changes in scale are easy to miss. Because the transform looks trivial, teams forget to record $J$ — but future data must be divided by the same $1 0^{J}$ to stay comparable.
Prefer z-score when outliers loom. Decimal scaling is dictated by the single largest absolute value, so one absurd entry inflates $J$ and crushes all other values toward zero — the same sensitivity min-max has.

Recap and bridge. Decimal scaling divides every value by $1 0^{J}$ , where $J$ is the number of digits of the absolute maximum — a one-line transform that forces the column into [−1, 1]. That completes the normalization toolkit: min-max (fixed interval, preserves relationships), z-score (mean and spread based, robust when bounds are unknown), and decimal scaling (simplest of all). Next we leave normalization behind and look at a different transformation idea: aggregation.

Exam note: the decimal scaling numerical example is being re-solved and corrected in the slides — use the reference values: range −986 to 917, absolute maximum 986, J = 3, so 917 → 0.917 and −986 → −0.986.

Where this matters in practice. Decimal scaling appears wherever data must quickly fit a bounded range with no assumed distribution — simple dashboards, coarse feature scaling in spreadsheets, and legacy pipelines that want a one-liner transform. Its simplicity makes it attractive; its sensitivity to one extreme value keeps it from being the default.

4.13 Aggregation

Hook. Do you need the sales of every three-month period, or just the year's total? If the question is yearly, storing four three-month numbers is wasted precision. Aggregation is the transformation that answers: what scale does this question actually need?

4.13.1 Changing the Scale of the Question

Aggregation changes the scale of the data. Sometimes you aggregate because the question you are asking does not need the fine-grained detail.

What aggregation means. Aggregation means combining multiple values so that the data carries more meaning for the specific question at hand. Sometimes it is done for a column, sometimes for some rows together.

Worked example: state-level column to country-level. A table has an attribute about the country, and the values are the names of different states. When you process the question, you find this attribute has a lot of information, but the question is not about any particular state — the answer is uniform throughout the country. So you can aggregate the whole column into the country itself. You can also aggregate a subset of tuples, based on the question and the requirement.

Sense-check: the aggregated column holds one value where it held many — the detail was not needed by the question, so it is gone.

4.13.2 Worked Example: Quarterly to Yearly Sales

Quarterly sales into one yearly value. Suppose you have sales information for a product for each year, recorded per quarter: quarter Q1 2008, quarter Q2 2008, quarter Q3 2008, and so on. But your mining question says: I do not want quarter-wise information, I just want year-wise information.

Using the lecture's numbers:

Quarter	Sales
Q1 2008	₹2,24,000
Q2 2008	₹4,08,000
Q3 2008	₹3,50,000
Q4 2008	₹5,86,000

Instead of storing four values, you aggregate them together into one yearly value:

$Sales 2008 = 2, 24, 000 + 4, 08, 000 + 3, 50, 000 + 5, 86, 000 = ₹15,68,000$

Sense-check: the four quarterly rows become one yearly row with exactly the information the year-wise question needs — nothing more, nothing less.

That is aggregation — based on the question, you combine values and the dataset gains more meaning for the concern at hand.

4.13.3 Worked Example: Rainfall in Australia, Monthly versus Yearly

The same signal at two scales. Here is a second example, about rainfall in Australia. Plot the same values two ways: monthly information and yearly information. The information content is the same in both — nothing is added or removed — the only difference is the scale of aggregation.

What changes is the spread. The monthly data shows a lot of variation, so its standard deviation is large. The yearly data averages those variations out, so its standard deviation is much smaller. Roughly speaking, aggregating $n$ independent values into one average shrinks the standard deviation by a factor of about $\frac{1}{n}$ — twelve monthly values average out to a far steadier yearly number.

Sense-check: both plots describe the same rainfall, but the monthly curve is jagged and high-variance while the yearly curve is smooth and low-variance — the aggregation decided how much noise-like variation survives in the data.

Sometimes that is exactly what the requirement wants; sometimes it is bad for the analysis. Based on your requirement, you may merge a particular group of attributes so that the average standard deviation comes down — that is just a requirement decision.

Q: How are the monthly and yearly data presented differently? A: In one case the same values are bucketed per quarter or per month; in the other they are combined per year. You represent the same data in the two forms by aggregating. The standard deviation will be different between the two representations — smaller when you aggregate more.

Pitfalls — the traps beginners fall into.

Aggregating when the question is fine-grained. If a later question needs the quarters, the yearly total cannot give them back — aggregation is one-way.
Averaging things that should be summed. Total sales aggregate by addition; a rate or ratio aggregates differently (you cannot simply average percentages from groups of different sizes).
Believing the smaller standard deviation means "better." Low variance after aggregation is a property of the scale, not a sign of higher quality — for some analyses the monthly volatility is exactly the information needed.
Aggregating across inconsistent groups. Mixing states, units, or time zones inside one aggregate silently corrupts the result.

Recap and bridge. Aggregation trades detail for scale: it combines values (a column, a subset of rows, or the whole table) so the data matches the question — and in doing so it shrinks the spread, lowering the standard deviation. It is both a transformation and, as we see next, a form of data reduction.

Where this matters in practice. Aggregation is everywhere in reporting: retail chains roll store sales up to regional and national totals, weather agencies roll hourly rainfall into monthly and yearly statistics, and data warehouses precompute aggregated cubes (the same idea as the data-cube question earlier) so dashboards answer instantly at any granularity.

4.14 Data Reduction: Compression and Histograms

Hook. You list every symptom you have — headache, sore throat, body ache, fever, leg pain, hand pain — and the doctor still writes medicine from a handful of parameters. She did not read your whole file; she reduced it. Data reduction does the same for datasets: shrink them before mining, keeping what matters.

4.14.1 Why Reduce Data

Data reduction shrinks the dataset before mining. There are two directions: reduce the data row-wise (reduce the volume of the data) or reduce it column-wise (reduce the attributes). The motivation is simple: if the data is huge, you need a lot of processing, and reducing it removes overall issues from the dataset.

Professor's analogy — the doctor shortlists. You go to a doctor and list every symptom — headache, nose not good, throat not good, body ache, fever, legs paining, hands paining. You give the doctor a huge amount of data. But when the doctor writes the medicine, he looks at a few parameters, not all. He shortlists a subset of the data to figure out what disease you have. The same logic applies to mining: if you have gathered a lot of data, there may be a lot of redundant, noisy, and irrelevant data in it that is not useful for your mining process. So you reduce the data — in volume, in attributes, or both.

4.14.2 Lossless and Lossy Compression

One form of data reduction is data compression, which comes in two flavors.

Lossless compression means that once you have compressed the data, you can regain the original information back — nothing is lost. The compressed form is a perfect encoder: decompress it and you recover the original exactly.

Lossy compression means that once you have transformed and reduced the data, you cannot get the original information back — something is discarded. The compressed form keeps the important shape of the data but not every detail.

Depending on your requirements, you can choose either. The same trade-off appears everywhere: lossless for things you must reconstruct exactly, lossy when you can afford to drop detail for a much smaller representation.

4.14.3 Worked Example: The Histogram, Both Flavors

A histogram is a very good example of data reduction. Suppose you have a one-dimensional dataset with a huge number of points — say 50 points. Can you transform it into another form with fewer data points? Yes.

Lossless flavor: value–frequency pairs. Transform the one-dimensional data into two-dimensional data. On one axis write each unique number, and on the other axis write its frequency — how many times it appears. If the value 1 appears twice, the point (1, 2) replaces two raw values. If 5 appears five times, the point (5, 5) replaces five raw values. If 8 appears twice, the point (8, 2) replaces two raw values.

Is this lossless or lossy? Lossless. Looking at the table (or graph), you can reconstruct the original data exactly: the value 1 occurred twice, 5 occurred five times, 8 occurred twice, and so on. You can go back to the original data whenever you want. The 50 raw numbers shrink to a handful of (value, count) pairs, yet no information is lost.

Sense-check: decompression is trivial — "expand each value by its count" — and the result is bit-for-bit the original list.

Lossy flavor: value ranges. Now do a different transformation of the same data: combine values into price ranges instead of unique values — values from 1 to 10 in one bucket, 11 to 20 in another, 21 to 30 in another — and count frequencies per range. The data is again mapped to two dimensions, but now the buckets hold ranges, not exact values.

This transformation is lossy. By looking at it, you cannot recover the original values: a bucket says "1 to 10" but not which values were inside it.

Sense-check: the compressed form answers "how many items fall in each band" perfectly, but the question "which exact values were in the 11–20 band?" has no answer — that detail was discarded.

So compression can be lossy or lossless, and the histogram shows both in one picture.

Visual intuition. Draw a histogram with price range on the x-axis and count on the y-axis. In the singleton version, each distinct price has its own thin bar (1 → 2, 5 → 5, 8 → 2, …), and every original value is recoverable from the bar heights. In the ranged version, three wide bars cover 1–10, 11–20, and 21–30 with the counts collapsed inside each band — the silhouette of the data survives, the exact values do not.

Pitfalls — the traps beginners fall into.

Treating a lossy reduction as lossless. The ranged histogram cannot answer value-level questions; use it only when the question is about the shape of the distribution.
Ignoring the bucket choice. Narrow ranges keep more detail but save less space; wide ranges save more but flatten the shape. The bucket width is the dial that controls the trade-off.
Reducing rows but not columns, or vice versa. The doctor example cuts attributes; the histogram cuts volume. A dataset may need both directions — reduce in volume, in attributes, or both.
Assuming "same analytical results" for free. Reduction methods are chosen to preserve the analytical content needed for the task; what is preserved for one question may be exactly what the next question loses.

Recap and bridge. Data reduction shrinks a dataset row-wise or column-wise, and compression comes in two flavors: lossless (perfectly reversible, like the value–frequency histogram) and lossy (detail discarded, like the ranged histogram). Next we look at the most used reduction tool of all: sampling.

Exam note: know the difference between lossless and lossy compression, and be able to classify a given transformation (like the two histogram flavors) into one of them with a one-line justification.

Where this matters in practice. Compression is the physics of every digital system: ZIP files and PNG images are lossless (reconstruct exactly), while JPEG photos, MP3 audio, and video streaming are lossy (detail traded for size). In data mining, histograms and sampling keep big datasets tractable — the same "shortlist what matters" principle the doctor used.

4.15 Sampling: Simple Random Sampling

Hook. Testing 10 lakh blood samples for a disease costs a fortune; testing 1,000 randomly chosen ones is cheap — provided the 1,000 faithfully stand in for the 10 lakh. Sampling is the bet that a small, carefully chosen subset carries almost all the properties of the whole.

4.15.1 The Key Idea: A Representative Subset

Sampling is a data selection technique. You have heard about sampling for years, but here is the data mining framing: the key idea of an effective sample is that the sample should work almost the same as the entire dataset. Whenever we create a sample, we say the sample is a representative of the original data — whatever properties the original data has, almost all those properties the sample also has.

Why sample? On the original dataset, processing may be very expensive and time consuming. A sample gives two advantages: the number of data points you have to process drops, so your efficiency goes up, and you have less to worry about. But this rests on an assumption: the sample preserves the properties of the original data. If the sample is not representative, it is of no use.

4.15.2 Worked Example: Without Replacement

Simple random sampling means you randomly choose data points from the original dataset. The classic example: four points P1, P2, P3, P4, and we want a sample of size 2.

Without replacement. In the first iteration, the probability of any one point being selected is:

$P (P 1) = P (P 2) = P (P 3) = P (P 4) = \frac{1}{4}$

Suppose P1 is chosen. P1 is removed from the dataset — it is not replaced with any other point. Now only three points remain. In the second iteration, the probability of choosing any of the remaining points is:

$\frac{1}{3}$

each. Suppose P3 is chosen. The sample is {P1, P3}. If you needed a third point, P1 and P3 are removed, and you would choose between the remaining P2 and P4, each with probability $\frac{1}{2}$ .

Sense-check: the probabilities track the shrinking dataset — after each draw, the denominator drops by one, and the same point can never appear twice.

So: random sampling without replacement means that once you have selected a data point for the sample, it is removed from the original dataset, and later selections come from the rest.

The same story with eight points: initially each point has probability 1/8. Once D5 is selected, D5 is removed, and the probability of each remaining point becomes 1/7, and so on.

4.15.3 Worked Example: With Replacement

With replacement. Same example: P1, P2, P3, P4, sample of size 2. First iteration: each point has probability 1/4. Suppose P1 is chosen. In random sampling with replacement, P1 is not removed from the dataset. So in the second iteration, all four points are still available, each again with probability:

$\frac{1}{4}$

The same point can be chosen again in the second, third, or any later iteration. It depends on the probabilities, but it is possible; in sampling without replacement it is not possible at all.

Sense-check: the denominator never changes — every draw starts from the full dataset again, so the sample {P1, P1} is a legitimate (if unlucky) outcome.

Another version of the example: a subset of size 4, where the chosen point is T4. Once T4 is selected as a sample, T4 can again be chosen — it is not removed from the dataset. That is how with-replacement sampling works.

The two flavors side by side.

	Without replacement	With replacement
After a draw, the chosen point…	is removed from the dataset	stays in the dataset
Probability on the next draw	$\frac{1}{N - 1}$ (denominator shrinks)	$\frac{1}{N}$ (unchanged)
Can the same point repeat in the sample?	No	Yes
Standard name	SRSWOR	SRSWR

Pick without replacement when every record should appear at most once; pick with replacement when you want every draw to be independent of the previous ones.

Pitfalls — the traps beginners fall into.

Forgetting which flavor you used. Reporting a sample as "random" without saying whether replacement was used makes the probabilities ambiguous — exams and audits care.
Expecting with-replacement samples to be distinct. Repetition is not a bug in SRSWR; it is the defining property.
Assuming any random sample is representative. Randomness only guarantees fairness of the draw, not that this particular sample kept the data's properties — which is exactly why stratified sampling (next) exists.
Sampling from unsorted or biased access paths. If the database pages that contain the data are accessed in order, "randomly" taking every 100th record can quietly align with a pattern in the data.

Recap and bridge. Simple random sampling draws a representative subset with each point equally likely to be chosen — either without replacement (denominator shrinks, no repeats) or with replacement (full dataset every time, repeats allowed). Its value rests on representativeness: a sample that does not preserve the data's properties is worthless. Next we see two smarter strategies that protect representativeness: stratified and cluster sampling.

Exam note: understand random sampling with and without replacement, and be able to state how the selection probabilities change after each draw in each flavor.

Where this matters in practice. Sampling is the workhorse of large-scale analytics: opinion polls sample voters, quality control samples manufactured parts, and data mining samples billion-row logs. The same mechanism powers bootstrap resampling and the random subset selection behind bagging in machine learning.

4.16 Stratified and Cluster Sampling

Hook. Plain random sampling can go wrong: if 80% of your data is middle-aged customers, a random sample of 100 might accidentally contain zero seniors — and then the model learns nothing about them. Stratified sampling makes that impossible: it locks the distribution of the sample to the distribution of the data.

4.16.1 Stratified Sampling: Preserve the Distribution

Stratified sampling keeps the distribution of the data intact. Look at a table with a particular attribute that takes multiple values — for example, youth, middle age, and senior. In the data, middle age dominates the other two groups. If you take a plain random sample, you might get an unbalanced representation, with most sample points drawn from the dominating group.

To create a balanced environment, look at the frequency of each unique attribute value. Stratified sampling says: the distribution of the unique attribute values in the sample must remain the same as in the original. So you create the sample so that each stratum is represented in proportion to its original frequency — the distribution of the strata stays the same in the final sample.

The method. Divide the data into non-overlapping groups called strata (singular: stratum) — one per distinct attribute value, such as youth, middle age, and senior. Then draw a simple random sample inside each stratum, with size proportional to the stratum's share of the data. The name "stratified" comes from these layers: every layer of the population is guaranteed a voice in the sample.

4.16.2 Worked Example: Youth, Middle Age, Senior

Halving every stratum. Suppose the original set has 16 values: youth appears 6 times, middle age 8 times, senior 2 times. We want a sample of 8 values.

Stratum	Original count	Share	Sample count
Youth	6	6/16	3
Middle age	8	8/16	4
Senior	2	2/16	1
Total	16	1	8

To keep the distribution the same, we divide each count by two: from youth we take any 3 values, from middle age any 4, and from senior any 1 — each stratum's count in the sample is exactly half its count in the original, so the sample's distribution (3 : 4 : 1) matches the original's (6 : 8 : 2).

Sense-check: without stratification, the senior group — only 2 of 16 values — might be missing from a random sample of 8; with stratification, senior is guaranteed its proportional 1 value. The distribution of the strata in the sample is the same as the distribution in the original set — that is the basic idea of stratified sampling.

4.16.3 Cluster Sampling: Representatives from Each Group

Cluster sampling takes a different route, and we will see clustering in depth later.

The method. First, find the clusters in the dataset. A cluster is a group of points that has the same or similar properties. Instead of putting the original data in the sample, you take representative points from each cluster: two points from this cluster, two points from that cluster, two more from another. If you started with 100 points, you end with five or ten representative points, each generated from a cluster. Each representative stands for its original cluster, and together the representatives form a sample of the original clusters.

Trace: 100 points into 6 representatives. Suppose the dataset is 100 customer records that cluster into three neighborhoods of 40, 35, and 25 points each. Cluster sampling might draw 2 points from the first cluster, 2 from the second, and 2 from the third — six representatives in total. Each representative carries its cluster's properties, so the six-point sample stands in for the 100-point dataset.

Sense-check: instead of drawing 100 points or scanning them all, we spend effort only on a few representatives per group — the reduction is dramatic, and every group is represented.

So: you can do clustering, then from each cluster take some representative points, and that is a valid way to sample.

Scope and pitfalls.

Stratified sampling protects one attribute's distribution. It guarantees the strata used; if a different attribute is the one that matters, you must stratify on that attribute — or on several at once.
Cluster sampling assumes clusters are meaningful. If the data forms no real clusters, "representative from each cluster" has no grounding — the representatives may be no better than a random sample.
Stratification is not balancing. The sample mirrors the original distribution (6 : 8 : 2). If you want the classes equalized instead — so the model sees all groups fairly — that is the class-imbalance business of the next section, not stratification.
Cluster sampling's cluster choice is yours. Whether clusters come from a clustering algorithm or from natural grouping (city blocks, database pages, hospital wards) changes the result — choose the grouping that matches how the data is organized.

Recap and bridge. Stratified sampling guarantees each stratum appears in the sample in proportion to its share of the data — protecting representativeness against unlucky draws — while cluster sampling picks a few representative points from each cluster, shrinking 100 points to a handful. Both are sampling strategies that put structure back into randomness. Next we face the last sampling questions: how big should the sample be, and what happens when classes are imbalanced?

Where this matters in practice. Stratified sampling is standard in market research (polling each age group proportionally), in medical trials (ensuring each patient group is represented), and in model evaluation (train/test splits that keep the class distribution — the same idea as stratified cross-validation). Cluster sampling is the natural fit for database-page sampling and geographic surveys.

4.17 Sample Size and Class Imbalance

Hook. Sample too small and the structure of your data evaporates; leave classes unbalanced and the model learns only the majority. Both problems are solved with the same tools — sizing and rebalancing — and both decide whether sampling helps or hurts.

4.17.1 Choosing the Sample Size

One of the issues in sampling is how to determine the sample size.

The sizing rule of thumb: keep the relationships. Suppose the original dataset has 8000 data points, and there is a visible relationship in it. If you drop points blindly and take a sample of, say, 2000 points, the relationship is still maintained. But if you take a sample that is too small, the relationship is totally gone.

So you should not take a sample so small that the original relationships in the dataset are lost. It is a tricky question; it depends on a lot of analysis. The point to keep: sampling means you take a subset of the original data that has almost the same properties as the original data. If you take too few points, you start losing those properties, and the sample becomes useless.

Scope: the size question has no single answer. The right sample size depends on the variance in the data, the strength of the relationships, and how much error you can tolerate. Statistics provides formal answers (based on the central limit theorem, a sample size can be chosen to estimate a quantity within a specified error — often far smaller than you would guess), but in practice you check empirically: does the sample still reproduce the distributions and relationships the full data has?

4.17.2 Undersampling and Oversampling

Two important sampling strategies for imbalanced data: undersampling and oversampling.

The imbalance problem. Suppose you have two groups: one group has 100 points, the other has 10 points. This is a class imbalance — one class vastly outnumbers the other. If you build a model on such data, the model can score high accuracy by simply predicting the big class every time, learning almost nothing about the small class. That is why a class-imbalance dataset cannot build a good data mining model — and why undersampling and oversampling exist.

Undersampling: shrink the bigger class. Make both groups equal by shrinking the bigger group: the smaller group of 10 stays as it is, and you take only 10 samples from the bigger group. The final dataset has 10 + 10 = 20 points — both classes balanced at 10 each.

Oversampling: grow the smaller class. Make both groups equal by growing the smaller group: the bigger group of 100 stays as it is, and you repeat the smaller group's points again and again until you generate 100 samples from it. The final dataset has 100 + 100 = 200 points — both classes balanced at 100 each.

Sense-check: undersampling keeps the minimum class as it is and takes fewer samples from the bigger class; oversampling keeps the bigger class as it is and increases the number of samples from the lower class. A second version of the same idea: one attribute value occurs 8 times and another occurs 4 times; oversampling duplicates the 4-value group until both are balanced.

Strategy	Big class (100)	Small class (10)	Final size
Undersampling	reduced to 10	kept as 10	20
Oversampling	kept as 100	duplicated to 100	200

These strategies exist because of the class imbalance issue — read about it as homework. There are other techniques beyond sampling too, like SMOTE, which generates artificial points to balance the classes: instead of duplicating existing points, SMOTE creates new synthetic points along the line between a minority-class point and one of its nearest minority neighbors, giving the model fresh examples rather than copies.

4.17.3 Sampling at Scale: Questions on Millions of Records

Q: How do we identify a sample in the case of millions of records? A: Go to scikit-learn and search for oversampling and undersampling tools. Typically what we do is this: you have 1 million points, and you say, give me 100 points randomly, and you assume the structure remains as it is. There is no guarantee here — it is an assumption. If you dig deeper, you can look into the attribute values and check that the distribution remains, or you force the distribution to remain by sampling within each stratum. Both variants exist in the tools: plain random sampling and stratified sampling. A lot of the time in practice you decide that randomness is enough; sometimes your requirement forces you to use stratified or cluster sampling.

Q: Does sample identification need to be done manually? A: Depends. Sometimes we do it — sometimes I have done it in the past. Again, there is no thumb rule; it depends on the requirement.

Q: In a real-life problem, if I need to build a model to classify 10 million records spread over 100 or maybe 1000 taxonomy classes, how exactly do I do the sampling, since sampling will again be used in classification? A: Agreed, totally agreed. This is one of the challenges we have in data mining. There is no fixed rule for it. What we have is different techniques to find out what sort of distribution we have in the dataset — exploratory data analysis (EDA) techniques that explain the distribution of various things in the dataset. That lets you take an informed decision about sampling. You do not do it manually; you use statistical techniques to understand the data better, and then design your sampling method based on that. This is a real-life challenge, and there is no fixed thumb rule to solve it.

Pitfalls — the traps beginners fall into.

Undersampling throws away data. Shrinking the big class to 10 discards 90 useful records — fine for balance, wasteful when data is scarce. Oversampling avoids the loss but duplicates points, which can overfit the model to the repeated examples.
Forgetting that "randomness is enough" is an assumption. The million-records answer is explicit: random sampling has no guarantee — verify the distribution survives, or force it with stratification.
Balancing before checking the question. If the business question is about the majority class, aggressive rebalancing can distort the answer. Rebalance when the small class matters, not as a reflex.
Duplicating blindly instead of synthesizing. SMOTE-style synthetic generation is often better than plain duplication when the minority class has very few examples.

Recap and bridge. Sample size is a trade-off: keep enough points to preserve the data's relationships, or the sample is useless. When classes are imbalanced, undersampling shrinks the majority and oversampling grows the minority — SMOTE goes further by synthesizing new points. This closes the sampling story and the preprocessing toolkit of this session: quality, noise, integration, transformation, reduction, and sampling.

Exam note: read about the class imbalance issue to understand why undersampling and oversampling exist, and read about SMOTE, which generates artificial points to balance the classes. Feature reduction will be covered in the next class, and the kNN discussion will show the distance-dominance problem that motivates normalization.

Where this matters in practice. Imbalanced data is the norm in industry, not the exception: fraud is rare, diseases are rare, machine failures are rare, and churners are a minority. Every credit-card fraud model, medical screening model, and predictive-maintenance model faces the 100-versus-10 problem — and the sampling strategies of this section are the first line of defense before SMOTE-style synthesis takes over.

Exam Guidance Summary

There is no mark distribution announced in this session, but the homework and course-planning signals are exam-relevant:

Exam note — homework questions to master.

Z-score versus min-max: does z-score normalization preserve the relationship between the original data points, like min-max does? And when do you use min-max versus z-score? The relationship property is one reason, but there are others. Be ready to reason about both.
Class imbalance: read about the class imbalance issue to understand why undersampling and oversampling exist. Also read about SMOTE, which generates artificial points.
Correlation: the formal definition of correlation is easy to find; the tutorial will cover computing correlation in scikit-learn.

Numerics to watch. Two worked examples were acknowledged as needing correction and are being re-solved in the slides — use the corrected versions:

The z-score example: the mean of 35, 65, 90 is 63.33, not 60; the corrected deviations are −28.33, +1.67, +26.67.
The decimal scaling example: the reference values are range −986 to 917, absolute maximum 986, J = 3, so 917 → 0.917 and −986 → −0.986.

Course planning. Five to six pre-recorded programming tutorials will be posted (module one: Python basics and some TensorFlow; module two: data preprocessing; module three: classification; module four: clustering; module five: applications), and one or two of the three quizzes will be replaced by one or two programming assignments, based on student feedback. Normal classes are unchanged.

What is coming next. Feature reduction will be covered in the next class, and the kNN discussion will show the distance-dominance problem that motivates normalization.

Key Industry Applications

Consolidated list of the real-world connections made in this session:

GPS and sensor data — GPS precision limits (10 kilometres on old hardware versus centimeters today) and water-level sensors with meter-level resolution show how technology constraints inject noise.
Data transmission — phone calls in bad weather show signal noise; the same jitter corrupts data moving between devices and servers.
Retail price data — hand-entered item prices in a mart motivate binning as a smoothing tool.
HR and compensation analytics — the age-versus-salary relationship (older employees tend to earn more) is the running example for regression smoothing.
Medical practice — the doctor who shortlists a few parameters from your long symptom list is the analogy for data reduction.
Distance-based models — kNN-style distance calculations get dominated by large-range attributes (salary versus age), which is why normalization matters before mining.
Dimensionality reduction — PCA and t-SNE as examples of attribute construction in practice.
Correlation analysis in scikit-learn — Python APIs for exploring feature relationships; live dataset merging into a data frame shown in the tutorial.
Class imbalance in industry — real datasets with unbalanced classes (100 versus 10 examples) require undersampling, oversampling, or SMOTE to build usable models.
Sampling at scale — classifying 10 million records across hundreds of taxonomy classes is a real data mining challenge, tackled with EDA-driven sampling decisions.
Climate data — Australian rainfall aggregated monthly versus yearly shows how aggregation changes the standard deviation of a signal.

DM Lecture 4 notes · Data Preprocessing: Noise, Integration, Transformation, and Reduction

Data Mining· postgraduate· 2026-08-05

Sections Breakdown

1Data Preprocessing: Objectives and Data Quality

The two objectives of preprocessing and the six data quality measures.

2Where Noise Comes From

What counts as noise and its five sources: sensors, entry, transmission, technology limits, and merging.

3Smoothing by Binning

The three-step binning procedure with worked examples on bin means and bin boundaries.

4Smoothing by Linear Regression

Fitting a minimum-error line and projecting noisy points onto it to smooth data.

5Outlier Analysis

Outliers versus noise and the cluster-first procedure for spotting suspicious points.

6Manual and Automated Noise Detection

Manual inspection and automated checks: global constraints and functional dependencies.

7Data Integration

Merging multiple sources: entity identification, value conflicts, tuple duplication, and file formats.

8Correlation Analysis

Pearson correlation: positive, negative, and zero correlation as a redundancy and exploratory tool.

9Data Transformation: Why Normalize

Why normalize: the dominance problem of large-range attributes in distance calculations.

10Min-Max Normalization

The min-max formula, worked example, and its relationship-preserving property.

11Z-Score Normalization

Z-score normalization with the corrected marks example and the absolute-deviation variant.

12Decimal Scaling

Decimal scaling by powers of ten with the range -986 to 917 example.

13Aggregation

Changing the scale of the question: quarterly to yearly sales and monthly versus yearly rainfall.

14Data Reduction: Compression and Histograms

Lossless and lossy compression with the two histogram flavors as worked examples.

15Sampling: Simple Random Sampling

Simple random sampling with and without replacement and the representative-subset idea.

16Stratified and Cluster Sampling

Stratified sampling preserves the distribution; cluster sampling picks representatives per group.

17Sample Size and Class Imbalance

Choosing sample size, undersampling and oversampling, and SMOTE for class imbalance.

18Exam Guidance Summary

Homework signals and corrected numerics the professor flagged for exams.

19Key Industry Applications

Real-world connections made across the lecture: from retail pricing to fraud detection.

Postgraduate students in Data Mining

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Data Preprocessing: Objectives and Data Quality

Must-know: Preprocessing serves two objectives: improve data quality, and modify the data so it fits a specific mining technique. Quality depends on the intended user's requirement, not on the data alone.

⚠️ Top pitfall: Assuming data quality is absolute — the same dataset can be high quality for one user and useless for another.

Self-check: Name the six measures used to assess data quality.

Connects to: Where Noise Comes From

Where Noise Comes From

Must-know: Noise is a random error in the measured variable; its five sources are faulty sensors, data entry problems, transmission problems, technology limitations, and naming inconsistencies when merging.

⚠️ Top pitfall: Treating noise as a pattern or a legitimate extreme value — randomness is the defining property of noise.

Self-check: Why does old GPS hardware inject noise even when nobody makes a mistake?

Connects to: Data Preprocessing: Objectives and Data Quality, Smoothing by Binning

Smoothing by Binning

Must-know: Binning is a three-step local smoother: sort, partition into equal-frequency bins, then replace each bin's values by its mean, median, or nearest boundary. It reduces noise impact without identifying which point is noisy.

$mean (B 1) = \frac{4 + 8 + 15}{3} = 9$

⚠️ Top pitfall: Hunting for the noisy point — binning changes every value in the bin; the wider the bin, the stronger the smoothing.

Self-check: For the sorted prices 4, 8, 15, 21, 21, 24, 25, 28, 34 with bins of size 3, what do the three bins become after smoothing by bin means?

Connects to: Smoothing by Linear Regression, Data Reduction: Compression and Histograms

Smoothing by Linear Regression

Must-know: When two attributes are proportionally related, a minimum-error line represents the data; the minor variation between a point and the line is likely noise, and projecting the point onto the line removes it.

$y = x + 1$

⚠️ Top pitfall: Using regression smoothing when the relationship is not linear — a wide cloud of points means the removed variation is real signal, not noise.

Self-check: A point (25, 25.8) is projected onto the line y = x + 1; what value replaces the measured salary?

Connects to: Smoothing by Binning, Outlier Analysis

Outlier Analysis

Must-know: Noise often appears as an outlier, but not every outlier is noise — outliers can be perfectly genuine. The outlier label is about difference; the noise label is about corruption.

⚠️ Top pitfall: Deleting every outlier — legitimate outliers carry real information and must be inspected, not automatically removed.

Self-check: Why does the cluster-first procedure inspect only the points outside all clusters?

Connects to: Where Noise Comes From, Manual and Automated Noise Detection

Manual and Automated Noise Detection

Must-know: Automated noise detection uses global constraints (an attribute must lie within a domain range, e.g., age < 100) and functional dependencies (one attribute validates another, e.g., age < 15 implies salary = 0) to shortlist suspect tuples for inspection.

$if age < 15 then salary = 0$

⚠️ Top pitfall: Relying on manual inspection at scale — it is expensive and tedious and does not scale.

Self-check: A tuple has age = 10 and salary = 8,500. What does the functional dependency age < 15 → salary = 0 say about it?

Connects to: Where Noise Comes From, Outlier Analysis

Data Integration

Must-know: Never merge blindly: check attribute meaning (entity identification problem), tuple representation (data value conflicts), duplication, and file-format uniformity before combining datasets.

⚠️ Top pitfall: Blind concatenation — cust ID and customer ID become two redundant columns, and mixed units (feet vs cm) corrupt the merged table.

Self-check: Table A calls the ID field cust ID and table B calls it customer ID. What problem is this, and what is the fix?

Connects to: Correlation Analysis

Correlation Analysis

Must-know: Correlation ranges from −1 to +1: +1 positive (both rise together), −1 negative (one rises as the other falls), 0 no linear relation. It is an exploratory tool, and correlation does not imply causality.

$r_{A, B} = \frac{\sum _{i = 1}^{n} ( a _{i} - A ˉ ) ( b _{i} - B ˉ )}{n \cdot σ _{A} \cdot σ _{B}}$

⚠️ Top pitfall: Reading causation into correlation — hospitals and car thefts can be correlated because a third attribute (population) drives both.

Self-check: What does r = 0 mean, and why does it not mean the two attributes are independent?

Connects to: Data Integration, Data Transformation: Why Normalize

Data Transformation: Why Normalize

Must-know: Normalization has two purposes: it speeds up the mining process, and it stops attributes with large ranges from dominating distance and other calculations (the dominance problem).

⚠️ Top pitfall: Leaving attributes on wildly different scales — in a Euclidean distance between two points, a salary difference of 1000 overwhelms an age difference of 5.

Self-check: In the distance between (25, 5,00,000) and (30, 5,01,000), which attribute contributes almost the entire distance, and why?

Connects to: Min-Max Normalization, Z-Score Normalization, Decimal Scaling

Min-Max Normalization

Must-know: Min-max normalization maps each value linearly onto a chosen range, and because it is linear it preserves the relationships between original values — relative spacing is only scaled, never distorted.

$v^{'} = \frac{v - min _{A}}{max _{A} - min _{A}} \cdot (new_max - new_min) + new_min$

⚠️ Top pitfall: Future values outside the original min–max range produce out-of-bounds results; min and max are also sensitive to outliers.

Self-check: Normalize age 30 from the range 10 to 80 into the range 0 to 1.

Connects to: Z-Score Normalization, Decimal Scaling

Z-Score Normalization

Must-know: Z-score normalization computes v' = (v − μ_A)/σ_A — subtract the mean, divide by the standard deviation. The corrected marks example: mean 63.33 (not 60), deviations −28.33, +1.67, +26.67.

$v^{'} = \frac{v - μ _{A}}{σ _{A}}$

⚠️ Top pitfall: Using the slipped mean of 60 for marks 35, 65, 90 — the correct mean is 190/3 ≈ 63.33.

Self-check: Why do z-scores always add to zero for a column?

Connects to: Min-Max Normalization, Decimal Scaling

Decimal Scaling

Must-know: Decimal scaling: v' = v/10^J with J the number of digits of the absolute maximum, forcing the column into [−1, 1]. Example: range −986 to 917, abs max 986, J = 3, so 917 → 0.917.

$v^{'} = \frac{v}{1 0 ^{J}}$

⚠️ Top pitfall: Counting digits on boundary cases (e.g., a maximum of exactly 1000) and forgetting to record J so future data is divided by the same 10^J.

Self-check: An attribute ranges from −986 to 917. What is J, and what does 917 become?

Connects to: Min-Max Normalization, Z-Score Normalization

Aggregation

Must-know: Aggregation changes the scale of the data based on the question: quarterly sales of 2008 sum to a yearly total (₹15,68,000), and monthly rainfall averaged yearly keeps the same information but a much smaller standard deviation.

$Sales 2008 = 2, 24, 000 + 4, 08, 000 + 3, 50, 000 + 5, 86, 000 = 15, 68, 000$

⚠️ Top pitfall: Aggregating when a later question needs the fine-grained detail — aggregation is one-way.

Self-check: Why is the standard deviation of yearly rainfall smaller than that of monthly rainfall?

Connects to: Data Transformation: Why Normalize, Data Reduction: Compression and Histograms

Data Reduction: Compression and Histograms

Must-know: Data reduction shrinks volume (rows) or attributes (columns); lossless compression recovers the original exactly (singleton-bucket histogram), lossy compression discards detail that cannot be recovered (range-bucket histogram).

⚠️ Top pitfall: Treating a lossy reduction as lossless — a ranged histogram cannot answer value-level questions.

Self-check: A histogram bucket says '1 to 10' — can you recover the exact original values? What does that make the transformation?

Connects to: Smoothing by Binning, Sampling: Simple Random Sampling

Sampling: Simple Random Sampling

Must-know: An effective sample is representative — it works almost the same as the entire dataset. Without replacement: probabilities are 1/4 then 1/3 and the same point can never repeat. With replacement: every draw keeps probability 1/4 and the same point can repeat.

$P (P 1) = P (P 2) = P (P 3) = P (P 4) = \frac{1}{4}$

⚠️ Top pitfall: Forgetting which flavor was used — and assuming any random sample is automatically representative of the data's properties.

Self-check: From P1–P4, sample size 2, without replacement: if P1 is drawn first, what is each remaining point's probability on the second draw?

Connects to: Stratified and Cluster Sampling, Sample Size and Class Imbalance

Stratified and Cluster Sampling

Must-know: Stratified sampling keeps each stratum's share in the sample proportional to its share in the original (6 : 8 : 2 halves to 3 : 4 : 1); cluster sampling picks representative points from each cluster, shrinking 100 points to five or ten representatives.

⚠️ Top pitfall: Confusing stratification (mirrors the original distribution) with balancing (equalizes groups) — they answer different questions.

Self-check: Original counts are youth 6, middle age 8, senior 2. What is the correct stratified sample of size 8?

Connects to: Sampling: Simple Random Sampling, Sample Size and Class Imbalance

Sample Size and Class Imbalance

Must-know: Sample size must preserve the data's relationships (8000 → 2000 keeps them; too small destroys them). Undersampling shrinks the big class (100 vs 10 → 20 points); oversampling duplicates the small class (→ 200 points); SMOTE generates synthetic points.

⚠️ Top pitfall: Building a model on imbalanced data — the model learns to predict the majority class and learns almost nothing about the minority.

Self-check: Classes of 100 and 10 points: what are the final sizes after undersampling and after oversampling?

Connects to: Sampling: Simple Random Sampling, Stratified and Cluster Sampling

Exam Guidance Summary

Must-know: Use the corrected numerics: z-score mean 63.33 (not 60) and decimal scaling range −986 to 917 with J = 3. Be ready to reason about whether z-score preserves relationships and when to use min-max versus z-score.

⚠️ Top pitfall: Carrying the slipped class arithmetic (mean 60) into answers — the corrected values are promised in the slides.

Self-check: Which two worked examples were acknowledged as needing correction in this session?

Connects to: Min-Max Normalization, Z-Score Normalization, Decimal Scaling, Sample Size and Class Imbalance

Key Industry Applications

Must-know: Each preprocessing technique maps to a named industry use: binning for retail prices, regression smoothing for age–salary analytics, normalization before distance-based models (kNN), and EDA-driven sampling for classifying millions of records.

Self-check: Which industry scenario motivates binning as a smoothing tool?

Connects to: Where Noise Comes From, Smoothing by Binning, Smoothing by Linear Regression, Data Transformation: Why Normalize, Aggregation, Sample Size and Class Imbalance

Previous LectureLecture 03 · Data and Data Preprocessing

Next LectureLecture 05 · Data Reduction, Discretization, and Classification

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.