—

Data Mining

Data Mining Fundamentals

Published: 2026-08-05

Level: postgraduate

Audience: Postgraduate students in Data Mining

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Origins and motivations of data mining — covered in Lecture 1 (Origins and Motivations of Data Mining)
Formal definition of data mining — covered in Lecture 1 (What Is Data Mining: Definition and Boundaries)
KDD pipeline and feedback loops — covered in Lecture 1 (The Knowledge Discovery in Databases (KDD) Process)
Predictive versus descriptive methods — covered in Lecture 1 (Fundamental Taxonomy of Data Mining Functions)

These notes cover the foundations of data mining: why the field matters, what the term means, how a data mining project flows from raw data to knowledge, and the two big families of tasks — prediction and description. Within prediction we meet classification and regression. Within description we meet clustering, association rule mining, and anomaly detection. Each task comes with real-world applications, from iPhone advertising to airport design. The notes end with the first steps of preparing data: how datasets come from physical entities, and how we talk about objects and attributes.

By the end of this session you should be able to answer four questions: What is data mining, and why does it exist? What is the pipeline that turns raw data into knowledge? Which mining task fits which problem — predict a label, predict a number, or describe structure? And where do datasets actually come from?

2.1 Why Data Mining

2.1.1 Three Reasons Data Mining Is a Big Deal

Hook: Imagine a machine that fills a library with books every day, with nobody writing any of them. That is roughly what has happened to data: the world now produces it automatically, faster than humans can read it. This session opens with the three trends that made data mining a field.

The class opened with a quick revision of three reasons the field has grown so fast.

The first reason: automatic data collection has improved drastically in the last few decades. Sensors collect huge volumes of data on their own, with no human typing anything in. A smartphone knows your location every few seconds; a supermarket scanner records every item that passes the checkout; a weather station logs temperature and wind speed around the clock. In each case the data is produced by a device, not by a person with a pen.

The second reason: storage has improved just as fast. Storage capacity that used to be measured in megabytes is now gigabytes and terabytes inside an ordinary laptop or phone. A few decades ago, storing even a million records was a serious engineering problem; today a single compact disc holds more than that, and a phone in your pocket holds thousands of times more. Database systems have also improved considerably, so once data is collected, it can be stored and organized properly — and, crucially, retrieved again when you want to analyze it.

The third reason: many physical interactions we used to perform face to face have migrated to computerized or online systems. We used to buy books from local physical stores; now we order from Amazon or Flipkart. We used to walk into a bank for every transaction; now we have internet banking and stock transactions online. The complete experience moved to a computerized version, and every one of those systems generates data. Every search, every purchase, every login leaves a digital trail.

Putting the three trends together. Take one ordinary day in a single customer's life:

Morning: you check the weather app on your phone — the forecast system collects your location and query (trend 1: sensors and devices).
Afternoon: you buy groceries with a card — the store records every item, price, and time (trend 3: online migration).
Evening: you stream a movie — the platform stores your choice, watch time, and pause points.

None of this needed a data-entry clerk. All of it is stored (trend 2) and all of it is available tomorrow for analysis. Put those three trends together and the result is simple: there is tons of data.

2.1.2 Drowning in Data, Thirsty for Answers

Here is the motivating tension of the whole course. We are drowning in data, but if we ask relevant questions of the data, can we get proper answers? That question is the reason data mining exists.

Intuition. Being rich in data is not the same as being rich in answers. A shop with a million sales records can still not know which customers will return next month — the records exist, but the answer has to be dug out of them. Data mining is the discipline that does the digging: it takes the mountains of stored records and answers questions that nobody typed anywhere.

The evolution of science itself has been pushed along by this ability to turn collected data into answers. Astronomy is a famous example: centuries of star observations sat in tables before patterns in them — planetary motion, variable stars, the expansion of the universe — were extracted and turned into theories. The same story repeats in medicine, finance, and commerce today: the observations pile up faster than ever, and the people who can turn them into answers are the ones who win.

Recap + bridge. Three trends — automatic collection, cheap storage, and the migration of daily life online — created the modern flood of data. The flood raises one question that drives everything in this course: can we get proper answers from all this data? The next section defines the activity that answers it: data mining.

Data mining sits in the middle of this story. The data is already there — the question is how to get knowledge out of it. That is the subject of the next sections: first the definition of the field, then the pipeline that produces knowledge, then the specific tasks.

2.2 What Is Data Mining

2.2.1 The Definition

Definition. Data mining is the extraction of non-trivial, implicit, previously unknown, and potentially useful patterns, structure, and knowledge from huge amounts of data.

Every adjective in that definition does work. Drop any one of them and the sentence stops describing data mining:

Non-trivial — the pattern is not something you would notice by eyeballing a small sample. A trivial statement ("customers buy bread") is not data mining; the value is in what a person would not spot on their own. The pattern must be surprising enough that a human analyst would not have seen it just by looking.
Implicit — the knowledge is hidden inside the data. It is not written down anywhere; the algorithm has to dig it out. Nobody publishes a booklet saying "customers who buy diapers also buy beer"; that fact is implicit in millions of receipts.
Previously unknown — the pattern is new information. If everyone in the company already knows it, mining it adds nothing. Re-mining a rule the marketing team has used for ten years is waste.
Potentially useful — the pattern can drive a decision or an action, even if usefulness is not proven the moment it is found. A rule that rearranges a shop's shelf layout or changes a coupon offer has practical value; a pattern that leads nowhere useful is not worth mining.

So the definition says: we are extracting pattern, structure, and knowledge from data, and the extraction is worthwhile only when the result is non-trivial, implicit, previously unknown, and potentially useful.

Testing the definition. Consider three statements a supermarket could make about its sales data:

Statement	Non-trivial?	Implicit?	Previously unknown?	Useful?	Data mining?
"We sell bread every day."	No — visible in any small sample	No — it is obvious	No	Mildly	No
"On days when the temperature crosses 40 °C, ice-cream sales double."	Yes — needs many records to see	Yes — buried in the data	Yes — nobody tracks it manually	Yes — plan stock	Yes
"In winter, weekday sales peak at 7 pm."	Yes	Yes	Maybe	Yes	Yes

The same data can yield trivial statements (not mining) and non-trivial ones (mining). The four adjectives are the filter that separates them.

2.2.2 Data Mining's Roots in Other Fields

Data mining does not come from a single subject. It draws ideas from multiple domains: machine learning, pattern recognition, statistics, AI, and database systems.

Machine learning supplies the algorithms that learn models from data — decision trees, neural networks, clustering methods.
Pattern recognition supplies methods for recognizing structure in signals and images.
Statistics supplies the formal language of probability, sampling, and significance — how confident can we be in a pattern?
AI supplies the broader goal of building systems that behave intelligently with knowledge.
Database systems supply the technology for storing, indexing, and querying the huge amounts of data in the first place.

Because of that mix, the field also goes by various other names. If you see "knowledge discovery," "pattern analysis," or similar labels in books or job posts, they are describing essentially the same activity under a different name.

Why the many names matter in practice. When you read a research paper titled "knowledge discovery" and a job posting for a "pattern analyst," the work is the same family of activity. The name varies by community — database researchers often say knowledge discovery in databases, statisticians say data analysis, machine learners say learning from data. Learning the synonyms is not pedantry; it is how you recognize the field in the wild.

Pitfall: mistaking the term for the trick. A common beginner error is treating "data mining" as a single algorithm. It is not — it is a family of techniques that borrow from five parent fields. When someone says "we will data-mine this," the useful question is always which technique: prediction or description, classification or clustering? The next sections draw that map.

Recap + bridge. Data mining = extraction of non-trivial, implicit, previously unknown, and potentially useful knowledge from huge data. It is a hybrid field with five parents and many names. Knowing what it is matters less than knowing how the work flows — which is exactly what the pipeline in the next section describes.

2.3 The Data Mining Process Pipeline

2.3.1 The Standard Pipeline

Purpose. Data mining is a process, not a single algorithm. A mining algorithm is one stage of a longer journey; the journey itself — from raw data to usable knowledge — has a fixed shape, and understanding that shape is what makes the later stages of the course navigable.

The typical pipeline has six stages:

Data — you start with raw data. This is the unprocessed output of the real world: sensor readings, sales receipts, log files.
Data selection — you pick the subset of the data that is relevant to the question at hand. If the question is about young customers, you keep the relevant records and set the rest aside.
Preprocessing — you clean and organize the selected data so that it is ready to use: filling missing values, fixing inconsistencies, removing obvious errors.
Transformation — you bring the data into a form that machine learning or data mining algorithms can consume — for example, turning text categories into numbers, or scaling values into a common range.
Data mining — you run the mining algorithms and get results: clusters, rules, or a trained classifier.
Interpretation — interpreters (humans or tools) make sense of the results and decide whether the patterns are real, useful, and actionable.

The output of the whole pipeline is knowledge.

Trace — a retail example. A supermarket chain wants to know what to stock together. Walk the six stages:

Data: every checkout receipt for the last year, stored in the billing system.
Selection: only receipts containing at least two items, from the two largest stores, to keep the analysis focused.
Preprocessing: remove cancelled receipts, fix receipts with negative quantities, fill in the missing product codes from the catalog.
Transformation: convert each receipt into a row with one column per product, value 1 if bought and 0 if not.
Mining: run an association rule algorithm over the transformed table.
Interpretation: a human looks at the top rules, checks they make business sense, and tells the store manager which products to place together.

The knowledge that comes out — "customers who buy X also buy Y" — is the product of all six stages, not of step 5 alone. If any earlier stage is sloppy, the final rules are suspect.

2.3.2 The Full Knowledge Discovery Loop

A fuller version of the picture starts from multiple data sources. The flow is: data from multiple sources goes through data integration and data cleaning into a data warehouse. From the warehouse you perform data selection to get a subset of the data. Then you apply the data mining technique to find patterns and structure. Then you check whether you are getting useful knowledge.

Intuition. Think of the full loop as cooking from many ingredients rather than opening one tin. The raw ingredients are scattered across different kitchens — sales systems, web logs, supplier records. Data integration merges them into one store (the warehouse), data cleaning removes the rotten bits, and only then does the actual mining — the cooking — begin. The dish is the knowledge; the chef checks whether it tastes right before serving it.

The key loop — feedback, not conveyor belt. If the knowledge is not useful, you do not just accept it. You go back and change the mining algorithm, or change the selection technique, or revisit the cleaning and integration steps, and try again until the answers are proper. The whole chain — preprocessing steps, mining steps, post-processing steps — is one feedback loop, not a one-way conveyor belt.

Concretely: a bank's first attempt to detect fraud with one algorithm produces too many false alarms. The bank does not ship the results; it loops back — different selection of features, stricter cleaning of the transaction data, a different mining technique — and reruns. This loop is where most real data mining time is actually spent.

Recap + bridge. The pipeline is: raw data → selection → preprocessing → transformation → mining → interpretation → knowledge. The fuller version adds integration, a warehouse, and — decisively — a feedback loop when the knowledge is not useful. With the process in place, the next question is what kinds of tasks the mining stage can perform: the two great families, prediction and description.

2.4 Predictive and Descriptive Functions

2.4.1 Two Families of Tasks

Every data mining function falls into one of two categories.

Predictive functions predict something for you. You feed the model the current facts and it tells you about a value or a label you do not know yet. The output points forward in time or to a missing value: "this customer will buy" or "tomorrow's price will be 105."

Descriptive functions do not predict anything. They find human-interpretable patterns and structure in the data. The deliverable is understanding, not a forecast. The output is a description of what is in the data: "these customers form a group," "these items co-occur."

Intuition. Predictive methods are like a weather forecaster: they look at today and tell you about tomorrow. Descriptive methods are like a mapmaker: they look at the whole territory and draw where the regions are. Both add value, but in different ways — the forecaster's value is the forecast, the mapmaker's value is the structure.

2.4.2 The Prediction Family

Two canonical predictive methods appear again and again:

Regression — the output is a continuous value. Predicting tomorrow's petrol price is regression, because the answer can be any value in a range (say 90 rupees, 100 rupees, 110 rupees, or 150 rupees). The output is a number with real magnitude — 104.5 is a meaningful answer.
Classification — the output is a class label. Predicting whether the weather will be hot, moderate, or cold is classification, because the answer must land in one of a few named buckets.

Both are discussed in detail in the next sections.

2.4.3 The Description Family

On the descriptive side:

Clustering — group data points so that points in the same group look alike and points in different groups look different.
Association rule learning — find rules that connect items, such as "customers who buy product X tend to also buy product Y."

The mapping to remember: classification is a predictive method, regression is a predictive method, clustering is a descriptive method, and association rule learning is a descriptive method. Getting this taxonomy straight is the foundation for everything that follows.

Pitfall: labeling by the word, not by the output. Students often misclassify clustering as predictive ("it groups future customers, so it predicts!"). Clustering outputs a grouping of existing points; it never outputs a value or label for a new observation. The test is always the same: does the method forecast something unknown about a new case (predictive), or does it describe structure in what we already have (descriptive)?

Dimension	Prediction family	Description family
Question answered	"What will the value or label be?"	"What is the structure?"
Output	A number (regression) or a class label (classification)	Groups (clustering) or rules (association)
Target of the answer	Something not yet known — a new or future case	Existing data, organized for human understanding
Typical use	Decide, forecast, automate	Understand, segment, explore

When to pick which: if the business question names an unknown value or label — how much, which class — pick a predictive method; if the question is "what hangs together in this data," pick a descriptive method.

Recap + bridge. Four tasks, two families: regression and classification predict (values and labels); clustering and association rule learning describe (groups and rules). The rest of this session walks through each of the four in turn, starting with classification — the predictive method built on class labels.

2.5 Classification

2.5.1 What Classification Does

Definition. Classification predicts a class label. A class label is one of a fixed, finite set of named categories — the answer must be exactly one of the names, nothing in between.

Imagine a dataset in tabular form where the class label is "cheat" — you are building a cheat detection system that decides whether a particular customer or user is a cheater. There are two classes: a person can be "cheat" (yes) or "no cheat" (no). Two class labels means binary classification.

The setup generalizes: three class labels give three-class classification, and n class labels give n-class classification. The number of classes is finite, and the classes are very different from one another. That finiteness and distinctness is what separates classification from regression: classification predicts one of a fixed set of named classes, while regression predicts a continuous value.

A concrete setup used throughout the discussion: rows represent different people (say 10 people), and columns represent attributes that describe each person. The first attribute is an ID, a unique identifier for the customer. Other attributes capture properties like refund history and marital status, and a final column holds the class label. This kind of table is the raw material for building a classification model.

Intuition. Classification is sorting into labelled boxes. The boxes are fixed in advance — "cheat" and "no cheat", "hot", "moderate", "cold" — and the model's only job is to decide, for each new person or day, which box it belongs in. Nothing is invented along the way; the answer is a choice among the boxes that already exist.

2.5.2 Binary, Three-Class, and N-Class

The weather prediction example from the earlier session shows the bucket idea. Weather can be quantified into three buckets: hot, moderate, or cold. You can make your own buckets — two, three, or more — and your prediction function will place each day into one of the three classes. That is a three-class classification problem. The same pattern scales to any n.

Counting classes. The number of classes defines the problem's name:

2 classes → binary classification. Example: a credit card transaction is fraudulent or non-fraudulent.
3 classes → three-class classification. Example: tomorrow's weather is hot, moderate, or cold.
n classes → n-class classification. Example: recognizing a handwritten digit is a 10-class problem — the digit must be one of 0, 1, ..., 9.

A fraud model output of "0.6 fraudulent" is impossible — the model must commit to one of the two labels. That commitment is what makes it classification. (How a model converts confidence into a committed label is a later-module question; for now, the output is a label.)

Real-world: the fraud setting is binary — a transaction is fraudulent or non-fraudulent. The weather setting is three-class. Everything in between scales to n classes. The rule of thumb: if the answer is one of a small fixed set of labels, it is classification.

2.5.3 Student Questions and Answers

Q: Is the result only categorical? But the input may be continuous.

A: Correct — the result is categorical, and the input may be continuous. We will come back to this point, but hold the thought: classification constrains the output to classes, not the input. Boolean output — yes or no, 0 or 1 — is classification.

Q: So Boolean output is classification, typically 0 or 1?

A: Yes, exactly. As you said, classification means predicting classes, not values. You can predict out of two classes — binary classification, like fraudulent versus non-fraudulent transactions — or three classes, like hot, cold, or moderate weather, or n classes. The classes are finite and distinct. Regression, by contrast, predicts a value of continuous nature.

Pitfall: confusing the output type with the input type. A student sees a continuous input — income of 125k — and wonders whether the problem stopped being classification. It did not. The thermometer is not the forecast: classification is defined by what comes out (a label), never by what goes in (which may be numbers, categories, or both).

Exam note. Classification predicts one label from a finite set of distinct classes; binary = 2 classes, three-class = 3, n-class = n. The input may be continuous or categorical — only the output is constrained. Be able to classify new problems by this test, and to name real binary (fraud), three-class (weather), and n-class (handwritten digits) examples.

2.6 The Classification Dataset: Attributes, Train and Test

2.6.1 The Cheat Detection Data

Here is the sample dataset used to illustrate classification in full. There are 10 people, each represented by a row. The columns are attributes:

ID	Refund	Marital Status	Taxable Income	Cheat
1	Yes	Single	125k	No
2	No	Married	100k	No
3	No	Single	70k	No
4	Yes	Married	120k	No
5	No	Divorced	95k	Yes
6	No	Married	60k	No
7	Yes	Divorced	220k	No
8	No	Single	85k	Yes
9	No	Married	75k	No
10	No	Single	90k	Yes

ID — a unique identifier for identifying the customer.
Refund — whether the person has refunded a previous loan. It takes two values: Yes or No.
Marital Status — whether the customer is single, married, or divorced.
Taxable Income — a dollar amount describing the customer's income.
Cheat — the class label, taking the values Yes (cheat) or No (not cheat).

The class label has only two values, so this is a binary classification problem.

Reading the table. Each row is one customer — one object; each column is one attribute. Notice the difference between the first column and the rest: ID (1, 2, 3, ...) only names the row; it says nothing that could help predict cheating. The label column (Cheat) is special too — it is the answer we want the model to learn to produce. The three middle columns are the ones the model will actually reason with.

2.6.2 Categorical and Continuous Attributes

The dataset also introduces the two broad kinds of attributes, with an early taste of a rule you will use constantly:

Categorical attribute — an attribute with a fixed number of values. Refund has two unique values (Yes, No). Marital status has three unique values (single, married, divorced). The count of unique values is one clue for deciding whether an attribute is categorical.
Continuous attribute — an attribute whose value can be anything in a range. Taxable income can range from roughly 75k to 125k. There is an infinite number of unique values possible in a continuous attribute.

The number of unique values an attribute has helps determine what type of attribute it is. That theme is developed in much more detail in the next class, when the full attribute taxonomy is covered.

Rule of thumb. Count the unique values. A small, fixed count (2 values for Refund, 3 for Marital Status) says categorical. An effectively endless count (income can be 74,321.50 or 74,321.51) says continuous. This counting trick is the first tool for typing any attribute you meet.

2.6.3 Training Data and Test Data

Once you have this dataset built from existing customers, you can build a classification model. The left table is called the train data, or training data. You build the model on the training data and learn from it. When training is done, you check how good the model's performance is on the test data — the right-hand table, where the class labels are not shown and the model must make predictions. The test data is how you judge whether the model is good or bad.

Intuition — study and exam. Training is the study phase: the model sees both the attributes and the correct labels and learns the pattern connecting them. Testing is the exam phase: the model sees only the attributes and must produce its own labels, which are then checked against the true answers. A model judged only on the questions it has already seen is a student graded on its own homework — it could have memorized instead of learned. That is why the test data is held out.

2.6.4 Worked Example: Predicting New Customers

Worked example. Suppose customer 11 arrives with values: refund = No, marital status = Single, taxable income = 75k. The class label is unknown — we do not yet know whether this customer will cheat.

The training phase already happened: the model studied customers 1–10 and their Cheat labels.
Customer 11 is one row: attributes (Refund = No, Marital Status = Single, Taxable Income = 75k); the Cheat column is empty.
The trained model maps those attributes to a label — in this dataset's pattern, single customers with mid-range income like customers 8 and 10 are the cheaters, so a reasonable model predicts Cheat = Yes.
The same happens for customers 12, 13, 14, and so on: each new row goes through the trained model, and the model outputs a predicted class label for each one.

The predicted label is the answer for customer 11. Whether that prediction is right is checked later, when the true label is discovered — just like a test answer is graded only when the answer key arrives.

That is the entire workflow: learn from labeled past data, then label unlabeled new data.

Pitfall: letting the ID column into the model. The ID column (1, 2, 3, ..., 10) has a unique value per row — under the "count unique values" rule it can even look categorical. But a column that merely names the row carries no predictive information: no pattern connects "customer 7" to cheating. Including such columns teaches the model noise, not signal. The classic early mistake in real projects is training on every column, identifier included.

Exam note. Be able to (1) explain each column of the cheat-detection table, (2) classify Refund and Marital Status as categorical (2 and 3 values) versus Taxable Income as continuous (infinite values), and (3) state the train/test split: model learns from training data with labels, then predicts labels for test data without them, and the test performance judges the model. The customer-11 walkthrough is the canonical exam-style flow.

2.7 Classification Applications

2.7.1 Direct Marketing

Real-world: imagine launching a new iPhone. You want to advertise it, and the potential customer base is the whole world. But advertising to everyone costs too much. You want selective advertisement: pick only the customers most likely to buy the new iPhone and advertise to them; skip the rest.

How do you identify those customers? You build a classification model. You have existing data where customer 1, customer 2, customer 3 are rows, and the columns are attributes like income, how many iPhones the person has owned previously, where the person lives, whether the person owns a car, and so on. The model learns the pattern connecting these attributes to buying behavior. Then, for each new customer, the model predicts whether this person is most likely to buy the new iPhone. The selected customers get the ads; the rest are skipped. A very simple, very profitable example of classification.

Worked numbers. Suppose a phone maker's model has two classes — will buy and will not buy — and scores every customer in its database of 10 million:

Attributes per customer: income bracket, number of previous devices owned, city, car ownership, age.
The model predicts will buy for the top 1 million customers.
Ads are sent only to those 1 million. At a marketing cost of 10 rupees per ad, the bill is 10 million rupees instead of 100 million for the full 10 million.

The saving comes straight from classification: the model concentrates the ad spend on the customers most likely to respond. Same budget, higher response rate — that is why direct marketing is one of the oldest profitable uses of classification.

2.7.2 Fraud Detection

Real-world: you have all used credit cards and debit cards. Whenever you use a card at a POS terminal or with any vendor, a transaction is done. Before the transaction completes, an automated system checks whether the particular transaction looks fraudulent or not. This is a binary classification problem: a transaction is fraudulent or non-fraudulent (benign).

The system works from your behavior profile. Two scenarios make this concrete:

Scenario 1 — amount anomaly. Suppose over five years you typically spend in the thousands. Suddenly you swipe your card for a 10 lakh transaction. That looks like a fraudulent transaction, because your normal behavior is small transactions and this one breaks the pattern.

Scenario 2 — location anomaly. You normally make transactions from a particular location, say inside Goa. Suddenly a transaction comes from the US. That transaction looks fraudulent, because your user profile says you transact in Goa and here is a transaction from the US.

You have seen this system yourself. You log into your Gmail from your office, and the experience is smooth. One day you log in from an offsite location or the US, and suddenly extra authentication appears: an OTP is sent, or verification through another email is required. Why? Because that login is not your normal behavior. It looks suspicious, so additional checks are put in place to decide whether this login is fraudulent or not.

Worked numbers. A bank builds a two-class model per customer — or a shared model per profile — where the attributes of a transaction are amount, time, merchant category, and location:

Attribute	Customer's normal profile	The flagged transaction
Amount	₹2,000–₹8,000	₹1,000,000
Location	Goa	United States
Hour	10 am – 9 pm	3 am

The new transaction is far outside the profile on every dimension, so the model assigns it a high fraud probability and blocks it for review. The blocking happens in seconds — before the transaction completes — which is exactly the timing the professor emphasized: the check runs automatically at the moment of the swipe.

The punchline: when you swipe a card, the system predicts, based on your previous behavior, whether this is a genuine transaction or a fraudulent one. The profile is built from credit card history — what the account buys, when it buys, whether payments are on time, what kinds of transactions occur — and every new transaction is judged against that profile.

2.7.3 Customer Churn Prediction

Real-world: customer attrition, or churn, is another huge classification application. Your HR department might do this job: based on your profile and current behavior, they predict whether you will stay in the company or leave.

The toothpaste story makes it vivid. Ten years ago many people used Colgate. Today these companies continuously check whether a customer is staying with the toothpaste brand or not. When Patanjali entered the market and started acquiring customers, Colgate's customer attrition became very high, because many Colgate customers switched to another brand. This is a classification problem: based on customer behavior, predict whether the customer stays with the brand or leaves.

The retention loop: if the model says the customer is likely to be lost to a competitor, the company gives additional benefits. Think about when Ola entered the Indian market, or when Amazon and Flipkart competed to retain customers. They pushed out lots of offers: use this coupon and get a 10% discount; use this coupon and get 100 rupees in your account. Behind those coupons was a system predicting whether the customer is going to stay on the platform or be lost to a competitor. If the system says the customer is likely to leave, a coupon is pushed so the customer stays. Binary classification: stay or leave, two classes.

Pitfall: treating the coupon as marketing luck. The coupons that arrive in your inbox the week you stop ordering are not random generosity. They are the output of a churn classifier: your usage pattern crossed the model's "likely to leave" threshold, and the retention offer fired automatically. Understanding this loop matters because the same pattern — model predicts, action fires — repeats across every retention system you will meet.

Recap + bridge. Three applications, one method: direct marketing (send ads only to likely buyers), fraud detection (block transactions that break the behavior profile), and churn prediction (offer benefits before the customer switches). All three are binary classification — two classes each. The next section turns to the other predictive member of the family: regression, which answers with a continuous number instead of a label.

2.8 Regression

2.8.1 Predicting Continuous Values

Regression is the second predictive system. It predicts a continuous value — a number that can take any value in a range.

Definition. A continuous value is a number that can be any real value within a range, including fractional values. If the answer to your prediction question is "a number," regression is the family; if the answer is "one of a fixed set of names," it is classification. The type of the output decides, never the type of the inputs.

If you build a system that predicts tomorrow's petrol price, the answer can be 90 rupees, 100 rupees, 110 rupees, or 150 rupees — any value in the range. Because the output is a continuous value, it is a regression model. Same for predicting oil price: continuous. Same for tomorrow's temperature: the temperature can range from minus 100 to plus 100 or anything in between, so it is a regression problem.

Worked numbers — petrol price. Suppose yesterday's price was ₹104 and the model's answer for tomorrow is ₹106.50. Notice what this answer is not:

It is not a choice between "cheap" and "expensive" — that would be classification.
It is a real number with a real magnitude: ₹106.50, ₹107.20, and ₹105.75 are all legal answers, and their ordering and distance carry meaning.

The model's output — ₹106.50 — is a point in a continuous range of possible prices. That is the entire signature of regression.

2.8.2 Regression Examples

The examples pile up easily:

Petrol price tomorrow — a continuous value.
Oil price — a continuous value.
Tomorrow's temperature — a continuous value.
Rainfall prediction — how much rain will fall this month, whether it is 10 centimeters, 50 centimeters, or 500 centimeters. A continuous value, so regression.

The contrast to keep in your head: two types of prediction exist. Regression predicts continuous values. Classification predicts a class label. After these introductory sessions, classification is covered in much more detail in module 4, association rule mining in module 5, and clustering in module 6. Right now the goal is motivation — showing how these problems appear in the real world.

Dimension	Classification	Regression
Output	A class label from a finite set (hot / moderate / cold)	A continuous number (₹106.50)
Number of possible answers	Fixed, small, named	Infinite values in a range
Example	Fraudulent vs non-fraudulent transaction	Tomorrow's petrol price
What a wrong answer looks like	Wrong bucket	Off by an amount

When to pick which: the output decides — if the question names a fixed set of labels, classify; if it names a measurement, regress.

Exam note. Regression predicts a continuous value: petrol price, oil price, temperature, rainfall — any number in a range. Classification predicts a class label: hot/moderate/cold, cheat/no-cheat. Be ready to classify new examples by asking only about the output type. Module map for later: classification → module 4, association rules → module 5, clustering → module 6.

2.9 Clustering

2.9.1 The Formal Definition

Hook. After 1947, the map of the subcontinent was redrawn into provinces — not by random lines, but by grouping towns whose people spoke the same tongue. That is clustering: turning a map full of villages into a small number of meaningful regions. The professor uses exactly this example, and it is worth holding onto — it is the clearest picture of the method anywhere in the course.

Clustering is the first descriptive method. Formal definition: given a set of data points — a huge amount of data represented as points, each having attributes — and a similarity measure among them, find groups called clusters such that:

data points in one cluster are more similar to one another, and
data points in different clusters are very different from one another.

Two properties make a clustering valid. First, inside one cluster, the points look very similar to each other. Second, take a point from cluster C1 and a point from cluster C2 — they should look very different. Both directions matter: similarity inside a cluster and dissimilarity between clusters.

Definition. A cluster is a group of data points such that points inside the group are mutually similar and points in different groups are mutually dissimilar. Both halves of that sentence are mandatory: a grouping that only ensures internal similarity (or only external difference) is not yet a good clustering.

Note what is missing: no prediction. Clustering is not predicting anything for you. It is finding patterns and structure in the data that humans can interpret. That is what descriptive methods do, always.

2.9.2 Similarity and Dissimilarity: Distance Measures

How do you decide whether two points are similar? You use distance as a measure. The classic choice is Euclidean distance — the straight-line distance between two points. For two points $x$ and $y$ in an n-dimensional space, with coordinates $x_{i}$ and $y_{i}$ along each dimension, Euclidean distance is:

$d (x, y) = i = 1 \sum n (x_{i} - y_{i})^{2}$

Here $x = (x_{1}, x_{2}, \dots, x_{n})$ and $y = (y_{1}, y_{2}, \dots, y_{n})$ are n-dimensional vectors, one per data point, and the sum runs over all n dimensions. Smaller distance signals more similar points.

Building the formula. The formula is just the Pythagoras rule you know from school, written for n dimensions:

In 1 dimension, two points $x$ and $y$ are $∣ x - y ∣$ apart.
In 2 dimensions, the straight-line distance between $(x_{1}, x_{2})$ and $(y_{1}, y_{2})$ is $(x_{1} - y_{1})^{2} + (x_{2} - y_{2})^{2}$ — the hypotenuse of the right triangle whose two legs are the horizontal and vertical gaps.
In n dimensions, the same idea: square each of the n coordinate gaps $(x_{i} - y_{i})^{2}$ , add all n squares, take the square root. The result $d (x, y)$ is one non-negative number — the straight-line distance.

Every symbol: $d (x, y)$ is the distance between the two points; $x = (x_{1}, \dots, x_{n})$ and $y = (y_{1}, \dots, y_{n})$ are the two n-dimensional vectors; $x_{i}$ and $y_{i}$ are their coordinates on dimension $i$ ; $n$ is the number of dimensions (attributes); the sum $\sum_{i = 1}^{n}$ runs over every dimension.

Why the square root? The squares make all gaps positive so nothing cancels out, and the square root returns the answer to the original units — if the coordinates are in centimeters, the distance is in centimeters. (The standard reference form confirms this: $d (x, y) = \sum_{k = 1}^{n} (x_{k} - y_{k})^{2}$ , where $n$ is the number of dimensions and $x_{k}, y_{k}$ are the kth attributes of the two points.)

Worked example with real numbers. Take two people from the cheat-detection table as points with two attributes each: $A = (income, refund)$ -style coordinates are not numeric here, so use a cleaner pair. Let $x = (2, 0)$ and $y = (5, 1)$ — think of x-coordinate as income in lakhs and y-coordinate as hours of use, say.

Step by step:

Gaps per dimension: $(x_{1} - y_{1}) = 2 - 5 = - 3$ , $(x_{2} - y_{2}) = 0 - 1 = - 1$ .
Squares: $(- 3)^{2} = 9$ , $(- 1)^{2} = 1$ .
Sum: $9 + 1 = 10$ .
Square root: $10 \approx 3.16$ .

$d (x, y) = 10 \approx 3.16$ . A third point $z = (2, 1)$ gives $d (x, z) = (2 - 2)^{2} + (0 - 1)^{2} = 0 + 1 = 1.0$ , so $z$ is much closer to $x$ than $y$ is — under this measure, $x$ and $z$ would cluster together before either pairs with $y$ . Sense-check: the straight-line distance between the two points should indeed be larger than 1 but comparable to the coordinate gaps — $10 \approx 3.16$ sits between the horizontal gap (3) and the diagonal, exactly as geometry demands.

Hamming distance was also named as another distance option. And the choice is problem-specific: you can design your own measure when a standard one does not fit the data.

Intuition — two rulers, one job. Euclidean distance is the measuring tape: the straight line between two points, good when coordinates are continuous numbers like income or height. Hamming distance is the counting rule: it counts how many coordinates differ between two objects, and is natural when attributes are binary — for example, two customers described by "owns a car: yes/no" over 10 features have a Hamming distance equal to the number of features on which they disagree. Both answer "how far apart?"; the right ruler depends on what kind of attributes the data has, and nothing stops you from designing your own when neither fits.

2.9.3 Worked Example: Forming the Indian States on Language

A real historical example of clustering: the formation of the Indian states after independence.

At independence there were many more states than today — around 200 states and princely territories, with many rulers and many local governments. The government of India decided to group the landmass based on an identity, and the identity chosen was language.

Think of each district, village, and town as a data point. The attribute is the language its people speak. Two points are similar when they share a common language, so they belong in one cluster; points that speak different languages belong in different clusters. Each cluster became a state:

Marathi speakers → Maharashtra
Gujarati speakers → Gujarat
Konkani speakers → Goa
Malayalam speakers → Kerala
Tamil speakers → Tamil Nadu
Kannada speakers → Karnataka
Telugu speakers → Telangana
Punjabi, Ladakhi, and Kashmiri speakers → their respective regions

Inside a cluster the points are very similar because of the common language. A point in Maharashtra and a point in Punjab are very different from each other because the languages differ. The notion of similarity and the simultaneous notion of dissimilarity both did the work.

Worked example — the cluster logic. Take three villages: $V_{1}$ in what became Maharashtra, $V_{2}$ also in Maharashtra, and $V_{3}$ in Gujarat.

Similarity: $V_{1}$ and $V_{2}$ share the Marathi language → same cluster → same state.
Dissimilarity: $V_{1}$ (Marathi) and $V_{3}$ (Gujarati) share nothing on the language attribute → different clusters → different states.

The two requirements of the formal definition are exactly what operated here: inside the Maharashtra cluster, villages are mutually similar (common language); between the Maharashtra and Gujarat clusters, points are mutually dissimilar (different languages). The point of the example: clustering was used to find the boundaries of states in post-independence India. No prediction happened — the algorithm only found patterns in the data.

2.9.4 Clustering in a Vector Space

The geometric picture: project the data points into a three-dimensional space. Run a clustering algorithm, and the algorithm groups similar items together. Points that sit close to each other in the vector space get grouped into the same cluster — cluster C1, C2, C3 and so on. Inside the red cluster, for example, the points are very similar to each other, sharing a lot of similarity and very little dissimilarity. Between the red cluster and the pink cluster, the points are very different — more dissimilarity than similarity.

Visual intuition. Draw a 3D scatter plot — three axes for three attributes (say income, age, and spending). Each customer is a dot; nearby dots are similar customers. A clustering algorithm then draws invisible fences around the dense regions: dots bunched near the front-left become cluster C1, the group near the back-right becomes C2, and so on. Landmarks to notice: the dense centers of each group (where the cluster is most "itself"), and the sparser boundary zones between groups (where points are ambiguous). One-sentence takeaway: in the vector space, closeness is similarity, and clusters are regions of closeness.

There is a variety of clustering algorithms. Center-based approaches include k-means. Density-based approaches include DBSCAN. All of these are discussed one by one in module 6, which is on clustering.

2.9.5 Hard and Soft Clustering

A natural question about clusters is membership: can a point belong to more than one cluster? Two answers exist.

Hard clustering — each point belongs to exactly one cluster. k-means is a hard clustering algorithm; it assigns every point to a single cluster.

Soft clustering — a point can be part of two or more clusters, with an affinity to each. The point sits between clusters and carries, say, high affinity to cluster C1 and lower affinity to cluster C2. Affinity and other parameters define how strongly the point belongs to each cluster.

Both styles are used in practice: some applications call for hard clustering, some for soft clustering.

Assumptions & scope. Clustering's similarity picture rests on a distance measure, and the measure carries assumptions: Euclidean distance assumes the coordinates are continuous numbers with comparable scales — if one attribute is income in rupees (thousands) and another is height in meters (about 1.7), the income gaps swamp the height gaps, and the "clusters" are really just income brackets. Fix by rescaling attributes, or by choosing a measure suited to the data type (Hamming for binary). Also, the number of clusters is not delivered by the formula — k-means needs k chosen in advance — and the question "how many clusters?" is an open design decision, previewed in the T-shirt case study later in this session.

2.9.6 Student Questions and Answers

Q: If we have clusters where we see more than one type of group mixed in, what happens? What about a point that sits between clusters?

A: Visualize it. Suppose there is a point in the middle, not very close to cluster C1 and not very close to cluster C2. By the simple picture, a point belongs to only one cluster — that is typically valid in most cases. But there are cases where a point can belong to more than one cluster, with more affinity to one than the other. That is soft clustering.

Q: Can we reject 10% of the points and consider 90% of the cluster, like staying in one state?

A: You are talking about border points — how to handle the points near the boundary. One option is to make a soft boundary. There is a lot of detail in clustering — hard and soft clustering, borders, and more — and we will come back to it in the clustering module.

Pitfalls.

Forgetting that clustering predicts nothing. A clustering result assigns existing points to groups; it does not forecast a label for a new customer. If your exam answer describes clustering as "predicting which group a new customer belongs to," re-read the definition — that is a prediction-style claim, and it is wrong for the descriptive family.
Using Euclidean distance on mixed-type data. Applying the measuring-tape distance to categorical attributes (Refund = Yes/No) produces meaningless numbers. Match the distance to the attribute type.
Ignoring scale. Without rescaling, one attribute with large values dominates the distance, and clusters hide the other attributes' structure.
Confusing hard and soft membership. Hard = one cluster per point (k-means); soft = partial membership with affinities, useful for border points. Do not describe border points as "rejected" — soft clustering keeps them, with weaker membership, instead of throwing them away.

Exam note. Clustering = group points so intra-cluster similarity is high and inter-cluster similarity is low. Euclidean distance $d (x, y) = \sum_{i = 1}^{n} (x_{i} - y_{i})^{2}$ is the measuring tape for continuous attributes; Hamming counts differing coordinates for binary data. Hard clustering (k-means) assigns each point to exactly one cluster; soft clustering allows membership in several with affinities. The Indian-states example is the canonical illustration: similar = same language = same cluster. Module 6 covers how many clusters, boundaries, k-means, and DBSCAN.

2.10 Clustering Applications

2.10.1 Document Clustering: Google News

Real-world: document clustering finds groups of documents that are very similar to each other based on the important terms appearing in them. The simple example is Google News.

Open Google News and look at the headlines: India, world, local, business, technology, sports, science, health, entertainment, and so on. Thousands of news articles arrive every day. How does each article get put into one of these buckets? The buckets are the clusters. The clustering algorithm takes an article about entertainment and puts it in the entertainment cluster, groups articles by similarity into one bucket, and uses dissimilarity to keep them out of other buckets.

Why is this valuable? It organizes the data. If you want entertainment news, you click the entertainment bucket and you do not have to scroll through every article. The similarity inside the business cluster is that all the articles talk about business deals and business aspects; inside the technology cluster, all articles are about technology.

How a document becomes a point. To cluster articles, each article is turned into a vector over its terms — one coordinate per important word, holding how often that word appears. Two articles are similar when their word-count vectors are close:

Term	Article A (tech)	Article B (tech)	Article C (sports)
"chip"	6	5	0
"launch"	4	3	0
"match"	0	1	8

A and B both speak mostly about chips and launches → small distance → same cluster. C speaks about matches → large distance from A and B → sports cluster.

The result is organized news: click "technology" and you see articles about technology, because the algorithm grouped them by shared terms. Note again: clustering is not predicting anything. Given a point, it just finds which cluster the point belongs to. Historical flavor: this was done in the past with around 3000 news articles placed into five or six buckets.

2.10.2 Stock Market Analysis

Real-world: another clustering application is stock analysis. Here we are not predicting any stock price. Instead, we ask whether certain groups of companies behave similarly in the stock market. One cluster contains companies from some sectors that behave very similarly — for example, they are all going down at a particular moment in time. A second cluster contains another group of companies, also moving together. The data points in a cluster behave in a very similar pattern, and based on that grouping you can do a lot of further analysis. Grouping companies or sectors by their behavior gives structure to a noisy market.

Visual intuition — stock movement as points. Think of the axes as time periods and the coordinates as percentage returns: each company is a point whose coordinates are its returns over the last ten trading days. Companies whose ten numbers are close are companies that moved in step. Plot them and you see two or three dense clouds — banks that all fell together, energy stocks that all rose together. Clustering draws the fences around those clouds. One-sentence takeaway: in return-space, nearness = moving in step, and clusters = sectors behaving as a unit.

Pitfall: assuming a cluster is a sector label. The cluster is found from the data — companies moving together — not from the official sector list printed by the exchange. Sometimes the data groups an IT firm with a bank, because their prices happened to move together. The cluster is a data pattern; mapping it to a business explanation is the analyst's next step, not the algorithm's output.

Recap + bridge. Clustering organizes the world: news articles into topic buckets (Google News), companies into behavior groups (stock analysis). In both, the work is the same — convert objects into points, measure distances, group near points — and in both, nothing is predicted. The next section turns one of these applications into a full case study: market segmentation with the T-shirt size problem.

2.11 Market Segmentation: A Clustering Case Study

2.11.1 The T-Shirt Size Problem

Real-world: market segmentation subdivides a market into distinct subsets of customers, where any subset may be selected as a target by the market. The example given in class: you own a T-shirt brand, serving customers from a five-year-old child to a 90-year-old person.

Plot each person on two axes — height on one, weight on the other. Children of five years have their own dimensions; a 50-year-old or a 95-year-old has different dimensions. The plotted data forms a blob in the height–weight plane.

Now, any store you walk into — Nike, Adidas, any brand — carries a limited number of sizes: small, medium, large, extra large, maybe extra small. How are those buckets made? You could make a thousand clusters, each catering to a particular group of people, but then your inventory would be enormous. Instead you make five or six clusters and serve everyone: a person buys the size of the nearest cluster. In real life you see only five or six sizes, and the question is how those clusters are found.

Worked numbers — the height–weight blob. Picture the plot: x-axis = height in cm (140 to 190), y-axis = weight in kg (35 to 90). Every customer is one dot. The dots form one continuous cloud — tall people are usually heavier, so the cloud runs diagonally — but within the cloud there are denser pockets: school children in one region, adults in another, and so on.

Option A — one size: every customer, one T-shirt. Terrible fit for most people, zero inventory cost.

Option B — 1000 clusters: every possible body type gets its own size. Perfect fit, but 1000 SKUs in stock — enormous inventory, unsellable.

Option C — five or six clusters (what brands actually do): the height–weight plane is divided into 5–6 regions; each customer buys the nearest region's size. The store carries six sizes and serves everyone — inventory small, fit acceptable. That is market segmentation: the market is subdivided into a few distinct subsets, and the brand serves them with a few standardized products.

2.11.2 How Many Clusters and Where Are the Boundaries

Two questions drive the whole case study:

How many clusters should there be — five, six, seven, or seventy?
Where are the boundaries — is it one centimeter to two centimeters that makes a small size?

The boundary question has a trap. If you decide by arbitrary ranges — one centimeter to two centimeters goes into small, two centimeters to three centimeters goes into the next size — the split is not very logical. Think about the population: few people are extra small and few are extra large, but a lot of people fall in the middle range. That is why there is more product variety in the middle sizes. There is a science behind running six sizes — Adidas, Nike, Peter England, any brand — and a logical reason the number is six and not sixty or two. The clusters should come from the data, not from an arbitrary cut.

The arbitrary-cut trap. Deciding boundaries by fixed ranges ("1 cm to 2 cm = small") ignores the actual population. The population is not uniform: almost nobody is extra-extra-small and almost nobody is 8XL, but a huge mass of customers sits in the middle sizes. A cut made without looking at the data either wastes sizes on empty regions (too many clusters) or forces most customers into one ill-fitting size (too few). The science is in reading the data — where the population is dense, sizes should be dense — which is exactly what clustering formalizes. That is why brands run six sizes and not sixty: the clusters come from the data, not from an arbitrary cut.

2.11.3 Bringing in Domain Knowledge

Clustering works on the data points, but additional information can also be plugged in from domain knowledge. Height and weight are logical attributes for sizing. What else helps?

One hint given in class: when do people purchase new clothes? When you migrate to college, you buy clothes. When you start a new job, you buy clothes. When you retire, you may buy clothes. That sort of event knowledge can be incorporated into forming the clusters. Sales data can also be taken into account — the boundaries can be inferred from what people actually bought.

Intuition. Pure clustering sees only numbers: height and weight. Domain knowledge adds the human context: people buy clothes when their bodies change, and their bodies change at life events — college, a new job, retirement. A size scheme that ignores when purchases actually happen will mismatch the market; one that folds in event knowledge (and what customers actually bought) lands its boundaries where real demand is. The algorithm proposes, the domain knowledge disposes.

Q: According to the measurement — but how do I decide the measurement?

A: Think about it. How do you decide the measurement? The question is not being answered now — the hint is that the decision should be logical and data-driven, not an arbitrary range. And you can use domain knowledge, like when people buy clothes, to help find the clusters.

The case study closes with the two open questions that the clustering module answers later: how many groups should there be, and how do we identify the group boundaries?

Exam note. The T-shirt problem is the course's canonical "how many clusters?" case: plot customers in the height–weight plane, and choose the number of size clusters (about 5–6, not 60 and not 2) with boundaries set by the data's density, not by arbitrary ranges. Domain knowledge (when people buy clothes, sales history) can guide the boundaries. These two questions — how many clusters? and where are the boundaries? — return in module 6 as the formal problem of choosing k and defining cluster boundaries.

2.12 Association Rule Mining

2.12.1 The Formal Definition

Association rule discovery, or association rule mining, is the second descriptive family. Formal definition: given a set of records, each of which contains some items, produce dependency rules that predict the occurrence of one item based on the occurrence of another item. The job is to find relationships between multiple items. Like clustering, no prediction is made — we are finding patterns and structure in the data.

Definition. An association rule is a statement of the form "if X, then Y" — formally written $X \to Y$ — where X and Y are sets of items that co-occur in the records. The word "predict" here means anticipate in the data, not forecast into the future: the rule says "whenever X appears in a record, Y tends to appear too." No prediction of new records is made; the deliverable is a description of how items hang together.

2.12.2 The Market Basket Example

Real-world: walk into any mart — Big Bazaar, Reliance Mart, or any other supermarket. Here is what the customer purchase history looks like. Customer 1 purchased bread, Coke, and milk. Customer 2 purchased cereal and bread. Customer 3 purchased some items, and so on.

Based on this transaction log, can we mine something? We want to know whether the sale of one product leads to the sale of another. Look at the log and ask:

If milk is bought, is there a 100% chance that Coke will also be bought by the same customer?
If diaper and milk are bought, is there an 80% chance that the same customer will buy beer as well?

Rules of this shape — if product X is bought, there is an 80% chance product Y will also be bought; if products A, B, and C are bought, there is an 80% chance product D will also be bought — are exactly what association rule mining finds. The rules let you do a lot of things later: store layout, promotions, bundling, inventory.

Worked example — a ten-transaction log. Here is a small market basket log (each row is one customer's basket):

Transaction	Items
T1	Milk, Coke, Diaper, Beer
T2	Milk, Coke, Diaper, Beer
T3	Milk, Coke, Diaper, Beer
T4	Milk, Coke, Diaper, Beer
T5	Milk, Diaper
T6	Coke
T7	Coke, Bread
T8	Coke, Bread
T9	Bread, Cereal
T10	Bread

Now count, for each proposed rule, the fraction of transactions where the right side appears among those where the left side appears:

Rule 1: milk → Coke. Milk appears in T1–T5, i.e., in 5 transactions. Coke appears in 4 of those (T1–T4). Strength $= \frac{4}{5} = 0.80$ , or 80%.

Rule 2: milk ∧ diaper → beer. Milk together with diaper appears in T1–T5, i.e., in 5 transactions. Beer appears in 4 of those (T1–T4). Strength $= \frac{4}{5} = 0.80$ , or 80%.

Rule 3: Coke → milk. Coke appears in T1–T4 and T6–T8, i.e., in 7 transactions. Milk appears in 4 of those (T1–T4). Strength $= \frac{4}{7} \approx 0.57$ , or about 57%.

The rules come straight out of the log: milk → Coke at 80%, milk ∧ diaper → beer at 80%, Coke → milk at only 57%. Sense-check: the same four milk-and-Coke transactions feed both Rule 1 and Rule 3, but their denominators differ (5 milk baskets vs 7 Coke baskets), which is exactly why the two directions give different numbers. The professor's illustrative figures from class — 80% for milk → Coke, about 60% for Coke → milk, 80% for diaper-and-milk → beer — are reproduced here from one concrete table (with 80% and 60% matching; the "100%" figure was posed in class as a question to check against a log, and this particular log answers it with 80%).

2.12.3 Rules Are Directional, Not Symmetric

A common trap: is "milk leads to the sale of Coke" the same as "Coke leads to the sale of milk"? No. The rule has a direction.

If you purchase milk, there is an 80% chance you will purchase Coke. But if you purchase Coke, there might only be a 60% chance you will purchase milk. The relationship exists in both directions, but with different strengths — the inverse relationship is not equal. In rule terminology, the left-hand side is the antecedent and the right-hand side is the consequent, and swapping them changes the rule entirely.

Formally, the strength of a rule "if X then Y" is a conditional probability:

$P (Y ∣ X) = \frac{P ( X \land Y )}{P ( X )}$

Here $P (Y ∣ X)$ reads "the probability of Y given X," $P (X \land Y)$ is the probability that both X and Y occur, and $P (X)$ is the probability that X occurs. So $P (Coke ∣ milk) = 0.80$ but $P (milk ∣ Coke) = 0.60$ — the two rules are different rules with different strengths.

Building the formula. The strength of "if X then Y" is: among all records containing X, what fraction also contain Y?

$P (Y ∣ X) = \frac{P ( X \land Y )}{P ( X )}$

$P (X \land Y)$ — the fraction of all records containing both X and Y (the numerator — "both happened").
$P (X)$ — the fraction of records containing X (the denominator — "the condition happened").
$P (Y ∣ X)$ — their ratio: the chance of Y given X.

In the worked log: $P (milk \land Coke) = \frac{4}{10} = 0.40$ , $P (milk) = \frac{5}{10} = 0.50$ , so $P (Coke ∣ milk) = \frac{0.40}{0.50} = 0.80$ . The rule $X \to Y$ is the antecedent X on the left and the consequent Y on the right; the standard reference in the textbook literature confirms this strength as the rule's confidence — the conditional probability of the consequent given the antecedent — with the direction built into the conditional.

Pitfall: assuming rules are symmetric. "Milk → Coke" and "Coke → milk" are different rules. The worked log shows why: the numerator is the same (4 shared transactions), but the denominators differ (5 milk baskets vs 7 Coke baskets), so the strengths differ — 80% vs 57%. A rule is an implication with a direction; swapping antecedent and consequent gives a new, weaker or stronger rule, not a restatement of the old one. (Note also: a strong association is a co-occurrence pattern, not proof that one purchase causes the other.)

2.12.4 The Beer and Baby Products Story

Real-world: a student posted the line "baby needs beer and wines," which the instructor called a very good and very famous example. There is a science behind it.

Why do supermarkets put beer and wine near baby products? The story: a father goes to buy baby needs — that purchase is fine on its own — but why connect beer and wine with baby needs? In the US this has been thoroughly studied by researchers, and they gave a logical reason why baby products and beer are put together in a supermarket or mall, and why they are always placed near each other.

This was given as homework: search for the famous study. The reason is famous, well studied, and somewhat funny. Search it on the internet; if you cannot find it, bring it up in the next class. The point for now: association rule mining is all around us. Companies maximize sales by exploiting these rules.

Working the story as a rule. The beer-and-baby-products case is a rule of exactly the shape mined from the market log:

Rule: diaper (baby product) → beer, with a high strength. The professor's own numbers say diaper-and-milk → beer at 80%. Whatever the famous study's exact figures, the pattern is the same as the worked log: the receipt data shows diapers and beer co-occurring often enough that the rule survives a check on the data — and the supermarket then acts on it, placing the items near each other so the association sells more of both.

The punchline of the story: the rule is directional and data-driven — some purchases (baby needs) statistically travel with others (beer), so stores rearrange the shop to profit from the pattern.

2.12.5 Advertising Psychology: Mining Consumers' Minds

Real-world: many famous advertisements deliberately exploit association-rule psychology. The instructor walked through three of them.

Coca-Cola — "Thanda matlab Coca-Cola." The ad tries to tell you: whenever you feel thirsty and want a cold drink, associate that feeling with Coca-Cola. Psychologically, the message is "please purchase Coca-Cola." The tagline links the feeling of being "thanda" (cold) with the brand.

Coca-Cola — "Things go better with Coke." This famous ad appeared in malls, canteens, and other places. The message: whatever you purchase, please purchase Coke with it, because the basic item will taste better. The company maximizes sales by using association in a psychological environment.

Mountain Dew — the "darr" tagline. People have that feeling multiple times a day, and the ad connects that feeling with the product. Whenever you feel that sort of thrill, the tagline sits in your brain and the recall of the brand Mountain Dew comes to your head.

The structure behind all of this: companies connect an antecedent (a feeling, a purchase) with a consequent (the product) so that they maximize the sales of their product by indirectly marketing to you. Association rule mining is not just a data technique — it is marketing psychology.

Intuition. An ad is an association rule aimed at the brain. The data-mining rule is "basket X → item Y at 80%"; the advertising rule is "feeling X → product Y at high recall." "Thanda" is the antecedent, Coca-Cola the consequent; "things" (any purchase) the antecedent, Coke the consequent; "darr" the antecedent, Mountain Dew the consequent. The ad's job is to make that association stick, so that when the feeling fires, the product is recalled automatically. Same rule shape — the shop mines receipts, the advertiser mines minds.

2.12.6 Student Questions and Answers

Q: Is "milk leads to the sale of Coke" the same as "Coke leads to the sale of milk"?

A: No, they are not saying the same thing. If you purchase milk, there is an 80% chance you will purchase Coke. But the inverse relationship might not hold with the same strength — if you purchase Coke, there might be a 60% chance you will buy milk. The relationship exists, but the inverse is not equal. The left side is the antecedent, the right side is the consequent, and the direction matters.

Q: Why are beer and baby products put together in a mall?

A: This is a very famous, thoroughly studied example in the US. Researchers gave a logical reason why baby products and beer are always placed near each other. It is homework to search for the study — the reason is interesting and a bit funny. If you cannot find it, bring it up in the next class.

Exam note. Association rule mining finds rules "if X then Y" — e.g., milk → Coke at 80%, diaper ∧ milk → beer at 80%. Strength = conditional probability $P (Y ∣ X) = \frac{P ( X \land Y )}{P ( X )}$ (the standard reference calls this the rule's confidence). Rules are directional: antecedent (left) and consequent (right) cannot be swapped. Be able to compute a rule's strength from a small transaction log — that is the most examinable skill in this section. Module 5 formalizes support, confidence, and the algorithms.

2.13 Association Rule Mining Applications

2.13.1 Inventory Management

Real-world: inventory management. The goal of an appliance repair company: anticipate the nature of repairs on its customers' products and keep the service vehicle equipped with the right parts, to reduce the number of visits to the customer's household. Think of the big appliance brands — LG, Bosch, and similar names — that run service fleets.

Two location stories make the pattern concrete:

Goa — humidity. In Goa there is humidity all year long because the state is near the sea. Because of that humidity, certain parts in refrigerators and washing machines go bad. There is a perfect sense to keep the inventory of those parts at the Goa location.

Indore — water quality. In Indore a lot of water is drawn from underground, and the water quality is bad — it has a lot of calcium and magnesium. When that happens, certain parts in the washing machine go bad because of the poor water quality.

In both cases, you want to understand the pattern and keep that inventory at the specific location. Find the association — this part fails when this environmental condition is present — and keep the part local so the repair can happen ASAP. That is association-driven inventory management.

Worked example — the rule behind the parts. An appliance company studies its repair logs across the country. Two rules surface:

Rule found in repair data	Consequence for inventory
"High humidity → compressor corrosion in Goa"	Stock compressors at the Goa depot; repair in one visit
"Hard water (high calcium/magnesium) → washing machine valve failure in Indore"	Stock valves at the Indore depot; repair in one visit

Before the rule: the Goa technician runs out of compressors on the job and must return for a second visit. After the rule: the part is already local, the repair finishes in one visit, and both the customer and the company save a trip. The general pattern: if X is happening, there is a high chance Y will also happen; if Y is happening, a high chance Z will also happen. Find that sort of association and you can build a lot of things around it.

Amazon and Flipkart warehouses. Both companies have huge warehouses at different locations in the country. The warehouse locations are chosen so that they can ship a particular item to the majority of customers in the nearby region at minimum cost and minimum time. That is inventory management being maximized. When Amazon promises one-day delivery, they can make that promise because they identified the areas — the map of India, then states, then districts — and placed warehouses so that customers around each warehouse can be served in one day. They know they can do it in one day, and they advertise it.

2.13.2 Supermarket Shelf Management

Real-world: supermarket shelf management. If you go to any mall, the items are not arranged in a random fashion. They are arranged in a particular fashion so that they can maximize their sales. Which products sit next to which, what is at eye level, what is near the checkout — all of it follows from the associations found in purchase data.

Intuition — shelf layout as executed rules. Every shelf position is a decision driven by an association rule from receipt data: the diaper-and-beer rule puts two frequently co-purchased items within reach of the same shopper; the eye-level rule places high-margin items where the gaze lands first; the checkout position exploits the final impulse purchase. The layout is the rule set made physical — and every mall you walk into is silently executing mined rules.

2.13.3 Airport Layout as Space Management

Real-world: the airport is a classic example of space management. Airports are typically designed so that a passenger entering the airport must traverse the maximum number of shops before boarding the plane. From the time they enter the airport to the time they board, the passenger travels past a lot of shops.

Why? If a customer passes through shops, they see the products and are more likely to purchase. Shop sales go up, and the airport can then charge higher rent — airport shops rent at huge cost. The airport enforces the layout that maximizes customer flow through the shops, and that design is standard in modern airports.

Pitfall: confusing correlation with causation in layout decisions. The associations behind shelves, depots, and airport routes are co-occurrence patterns — "these two sell together," "this part fails with this climate." Acting on them is smart, but the rules never prove that one event caused the other. The airport layout does not prove that more shop exposure caused higher sales — it is a reasoned bet on that relationship. In your own analysis, treat mined rules as strong hints to test, not as proven laws.

Recap + bridge. Three application arenas for one rule shape: inventory management (stock the part where the failure condition is — Goa humidity, Indore water), supermarket shelf management (layout follows purchase associations), and airport design (route passengers past maximum shops). Each is "find the association, then build the business decision on it." The final descriptive-family topic is anomaly detection — in a sense, the reverse of association: not what travels together, but what does not fit the pattern.

2.14 Anomaly Detection

2.14.1 Profiling Normal Behavior

Anomaly detection, also called deviation detection, is very common these days. The idea: you build a profile of a customer, and if there is a small anomaly or change in behavior, the system calls it out. It is a prediction strategy — whenever there is a change in the user's behavior, the system alerts you.

The difference from classification: classification learns a boundary between classes. Anomaly detection instead learns what normal looks like for an individual, and flags behavior that deviates from that profile.

Definition. An anomaly (also called an outlier or deviation) is a behavior or data point that deviates significantly from the rest of the data — "as if it were generated by a different mechanism," as the standard reference puts it. Anomaly detection is the strategy of first profiling what is normal for a customer or system, then alerting whenever new behavior strays from that profile.

Worked example — the two fraud scenarios, formally. The credit card stories from section 2.7 are anomaly detection in action:

What is normal (the profile)	The deviation	The alert
Five years of ₹2,000–₹8,000 transactions in Goa	A ₹10,00,000 swipe	Block + review
Logins from the office network	A login from the US at 3 am	OTP / second verification

In both rows, the system did not know "this is fraud" — it knew "this is not the profile," and that alone triggered the check. The alert is a call for attention, not a verdict; a human or a further model decides.

Intuition — the guard who knows your habits. A new guard at your building knows nothing about you, so nothing looks wrong. A guard who has watched you for a year knows you leave at 6 pm — so 10 pm looks odd, and a stranger leaving with your bag looks wrong. Anomaly detection is the year-long guard: it builds the "normal" picture first, then reacts to what does not match. That is why the professor describes it as a prediction strategy — it anticipates that a deviation now means trouble ahead.

2.14.2 Anomaly Detection Use Cases

Real-world: the standard use cases are credit card fraud detection and network intrusion detection. Both work on the same principle — a profile of normal behavior, and an alert when behavior deviates. The course covers only one class on anomaly detection, so treat it as a focused topic: the profiling idea, the two headline applications, and the connection back to prediction.

Assumptions & scope. Anomaly detection assumes that "normal" is learnable and stable: the profile is built from history, so it is only as good as that history. Two failure modes follow. First, normal changes — spending habits drift with life (a new job, a move), and a static profile starts firing false alarms. Second, the rare attack — if an intrusion pattern has never been seen before, there is nothing in the profile to match against, and the deviation may look normal. The method flags deviations; it does not identify what kind of attack or fraud they are. (For the record: the standard reference defines an outlier as an object that deviates significantly from the rest, and lists exactly these headline uses — credit card fraud and intrusion detection.)

Dimension	Classification	Anomaly detection
What is learned	A boundary between classes (cheat vs no-cheat)	A profile of normal behavior for a customer/system
What triggers a decision	Every new case is assigned a class	A new case is flagged only when it deviates from the profile
Classes	Both classes are known and labeled in training	"Normal" is known; "anomalous" is whatever is not normal
Typical use	Direct marketing, churn, medical diagnosis	Credit card fraud, network intrusion

When to pick which: if you have labeled examples of both outcomes, classify; if the interesting cases are rare, unlabeled, and only recognizable as "different from normal," detect anomalies.

Exam note. Anomaly detection (deviation detection) = build a profile of normal behavior, then alert when behavior deviates. Headline applications: credit card fraud detection and network intrusion detection. Contrast with classification: classification learns a boundary between known classes; anomaly detection learns what normal is and flags everything else. The professor flags this as a one-class topic — the profiling idea, the two applications, and the link back to prediction are the whole of the exam surface here.

2.15 Data Preprocessing: From Physical Entities to Datasets

2.15.1 Why Preprocessing

After the tour of mining techniques — classification, clustering, association rule mining, anomaly detection, and real-world deployments of each — the class turned to the data itself. Data preprocessing is what you do once you get the data: how do you arrange and structure the data so that you can apply data mining algorithms on them?

You cannot apply a data mining algorithm to rough data. The data needs to be prepared in a proper fashion first. The process of converting raw data into a form where a mining algorithm can be applied is called data preprocessing, and it has many steps, seen one by one.

Definition. Data preprocessing is the family of operations that turns raw, real-world data into a clean, structured form that mining algorithms can consume. The reference literature groups the operations into four families: data cleaning (fix missing values, noise, inconsistencies), data integration (merge multiple sources into one coherent store), data reduction (shrink the data while keeping the analysis results almost unchanged), and data transformation (rescale, normalize, discretize). The lecture's pipeline stage "preprocessing" covers exactly this ground, in more steps than one slide can show.

2.15.2 Every Dataset Comes from a Physical Entity

What is a dataset? A dataset is a table, or a group of data, on which you want to apply a data mining algorithm. The crucial observation: datasets are not generated randomly. Typically a dataset is generated from some physical entity — a physical activity, a physical environment.

All datasets are actually derived from, or generated from, a physical entity or environment. When you do this mapping from the physical world into a table, the mapping can have its own issues. That is where preprocessing begins.

Intuition — the map is not the territory. A dataset is a model of a physical entity, like a map is a model of a city. The map is useful precisely because it is smaller and simpler than the city — but the simplification can lie: streets get renamed, lanes get added, and a map drawn last year can send you the wrong way today. The same happens when a physical process becomes a table: the table is a filtered, possibly distorted picture, and the distortions are exactly what preprocessing must handle.

2.15.3 Worked Example: The Bank Passbook Table

Take a bank passbook. It is a physical book in your hand, and from it you can generate a dataset. The passbook has columns for account or branch ID, transaction ID, debit, credit, and balance.

You make a digital copy of the physical book, and the resulting table — with those columns and a row per transaction — is the dataset. Note carefully: this table is not ready for data mining yet. Preprocessing has to be done to get it into the right shape before a mining algorithm can run. But the table itself is already called a dataset.

Worked example — the passbook becomes rows. A physical passbook page shows handwritten entries; the dataset is its digital copy:

Account ID	Transaction ID	Debit	Credit	Balance
A-1041	T0001	—	10,000	10,000
A-1041	T0002	2,500	—	7,500
A-1041	T0003	—	15,000	22,500

The table is the dataset: one row per transaction, columns for IDs, debits, credits, balance. But note what preprocessing still faces: missing values (the dash cells), possibly inconsistent date formats, branch IDs written differently on different pages. The table is already a dataset — it is not yet a clean one.

2.15.4 Worked Example: The Medical Database

The same pattern repeats in medicine. A medical database is built from patients and their treatment. You create a log book with patient 1 and the treatment the patient is getting, the medicine being given, the doctor, the room number, and so on. The physical entity — the hospital floor, the patients, the treatments — is converted into a more tabular form, and that table is the dataset.

Datasets need not be tabular by nature. The point of arranging the data is to put it into a digital, structured form. Structure is discussed in much more detail in the next few slides.

Worked example — the hospital floor becomes rows. The physical entity is the ward: patients, treatments, medicines, doctors, rooms. The digital log book arranges it:

Patient ID	Treatment	Medicine	Doctor	Room
P1	Physiotherapy	—	Dr. Rao	204
P2	Surgery	Antibiotic X	Dr. Rao	205
P3	Physiotherapy	Painkiller Y	Dr. Iyer	204

Each patient becomes a row, each recorded property a column — the table is the dataset. Notice the "—" for P1's medicine: the physical world has no such empty cell (everyone either took medicine or did not), and the mapping into the table introduced ambiguity. That is a miniature example of the mapping issues the professor warned about.

2.15.5 The Hardest Step: Modeling the Real World

Suppose you are building a tsunami detection system or a weather prediction system. For weather, you declare three classes — hot, moderate, cold — and you decide to capture temperature, wind speed, and maybe a few more entities to model the environment, then run a classification on those attributes.

Now the key question: are these attributes enough to represent the real world? If they are not enough, the modeling is going to go bad, because the dataset is not proper. You have a physical entity represented in a dataset and a data mining model built on it — but if the dataset does not capture the complete physical entity, the mining algorithm will also go bad.

The same failure appears in crude oil prediction. You choose attributes that you believe affect the price of crude and build a model. If the model's performance is not good, one of the reasons may be that you could not represent the entities that actually predict the crude price.

The modeling trap. The dataset is a simplification of reality, and the model inherits the simplification. Three concrete failure paths:

Missing attributes. Weather affects prices, rainfall, and logistics — model weather with only temperature and wind speed, and the missing humidity is still doing work in reality, just not in your dataset.
Wrong attributes. Capturing what is measurable (sensor readings) instead of what is causal (pressure gradients) trains the model on proxies.
Coarse classes. Declaring hot/moderate/cold throws away the exact temperature, and with it the information that separates a mild day from a dangerous one.

The sobering conclusion: modeling a physical entity into a digital form is an extremely hard task. Building a classification or regression model from a given dataset is very easy — a few dozen lines of code. But choosing what to capture so the dataset represents reality is the genuinely difficult part. You do not know all the things that can affect the weather, or all the attributes that help model tomorrow's crude oil price, or what to capture for a tsunami prediction system. When your engine underperforms, the problem may not be the classification algorithm at all — it may be the way you collected the data or the way you modeled the data itself.

2.15.6 Real-World Warning: No One Hands You a Dataset

Here is a reality check for the working world. Nobody is going to hand you a dataset and ask you to build a classification model on it. That task is child's play now — about 20 lines of code. What people actually ask: can you build a tsunami detection system for me? Can you predict tomorrow's crude oil price?

That is the problem statement. You have to go back to the domain and visualize what all actually affects the price of crude, or the weather, or a tsunami. Then you capture data for those entities and put them into a dataset so that the mining can happen.

The takeaway to remember: dataset creation is not an easy task, and it is very subjective — it depends on the question you are going to ask from the dataset. Two people asked different questions will build different datasets from the same physical world. This theme returns in the next few sessions.

Exam note. Two worked examples anchor this section: the bank passbook (Account ID, Transaction ID, Debit, Credit, Balance → one row per transaction) and the medical log (Patient, Treatment, Medicine, Doctor, Room). Both show the core message: every dataset is a mapping of a physical entity, the mapping can lose or distort information, and that is why preprocessing exists. The hardest step is choosing what to capture — a few dozen lines of code build the model; the dataset itself decides whether the model can ever be right. Dataset creation is subjective: it depends on the question asked.

2.16 Objects and Attributes

2.16.1 Data as a Collection of Objects and Attributes

One standard way to arrange data is the tabular form. Data, in this view, is a collection of objects and attributes.

An attribute is a property or characteristic of an object. Think of yourself as an object: your name is an attribute, your height is an attribute (say 5.8 feet), your weight is an attribute (say 60 kg). Attributes help us define objects. If you make this kind of table, it is essentially an n-dimensional representation: each row is an object, and the columns are the attributes that define it.

Definition. An object is one thing the data is about — a customer, a transaction, a patient. An attribute is a property or characteristic of an object — height, marital status, income. Data arranged as a table is a collection of objects (rows) each described by attributes (columns).

You as a row. If you are the object, your row has one cell per attribute: Name = "you", Height = 5.8 feet, Weight = 60 kg. Any attribute you could fill in is a column; the set of all attributes is what defines you as a data point.

2.16.2 The Vocabulary of Data Mining

The same ideas appear in many textbooks under different names. Objects are also called records, points, samples, entities, or instances — essentially a unique instance. In a table, rows represent objects, and columns represent attributes. Each object is represented by its attribute values. If a row has n attributes, the object behaves like an n-dimensional vector, with one coordinate per attribute.

Knowing all five synonyms — record, point, sample, entity, instance — matters, because papers and tools use different words for the same thing.

Vocabulary — five names, one idea. The same thing is called:

Term	Where you meet it
record	Database and table language (a row of a table)
point	Geometry — a dot in an n-dimensional space
sample	Statistics — one draw from a population
entity	Database modeling — a thing in the real world
instance	Machine learning — one case fed to an algorithm

The reference texts confirm the mapping: "the rows of a database correspond to the data objects, and the columns correspond to the attributes," and a set of attributes describing one object is called an attribute vector (or feature vector). So a row with n attributes genuinely behaves like an n-dimensional point, with one coordinate per attribute — the professor's line, which the reference texts state in exactly this form.

The n-dimensional row. Take the cheat-detection table from section 2.6: each customer is a row with attributes ID, Refund, Marital Status, Taxable Income, and Cheat. If we keep the four descriptive columns (dropping the ID), each customer is a point in a 4-dimensional space:

Customer 8: Refund = No, Marital Status = Single, Income = 85k, Cheat = Yes → the point $(N o, S in g l e, 85 k, Y es)$ .

The row is a vector: one coordinate per attribute. Different attributes are different axes; the Euclidean distance from section 2.9 measures how far two customers are along those axes. That is why "rows are objects, columns are attributes" and "rows are points in space" are the same sentence in different words.

2.16.3 Attribute Values

An attribute is a property, but a property takes values. Height is an attribute; the height attribute can range from, say, one centimeter to 180 centimeters. Your height might be 170 centimeters and another person's height might be 150 centimeters. The attribute value varies from object to object.

Same idea for the financial attribute: taxable income is an attribute, and its value can range from maybe 0 to 125k. So: you have attributes that define objects, and each attribute value ranges from some X to some Y.

Attribute values in the cheat table. Each attribute has a value set, and the value varies from row to row:

Attribute	Value range in the dataset	Kind of variation
Refund	Yes, No	Picks one of two values
Marital Status	Single, Married, Divorced	Picks one of three values
Taxable Income	60k – 220k	Any number in the range
Cheat	Yes, No	The label varies by row

The pattern to carry forward: an attribute is the name of a property; the attribute value is the specific entry in a cell, and the entry can vary across objects — 170 cm for one person, 150 cm for another, 125k for one customer, 85k for another.

2.16.4 Types of Attributes: A Preview

There are different types of attributes, discussed in full in the next class: nominal, ordinal, interval, and ratio; categorical and continuous, and so on. For now, keep the two broad classes from the cheat-detection dataset — categorical (fixed number of values) and continuous (infinite values in a range) — and remember that the number of unique values an attribute can take helps you decide which type it is.

Preview, not yet the full story. The cheat table gives two broad types — categorical (Refund: 2 values) and continuous (Income: infinite values) — and the counting trick to separate them. The full taxonomy comes next class: nominal (names, no order), ordinal (ordered categories, e.g., small < medium < large), interval (equal differences, no true zero, e.g., temperature in Celsius), and ratio (equal ratios, true zero, e.g., weight). Do not over-extend today's rule: the unique-value count is the first clue, but it is not the whole definition of the four-part taxonomy.

2.16.5 Other Ways to Represent Data

The table is one way to represent a physical entity, and it is the one used for most of the course. There are other ways to represent data — for example, using graphs. Once the tabular form is covered, the next class moves on to those other representations.

Exam note. Data as a table = collection of objects (rows) with attributes (columns). Synonyms for object: record, point, sample, entity, instance — all mean one unique row. A row of n attributes behaves like an n-dimensional vector, one coordinate per attribute (the reference texts confirm the term attribute vector). Attributes take values that vary from object to object. Categorical (fixed count of values) vs continuous (infinite values in a range) is the two-bucket preview; the full nominal/ordinal/interval/ratio taxonomy arrives next class.

Exam Guidance Summary

There is no mark distribution or exam pattern announced in this session, but the course structure itself is exam-relevant planning information:

Classification is module 4 — the deep dive into classification algorithms happens there. The introductory motivation in this session (class labels, binary versus n-class, train/test, applications) is the foundation for it.
Association rule mining is module 5 — the formal rules, measures, and algorithms come later. The skill practised here — computing a rule's strength from a transaction log — becomes the formal support and confidence framework there.
Clustering is module 6 — including the open questions raised here: how many clusters (the T-shirt size question), how to find cluster boundaries, center-based versus density-based methods, hard versus soft clustering.
Anomaly detection gets one class only — a focused topic: profiling normal behavior, deviation alerts, credit card fraud and network intrusion.
Study advice: the beer-and-baby-products story is assigned as homework — search the famous US study and understand why the rule is what it is. The reasoning reappears when association measures are formalized.
Conceptual checklist for this session: be able to distinguish prediction from description; classification (class label, finite classes) from regression (continuous value); categorical from continuous attributes; hard from soft clustering; and to walk through the full data mining pipeline with its feedback loop.

Key Industry Applications

Consolidated list of every real-world connection made in this session:

Direct marketing — iPhone-style launches: classification selects the customers most likely to buy, so ads are sent only to them.
Fraud detection — credit and debit card transactions checked automatically at the POS; amount anomalies (10 lakh swipe after years of thousands) and location anomalies (Goa profile, US transaction) flagged as suspicious.
Login security — Gmail-style offsite logins trigger extra authentication (OTP, second email) because the behavior deviates from the user's profile.
Customer churn — Colgate versus Patanjali toothpaste, Ola, and Amazon-Flipkart coupon wars: models predict which customers will leave, and coupons retain the ones about to switch.
Document clustering — Google News buckets articles into India, world, local, business, technology, sports, science, health, entertainment.
Stock market analysis — clustering companies by similar behavior (e.g., moving down together) to find sector structure.
Market segmentation — T-shirt brands run five or six sizes because the population density in the middle drives cluster boundaries; domain knowledge (when people buy clothes) can improve clusters.
Inventory management — appliance repair companies stock service vehicles by location (Goa humidity kills fridge parts; Indore hard water kills washing machine parts); Amazon and Flipkart place warehouses to guarantee one-day delivery.
Supermarket shelf management — shelf layouts follow purchase associations, not randomness.
Airport design — passengers are routed past maximum shops before boarding, raising shop sales and airport rent.
Advertising psychology — Coca-Cola ("Thanda matlab Coca-Cola", "Things go better with Coke") and Mountain Dew ("darr") taglines tie a feeling to a brand: antecedent to consequent.
Anomaly detection — credit card fraud detection and network intrusion detection through behavior profiles.

DM Lecture 2 notes · Data Mining Fundamentals

Data Mining· postgraduate· 2026-08-05

Sections Breakdown

1Why Data Mining

Three trends - automatic collection, cheap storage, and online migration - created the modern flood of data

2What Is Data Mining

The four-adjective definition and the field's roots in five parent disciplines

3The Data Mining Process Pipeline

The six-stage pipeline from raw data to knowledge, and the feedback loop when knowledge is not useful

4Predictive and Descriptive Functions

Two families: regression and classification predict; clustering and association rules describe

5Classification

Class labels from a finite set: binary, three-class, and n-class problems

6The Classification Dataset: Attributes, Train and Test

The cheat-detection table, categorical versus continuous attributes, and the train/test workflow

7Classification Applications

Direct marketing, fraud detection, and customer churn prediction as binary classification

8Regression

Predicting continuous values such as prices, temperature, and rainfall

9Clustering

Grouping similar points with Euclidean and Hamming distance; hard and soft clustering

10Clustering Applications

Google News document clustering and stock market behavior analysis

11Market Segmentation: A Clustering Case Study

The T-shirt size problem: how many clusters and where the boundaries lie

12Association Rule Mining

Rules of the form X to Y, conditional-probability strength, and directionality

13Association Rule Mining Applications

Inventory management, supermarket shelves, and airport layout driven by associations

14Anomaly Detection

Profiling normal behavior and alerting on deviations

15Data Preprocessing: From Physical Entities to Datasets

Why preprocessing exists, and how datasets are mapped from physical entities

16Objects and Attributes

Rows as objects, columns as attributes, the synonym vocabulary, and attribute values

17Exam Guidance Summary

Course structure and study advice for classification, association rules, and clustering

18Key Industry Applications

Every real-world connection made in the session, method by method

Postgraduate students in Data Mining

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Why Data Mining

Must-know: Data mining exists because automatic collection, cheap storage, and online migration produce far more data than humans can interpret; the motivating question is whether relevant questions get proper answers.

⚠️ Top pitfall: Assuming having data is the same as having answers — data must be mined to yield answers.

Self-check: Name the three trends behind the growth of data mining.

Connects to: What Is Data Mining

What Is Data Mining

Must-know: The four-adjective definition — non-trivial, implicit, previously unknown, potentially useful — and the five parent fields.

⚠️ Top pitfall: Treating data mining as one algorithm instead of a family of techniques from five parent fields.

Self-check: Why is 'customers buy bread' not data mining?

Connects to: Why Data Mining; The Data Mining Process Pipeline

The Data Mining Process Pipeline

Must-know: The six pipeline stages in order and the feedback loop: if knowledge is not useful, go back and change mining, selection, cleaning, or integration and retry.

⚠️ Top pitfall: Viewing the pipeline as a one-way conveyor belt instead of a feedback loop.

Self-check: Name the six stages of the standard pipeline and the output.

Connects to: What Is Data Mining; Predictive and Descriptive Functions

Predictive and Descriptive Functions

Must-know: The mapping: classification and regression predict; clustering and association rule learning describe. Test: does the method forecast an unknown value/label, or describe structure?

⚠️ Top pitfall: Calling clustering predictive because it groups 'future customers' — descriptive methods never output a value or label for a new case.

Self-check: Which family does association rule learning belong to, and why?

Connects to: Classification; Regression; Clustering; Association Rule Mining

Classification

Must-know: Classification predicts one of a finite set of class labels; binary = 2 classes, three-class = 3, n-class = n; the input may be continuous, only the output is a class.

⚠️ Top pitfall: Confusing output type with input type: continuous inputs do not make the problem regression.

Self-check: A handwritten-digit recognizer predicts one of 10 digits — how many classes, and is it classification?

Connects to: The Classification Dataset: Attributes, Train and Test; Regression; Predictive and Descriptive Functions

The Classification Dataset: Attributes, Train and Test

Must-know: Cheat table columns; Refund and Marital Status are categorical (2 and 3 unique values), Taxable Income is continuous (infinite values); the model learns from train data with labels and is judged on test data without them.

⚠️ Top pitfall: Feeding the ID column to the model — unique identifiers carry no predictive signal.

Self-check: Customer 11: No refund, Single, 75k — what does the trained model output, and what role did customers 1-10 play?

Connects to: Classification; Objects and Attributes

Classification Applications

Must-know: Direct marketing selects likely buyers to save ad spend; fraud detection is binary classification judged against a behavior profile (10 lakh swipe, Goa-to-US login); churn models drive the retention coupon loop (stay or leave).

⚠️ Top pitfall: Treating the retention coupon as random marketing instead of the output of a churn classifier.

Self-check: Why does a 10 lakh swipe after years of thousands look fraudulent?

Connects to: Classification; Anomaly Detection

Regression

Must-know: Regression predicts continuous values (petrol price, oil price, temperature, rainfall); classification predicts class labels. The output decides the family.

⚠️ Top pitfall: Judging the family by the input instead of the output.

Self-check: Predicting 'how much rain in centimeters' — regression or classification?

Connects to: Classification; Predictive and Descriptive Functions

Clustering

Must-know: Clustering definition (similar inside, dissimilar across); Euclidean distance d(x,y) = sqrt(sum (x_i - y_i)^2) for continuous attributes, Hamming counts differing coordinates for binary; hard (k-means) vs soft (affinity) membership; no prediction happens.

$d (x, y) = i = 1 \sum n (x_{i} - y_{i})^{2}$

⚠️ Top pitfall: Claiming clustering predicts a label for a new customer; using Euclidean distance on mixed-type data; ignoring attribute scale.

Self-check: Two villages share Marathi, a third speaks Gujarati — which cluster assignments follow, and why?

Connects to: Clustering Applications; Market Segmentation: A Clustering Case Study; Objects and Attributes

Clustering Applications

Must-know: Google News: articles clustered by important terms into buckets (business, technology...); stock analysis: companies clustered by similar behavior (e.g., falling together); clustering organizes, never predicts.

⚠️ Top pitfall: Assuming a cluster is an official sector label — clusters come from the data, not from printed lists.

Self-check: How does a news article become a point that can be clustered?

Connects to: Clustering; Market Segmentation: A Clustering Case Study

Market Segmentation: A Clustering Case Study

Must-know: The two open questions of clustering: how many clusters (five or six sizes, not sixty or two) and where the boundaries lie (data-driven, not arbitrary ranges); domain knowledge like college-migration purchases and sales data can be folded in.

⚠️ Top pitfall: The arbitrary-cut trap: fixed ranges like 1-2 cm = small ignore the population distribution.

Self-check: Why do brands carry six sizes rather than one thousand or two?

Connects to: Clustering; Clustering Applications

Association Rule Mining

Must-know: Rule strength is a conditional probability P(Y|X) = P(X and Y)/P(X), the standard reference calls it confidence; a 10-transaction log computes milk -> Coke at 80%, milk and diaper -> beer at 80%, Coke -> milk at about 57%; rules are directional, swapping antecedent and consequent changes the rule.

$P (Y ∣ X) = \frac{P ( X \land Y )}{P ( X )}$

⚠️ Top pitfall: Assuming rules are symmetric — milk to Coke (80%) differs from Coke to milk (~57%) because the denominators differ; association is co-occurrence, not causation.

Self-check: In a log of 10 baskets, milk appears in 5 and Coke in 4 of those; what is P(Coke | milk)?

Connects to: Association Rule Mining Applications; Predictive and Descriptive Functions; The Data Mining Process Pipeline

Association Rule Mining Applications

Must-know: Find the association (part fails when condition present) and keep inventory local; warehouse placement enables one-day delivery promises; shelf layout and airport routing follow the same rule shape.

⚠️ Top pitfall: Treating mined co-occurrence rules as proof of causation.

Self-check: Why does an appliance company stock compressors at its Goa depot?

Connects to: Association Rule Mining

Anomaly Detection

Must-know: Anomaly detection = profile of normal behavior + alert on deviation; uses: credit card fraud and network intrusion; difference from classification: classification learns a boundary between classes, anomaly detection learns normal and flags the rest.

⚠️ Top pitfall: Assuming the profile is static — normal behavior drifts, and never-seen attack patterns may look normal.

Self-check: Why does a US login from a Goa-based profile trigger extra authentication?

Connects to: Classification Applications; Classification

Data Preprocessing: From Physical Entities to Datasets

Must-know: Preprocessing converts raw data into a form mining algorithms can run on; passbook example (Account ID, Transaction ID, Debit, Credit, Balance) and medical log example (Patient, Treatment, Medicine, Doctor, Room); the hardest step is choosing what to capture — dataset creation is subjective and depends on the question asked.

⚠️ Top pitfall: Blaming the algorithm when the model underperforms — the failure is often the dataset: missing attributes, wrong attributes, or coarse classes.

Self-check: What columns does the bank passbook dataset have, and why is it not yet ready for mining?

Connects to: Objects and Attributes; The Data Mining Process Pipeline

Objects and Attributes

Must-know: Rows = objects, columns = attributes; the five synonyms (record, point, sample, entity, instance) mean one unique row; a row of n attributes is an n-dimensional vector (attribute vector/feature vector in the references); categorical (fixed values) vs continuous (infinite values) is the preview taxonomy.

⚠️ Top pitfall: Treating attribute and attribute value as the same thing — the attribute is the property, the value varies from object to object.

Self-check: Why is a row with four attributes an n-dimensional point, and what is n here?

Connects to: The Classification Dataset: Attributes, Train and Test; Clustering

Exam Guidance Summary

Must-know: Course structure: classification module 4, association rule mining module 5, clustering module 6 (T-shirt size question, boundaries, hard/soft), anomaly detection one class only; conceptual checklist: prediction vs description, classification vs regression, categorical vs continuous, hard vs soft clustering, pipeline with feedback loop.

Self-check: Which module covers clustering, and which two open questions does it answer?

Connects to: Classification; Clustering; Market Segmentation: A Clustering Case Study; Association Rule Mining; Anomaly Detection

Key Industry Applications

Must-know: Named applications per method: classification (direct marketing, fraud, churn), clustering (Google News, stocks, T-shirt sizes), association rules (inventory by location, shelves, airport layout, ad psychology), anomaly detection (fraud and intrusion via behavior profiles).

Self-check: Which application demonstrates clustering in stock analysis, and what exactly is being grouped?

Connects to: Classification Applications; Clustering Applications; Market Segmentation: A Clustering Case Study; Association Rule Mining Applications; Anomaly Detection

Previous LectureLecture 01 · Data Mining Foundations and Core Concepts

Next LectureLecture 03 · Data and Data Preprocessing

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.