—

Data Mining

Data and Data Preprocessing

Published: 2026-08-05

Level: postgraduate

Audience: Postgraduate students in Data Mining

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Objects and attributes — covered in Lecture 2
Attribute values — covered in Lecture 2
From physical entity to digital form — covered in Lecture 2
Data preprocessing — covered in Lecture 2
Types of attributes (preview) — covered in Lecture 2

These notes cover the raw material of every data mining project: data itself. We start with what data is — physical entities turned into digital tables of objects and attributes — and then meet the four attribute types (nominal, ordinal, interval, ratio), the mathematical operations each type permits, and the second, independent split into discrete and continuous attributes. Next come the three ways data sets are represented: record form (data matrices, document data, transaction data), graph form (the web, chemical molecules), and ordered form (sequences, spatio-temporal data). The second half of the session turns to data preprocessing: the six parameters of data quality, the four quality problems (noise, outliers, missing values, duplicates), the six methods for handling missing values, and data smoothing for noise.

By the end of this session you should be able to answer five questions: What exactly is an object and an attribute, and how do they map to a table? How do I classify an attribute into one of the four types, and which mathematical operations does each type allow? What is the difference between discrete and continuous attributes? Which of the three data forms — record, graph, ordered — fits a given data set? And what are the standard preprocessing steps for fixing poor-quality data before any mining begins?

3.1 What Is Data

3.1.1 From Physical Entity to Digital Form

Hook: A bank passbook sits in your drawer — paper, ink, and numbers. You cannot run a mining algorithm on the paper itself. You can run one on the table of transactions that the passbook becomes once it is typed into a computer. That single step — turning a physical thing into digital form — is where all data mining begins.

Data mining works on data that has been turned into a digital form. Any physical entity — a person, a transaction, a web page, a molecule — can be represented digitally, and once it is in that digital form we can run advanced statistical methods on it to pull information out. The point of data mining is to take collected data and extract knowledge from it.

A familiar example is a bank passbook. The passbook is a physical entity: a booklet of printed rows. Convert it to a digital form and you get a table whose entries follow a fixed structure: first the date of each transaction, then a transaction ID, then a description, then whether the entry is a debit or a credit, and finally the running balance. Once the passbook lives as a table, a mining algorithm can process it. The physical entity itself cannot be mined; its digital representation can.

Worked example — the passbook becomes a table. Take a passbook with four transactions and write them as rows with fixed columns:

Date	Transaction ID	Description	Debit / Credit	Balance
2026-07-01	T-1001	Salary credit	Credit	45,000
2026-07-05	T-1002	Grocery store	Debit	43,200
2026-07-12	T-1003	ATM withdrawal	Debit	41,000
2026-07-20	T-1004	Interest	Credit	41,030

Every row has the same structure — five fixed columns, in a fixed order. The physical booklet has become a table, and that table is what an algorithm can scan, sort, sum, and mine. The sense-check: no information was invented in the conversion; the same entries exist in the booklet, only now in machine-readable form.

3.1.2 Objects and Attributes

Definition. Data is a collection of objects and their attributes. An object — one entity, one thing we care about — is described by a collection of attributes. So an object is defined by its attributes, and the attribute values are what actually pin the object down.

Example: a person (object) with identifier TID 1 is described by four attributes: refund, marital status, taxable income, and cheat-or-not. Change the attributes and you are describing a different object. Another example: to define a person you might use the name, the department they belong to, the college they attend, and the city they live in — each of these is an attribute, and the specific values (the department name, the city name) are the attribute values that define that particular person.

The same idea travels under many names. An object is also called a record, a point, a sample, an entity, or an instance — they all mean the same thing: one object. An attribute is also called a variable, a field, a characteristic, a dimension, or a feature. When you read "feature vector" or "data point" in another book, the speaker is still talking about one row of objects and attributes.

Worked example — the TID 1 person. Here is a small table of four objects, each defined by the same four attributes:

TID	Refund	Marital Status	Taxable Income	Cheat
1	Yes	Single	125,000	No
2	No	Married	100,000	No
3	No	Single	70,000	No
4	Yes	Married	120,000	No

Each row is one object; each column is one attribute; each cell is one attribute value. If we changed row 1's marital status from Single to Divorced, the cell changes but the attribute stays the same — we would be describing a different version of the object, not a different attribute. Sense-check: delete any column and the rows no longer carry the same information, which is why the choice of attributes matters.

The natural way to lay data out is in a table: rows represent objects and columns represent attributes. This tabular picture is the mental model for almost everything in this material. A row with $d$ attributes behaves like a point in $d$ -dimensional space — a table with $n$ rows and $d$ columns is an $n \times d$ grid where each row is one object.

Intuition + analogy. Think of a registration form with blanks on it. The blanks are the attributes — name, date of birth, address, phone number. What you write into each blank is the attribute value. A form with the same four blanks can describe a million different people, because the values change; the set of blanks is what stays fixed. The form is the schema, the filled forms are the objects.

3.1.3 Attribute Values

An attribute can take one or more values. The set of values an attribute can hold matters. Refund takes exactly two values, yes and no. Marital status takes three: single, married, and divorced. Taxable income is different — it is a continuous attribute, so it can take an infinite number of values. The attribute values are the numbers or symbols assigned to an attribute, and they are what let us actually represent each object.

Worked example — counting what an attribute can hold. Take the four attributes of the table above:

Refund: value set {yes, no} — 2 values.
Marital status: value set {single, married, divorced} — 3 values.
Taxable income: any real number, say 70,000 to 220,000 — infinitely many values (between any two, another exists).
Cheat: value set {yes, no} — 2 values.

The size of the value set is not decoration: it decides which attribute types and operations are possible, which the next section formalizes. Sense-check: refund and cheat both have two values, so any operation that works on one works on the other — but taxable income, with its infinite value set, allows operations the others cannot.

Pitfalls.

Attribute versus attribute value. The attribute is the property ("marital status"); the value is what a particular object has ("single"). Mixing them up leads to nonsense like "the marital status of the table is single."
Looks numeric, behaves symbolic. A customer ID written as digits is still a label, not a number — subtracting two IDs produces a meaningless result. The next section explains why.
A table row is not the data set. Objects and attributes are the vocabulary; the full data set includes the choice of which attributes exist, which is exactly what preprocessing later questions.

Recap + bridge. Data = objects (rows) + attributes (columns); objects are also called records, points, samples, entities, or instances, and attributes are variables, fields, characteristics, dimensions, or features. The values an attribute can hold determine what we can do with it — which leads directly to the question of the next section: what kinds of attributes exist, and what each kind permits.

In real life the same idea shows up everywhere: a bank's transaction table, an online store's order table, a hospital's patient table — all are collections of objects and attributes, all laid out as rows and columns, and all ready for mining once the physical world has been digitized.

3.2 The Four Attribute Types

3.2.1 Nominal Attributes

Definition. A nominal attribute is one whose values are essentially symbols or identifiers. The values carry no order and no quantity; they just label or identify. Given two values, the only thing we can say is "same" or "different."

Examples:

An ID number, such as an enrollment-style identifier like "2020A7 1001G", uniquely identifies one person — a fixed value, an identifier. (The exact digits are arbitrary; what matters is that the ID is a fixed label pointing at exactly one object.)
A name is a nominal attribute — it identifies a person.
Eye color is nominal — its values are labels like black, brown, green.
A zip code is nominal — it is an identifier that uniquely points to a specific area on the map.

Intuition + analogy. Nominal values are like license plates: each plate labels one car, but a plate number is not "greater than" another plate in any meaningful sense. Saying "plate 1234 equals plate 1234" works, and "plate 1234 differs from plate 5678" works. Everything beyond that — sorting, adding, comparing — is not meaningful for a label.

The key property: nominal attributes provide only enough information to distinguish one object from another. Given two values, the only statements that make sense are equality $=$ and inequality $\neq =$ . A zip code of 110001 and a zip code of 110002 are different areas; there is no arithmetic meaning to subtracting one from the other.

Pitfall: digit-shaped labels. A nominal attribute is often stored as digits — employee IDs, zip codes, phone numbers, PINs. The digits make it tempting to compute with them, but every such computation is meaningless: the mean of two zip codes is not a zip code. The value is a symbol that happens to be printed with digits. The reference texts make the same point with employee ID numbers: even when represented as integers, they should be treated like symbols.

3.2.2 Ordinal Attributes

Definition. An ordinal attribute is one whose values can be arranged in a particular order. The order is meaningful even though the gaps between values are not.

Example: grades. A is better than A−, which is better than B, which is better than B−, which is better than C, which is better than C−. The grade values can be sorted from highest to lowest, so grade is an ordinal attribute. The ranking of a food product on a scale from 1 to 10 is ordinal, and so is height expressed as large, medium, short — the values line up in a sequence.

Worked example — the grade ordering. Sort the six grade values from best to worst:

A > A− > B > B− > C > C−

Now read it in reverse: C− < C < B− < B < A− < A.

The ordering carries real information — "A beats B" is a valid statement — but the size of the gap does not. The distance from A to A− is not equal to the distance from B to B−, and we cannot say that A is "twice as good" as C. That is exactly why grade is ordinal and not interval or ratio. Sense-check: the one statement that is definitely false for grades is any arithmetic one, such as "A minus B equals B minus C."

Other standard examples include hardness of minerals (the Mohs scale: talc < gypsum < calcite < ... < diamond), quality ratings like {good, better, best}, and street numbers on a road — all of them order objects without promising equal spacing. The median is a sensible summary for ordinal values ("half the class is above B"), but the arithmetic mean is not, because averaging assumes the gaps are real numbers.

3.2.3 Interval Attributes

Definition. An interval attribute is one where there is a fixed interval between the values — the gaps between consecutive values are equal and meaningful. Differences are meaningful; ratios are not, because the zero point is arbitrary.

Example: temperature. Between 1 degree and 2 degrees there is a fixed gap of one degree. Calendar dates are another example: between the 28th and the 29th there is a fixed 24-hour gap, and the same 24-hour gap separates the 29th from the 30th. The interval is constant, which is what makes the attribute interval-type.

Worked example — date arithmetic. Calendar dates are interval attributes because the gaps are equal:

28th + 1 day = 29th (the gap from 28th to 29th is 24 hours).
29th − 1 day = 28th (subtraction works in the other direction).
29th − 28th = 1 day, and 30th − 29th = 1 day — the two gaps are the same 24 hours, which is the defining property of an interval scale.

Temperature in Celsius works the same way: the step from 20° to 21° is one degree, exactly the step from 35° to 36°. But "today is twice as warm as yesterday" is meaningless on the Celsius scale, because 0° Celsius is an arbitrary point (the freezing point of water), not a true absence of heat. Sense-check: an interval scale supports statements of difference, never statements of ratio.

The reference texts confirm the same examples: calendar dates and temperature in Celsius or Fahrenheit are the standard interval attributes. The rule behind it: two valid scales for the same interval attribute differ by a linear shift and a rescaling, $new value = a \times old value + b$ , and the zero point ( $b$ ) is arbitrary. Because gaps stay equal under such a shift, the mean and standard deviation are legitimate summaries for interval attributes.

3.2.4 Ratio Attributes

Definition. A ratio attribute is one where the ratio between two attribute values is meaningful — you can divide one value by another and the result tells you something real. Ratio scales have a true, non-arbitrary zero, so both differences and ratios make sense.

Example: distance. Say Delhi to Nagpur is 1000 kilometers and Delhi to Bangalore is 2000 kilometers. We can say the distance from Delhi to Bangalore is twice the distance from Delhi to Nagpur — the ratio

$\frac{2000 km}{1000 km} = 2$

is a valid statement. Length, time, and count are also ratio attributes. Ratio attributes support addition, subtraction, multiplication, and division between values.

Worked example — distance ratios. Two road distances from Delhi:

Delhi to Nagpur: 1000 km.
Delhi to Bangalore: 2000 km.

The ratio is $\frac{2000}{1000} = 2$ , so Bangalore is twice as far as Nagpur. (The ratio survives any change of unit: in miles, the same distances give $\frac{1243}{621} \approx 2$ — a ratio scale has no arbitrary zero, so the comparison is unit-independent.)

The same logic applies to age: a 40-year-old is twice as old as a 20-year-old, because age has a true zero (birth). It applies to counts: 4 items is twice as many as 2 items. Sense-check: every ratio statement survives changing the unit of measurement, which is the signature of a true zero.

Temperature makes a useful contrast: on the Kelvin scale, which has a true zero at absolute zero, 200 K is twice 100 K in a physically meaningful way; on the Celsius or Fahrenheit scales, whose zeros are arbitrary, the same doubling statement is not physically meaningful. That is why temperature is interval on Celsius/Fahrenheit but ratio on Kelvin. Other standard ratio examples are monetary quantities, counts, age, mass, length, and electrical current.

The four types form a ladder: each higher type keeps everything the types below it can do, and adds a new power — nominal can only distinguish, ordinal adds order, interval adds equal gaps, ratio adds a true zero and meaningful ratios. That cumulative structure is the subject of the next section, which makes the ladder precise by counting operations.

Recap + bridge. Four attribute types: nominal (labels, only same/different), ordinal (order, no equal gaps), interval (equal gaps, arbitrary zero), ratio (true zero, meaningful ratios). The ladder is cumulative — ratio is the strongest, nominal the weakest. The natural next question: exactly which mathematical operations does each type permit? The next section counts them.

3.3 Mathematical Operations on Attributes

3.3.1 The Eight Operations

Once we know the attribute types, we can ask which mathematical operations apply to each type, because the operations decide what kind of classification, clustering, or other data mining model we can build. The four attribute types accept the operation families in increasing order. There are four families of operations, two operations each, for eight operations in total:

Definition — the four families of operations.

Distinctness — checking whether two attribute values are equal or not equal: $=$ and $\neq =$ .
Order — comparing values: $>$ and $<$ .
Addition — combining values with $+$ ; the family also includes subtraction $-$ .
Multiplication — scaling values with $\times$ , including division $\div$ .

In symbols:

$family 1: {=, \neq =} family 2: {<, >} family 3: {+, -} family 4: {\times, \div}$

Two operations per family, four families, eight operations total. The reference texts list the same four properties of numbers — distinctness, order, addition, and multiplication — as the building blocks used to define attribute types.

For a name attribute, "the same name is equal to the same name" and "one name is not equal to a different name" are both valid statements. These two operations — equal to, not equal to — are what distinctness means. With grades, A > B > C > D, and reading the other way, D < C < B < A. For calendar dates, today is the 28th; add one day and it becomes the 29th; subtract one day and it becomes yesterday. Both operations work on dates. With distances, we can divide one distance by another and get a ratio, which is exactly why distance is a ratio attribute.

Worked example — one operation family per attribute. Run each family on a concrete value:

Family	Example value	Statement that works	Statement that fails
Distinctness	Name = "Ananya"	"Ananya" = "Ananya"	—
Order	Grade = A	A > B	A + B (not defined)
Addition	Date = 28th	28th + 1 day = 29th	28th × 2 = ? (no meaning)
Multiplication	Distance = 1000 km	2000 km ÷ 1000 km = 2	1000 km + name (not defined)

Each family is legal exactly on the attribute types whose definition contains it — which is the ladder of the next subsection. Sense-check: every operation statement above is either a true statement about the value or a meaningless combination, and nothing in between.

3.3.2 Operations Allowed for Each Attribute Type

The four attribute types accept the operation families in increasing order:

Nominal — only distinctness: $=$ , $\neq =$ . Two operations. We cannot multiply two names, and we cannot subtract one eye color from another.
Ordinal — distinctness plus order: $=$ , $\neq =$ , $<$ , $>$ . Four operations. A grade A equals A, A is greater than B, A is not equal to B — but we still cannot add or multiply grades.
Interval — distinctness, order, and addition: $=$ , $\neq =$ , $<$ , $>$ , $+$ , $-$ . Six operations. Temperature differences are meaningful, so 20° plus one degree is 21°.
Ratio — all four families: $=$ , $\neq =$ , $<$ , $>$ , $+$ , $-$ , $\times$ , $\div$ . Eight operations. Distance supports everything.

So the hierarchy is: nominal supports 2 operations, ordinal 4, interval 6, ratio 8. Every higher type also supports the operations of the types below it.

Formalize — the operation ladder. Count the operations as families are granted:

$nominal: 2 = {=, \neq =}$ $ordinal: 4 = {=, \neq =, <, >}$ $interval: 6 = {=, \neq =, <, >, +, -}$ $ratio: 8 = {=, \neq =, <, >, +, -, \times, \div}$

The counts grow by two at every step: $2 \to 4 \to 6 \to 8$ . Each type has all the operations of the types above it in the ladder — in the reference treatment, the definition of the attribute types is cumulative, and any operation valid for a nominal, ordinal, or interval attribute is also valid for a ratio attribute.

Attribute type	Operations allowed	Number of operations
Nominal	$=$ , $\neq =$	2
Ordinal	$=$ , $\neq =$ , $<$ , $>$	4
Interval	$=$ , $\neq =$ , $<$ , $>$ , $+$ , $-$	6
Ratio	$=$ , $\neq =$ , $<$ , $>$ , $+$ , $-$ , $\times$ , $\div$	8

Pitfall: assuming the reverse. The ladder is one-directional. Nominal does not gain order by being written as numbers, and interval does not gain ratio just because its values look like decimals. Adding two ordinal grades ("A + B") stays meaningless no matter how the grades are coded.

3.3.3 Treat Attributes at Their Highest Level

The rule of thumb. Treat every attribute at its highest possible level, because that maximizes the mathematical operations we can apply, and more operations means more information extractable from the feature. Ratio sits at the highest level, nominal at the lowest — ratio accepts every operation, nominal accepts only two.

A ratio attribute like distance can be treated as a nominal attribute. The problem: we then cut our operations down to just $=$ and $\neq =$ — no subtraction, no multiplication, no division. We would not be able to apply many mathematical functions, and we would not extract much information from that feature.
The reverse direction fails. If we try to treat a nominal attribute like a name as a ratio attribute, we would have to define what it means to add two names or subtract two names — and that is not possible in almost all cases.

The recommendation: always treat distance as a ratio attribute rather than a nominal attribute, and always treat temperature as an interval attribute rather than a nominal attribute, because downgrading the type restricts the operations and can block information extraction.

Worked example — what downgrading costs. Take one employee's taxable income, say 60,000, and compare two treatments:

Treated as ratio (highest level). Every operation works: we can compare (60,000 < 70,000), subtract (70,000 − 60,000 = 10,000), scale (60,000 × 0.3 = 18,000 tax estimate), and divide (income ÷ per-capita income = relative wealth). A model can ask "income above 50,000?" or "income in the top 20%?" — rich questions, each backed by a legal operation.
Treated as nominal (downgraded). Only $=$ and $\neq =$ survive: we can ask "is income exactly 60,000?" and nothing else. The question "is this customer richer than the average?" is illegal — comparing is an order operation, which nominal does not have.

The same data, two information levels: ratio treatment extracts all eight operations' worth of information; nominal treatment leaves only equality. The recommendation is to pick the highest level the attribute genuinely supports — not the lowest.

3.3.4 Student Questions and Answers

Q: A student asked something about nominal attributes and the equals and order operations — "nominal equals and order" was all that came through, and the rest of the question was garbled.

A: The question did not come through in full, so a repeat was requested. The follow-up exchange clarifies the underlying doubt: the relationship that exists between the values of an attribute — equality only, or also order — is what determines the attribute type. If only equality relationships exist between values, the attribute is nominal; if an ordering relationship also exists, it is ordinal. The type is a property of the relationship between values, not of how the attribute was typed into a computer.

Recap + bridge. Four families (distinctness, order, addition, multiplication) give eight operations; nominal allows 2, ordinal 4, interval 6, ratio 8; the ladder is cumulative, so an attribute should always be treated at its highest possible level. The next section adds a second, independent way to categorize attributes: discrete versus continuous.

3.4 Discrete and Continuous Attributes

3.4.1 Discrete Attributes

Besides the four types above, there is a second, independent way to categorize an attribute: discrete or continuous.

Definition. A discrete attribute has a fixed, countable set of values — the number of attribute values is fixed and countable.

The cheat-detection attribute takes only two values, yes and no. Two attribute values, so it is discrete — and because it has exactly two values it is also called binary.
Refund is the same: two values, discrete and binary.
Marital status has exactly three unique values — single, married, divorced — so it is discrete and ternary (three values).

The reference texts define a discrete attribute as one with a finite or countably infinite set of values, often stored as integer variables. Binary attributes are the special case that assumes only two values — true/false, yes/no, male/female, or 0/1 — and are often represented as Boolean variables.

3.4.2 Continuous Attributes

Definition. A continuous attribute has values of real-number type, which means it can take an infinite number of values.

Temperature, height, weight, and taxable income are continuous attributes. There is no fixed list of possible values; between any two values there is always another one. Height is a good example: between 170.0 cm and 170.1 cm there is 170.05 cm, and between that and 170.0 there are infinitely many more — the value set is a whole stretch of the real line, not a countable list.

Intuition + analogy. Discrete is a light switch — a fixed set of positions (off, on). Continuous is a dimmer knob — infinitely many positions between minimum and maximum. The switch can only answer "off or on"; the dimmer can answer "how bright, exactly?" That is the practical difference: discrete attributes support counting and listing, continuous attributes support measurement and arithmetic.

Pitfalls.

The two axes are independent. The four types (nominal → ratio) and the discrete/continuous split are different questions about the same attribute. A combination can be unusual — a continuous binary attribute is hard to imagine, since "two values" and "real-valued" pull against each other — but ordinary pairs like discrete-ratio exist: the number of children in a family is discrete (0, 1, 2, ...) and ratio (true zero, ratios meaningful). In the standard treatment, nominal and ordinal attributes are typically discrete, while interval and ratio attributes are typically continuous — but counts, which are discrete, are also ratio attributes.
"Infinite" does not mean "any number." A continuous attribute like taxable income still has practical bounds; the point is that between any two legal values another legal value exists.

3.4.3 Student Questions and Answers

Q: For string attributes, how are we going to apply these mathematical operations?

A: The how — the concrete mathematical machinery for handling each type — is a data mining techniques topic and comes later. What matters at the data stage is the relationship between attribute values: if the values of an attribute carry an ordering relationship (equal/not equal, greater/less), the attribute is called an ordinal attribute; if only equality relationships exist, it is nominal. The presence of the relationship is what defines the type. How we use it is discussed later.

Worked example — classifying attributes on both axes. Fill in both categorizations for six attributes:

Attribute	Four-type ladder	Discrete or continuous
Refund (yes/no)	Nominal	Discrete — binary (2 values)
Grade (A, A−, B, ...)	Ordinal	Discrete (6 values)
Marital status (single/married/divorced)	Nominal	Discrete — ternary (3 values)
Temperature (Celsius)	Interval	Continuous
Distance (km)	Ratio	Continuous
Number of children	Ratio	Discrete (0, 1, 2, ...)

The table shows the two axes are independent: marital status is nominal and discrete, while number of children is ratio and discrete. Sense-check: every cell in the right column is decided by counting the value set — countable means discrete, real-valued stretch means continuous — while the middle column is decided by the relationships between values.

Recap + bridge. Discrete attributes have a fixed, countable value set (binary and ternary are special cases); continuous attributes take real-number values with infinitely many possibilities. The two categorizations are independent: nominal/ordinal/interval/ratio describes relationships, discrete/continuous describes value-set size. Next we step up from single attributes to whole data sets and the three forms they come in — record, graph, and ordered.

3.5 Record Data

3.5.1 Data Matrix

A data set is a collection of objects and attributes, and there are three main ways of representing it: record form, graph form, and ordered form. The form matters because different data mining algorithms operate on different forms.

Definition. Record data is the tabular form discussed earlier: data that consists of a collection of records, where each record consists of a fixed number of attributes. Any data that can be laid out as a table is record data. The reference texts add that each record uses the same fixed set of fields, and record data is usually stored in flat files or relational databases.

The cheat-detection example — customers and their attribute values in a table — is record data. There are three kinds of record data: data matrix, document data, and transaction data.

A data matrix is the simplest tabular form: rows represent objects and columns represent attributes. That is the entire definition — a table where rows are objects and columns are attributes. An example is a table of parcels, parcel one and parcel two, with attributes like X load and Y load recorded for each.

Worked example — a parcel data matrix. Four parcels, each measured on four numeric attributes:

Parcel	X load	Y load	Distance	Thickness
P1	10.23	5.27	27	1.2
P2	12.65	6.25	22	1.1
P3	13.54	7.23	23	1.2
P4	14.27	8.43	25	0.9

Rows are objects (the parcels), columns are attributes (the measurements). Because every value is numeric, each row is a point in 4-dimensional space, and the whole table is a 4 × 4 matrix. That is what makes a data matrix special among record forms: standard matrix operations can be applied to transform and manipulate it, which is why the data matrix is the standard format for most statistical data. Sense-check: swap rows and columns and the table stays meaningful (objects as columns, attributes as rows), confirming the pure tabular structure.

3.5.2 Document Data

Document data represents documents (articles, research papers) as rows of a table. It answers the question: how do we turn a pile of documents into a table a mining algorithm can read?

The motivating problem is a plagiarism detection system. Suppose thousands of research papers and news articles are published every year, and we want to ensure no article is added to a repository if it is completely copied from an existing source. The input is paper one, paper two, paper three, and so on, and we want to detect whether two papers match each other — if a new paper matches an existing one, it is a plagiarized copy.

How do we represent the papers? We convert each document into a row of a table:

Go through the documents and find the important words in each — for document one, words like "team", "coach", "plays"; for document two, words like "ball", "score", "game".
Collect the unique important words across all documents — say 1000 unique words. These become the columns.
Fill each cell with the frequency of that word in that document: how many times the word appears.

So in document one, "team" appears 3 times, "coach" appears 0 times, "plays" appears 5 times. In document two, "coach" appears 7 times while "team" and "plays" do not appear at all. The table cells hold word frequencies, and that frequency matrix is document data — a tabular form of record data for text.

Worked example — the document-frequency matrix. Two sports articles and six important words:

Document	team	coach	plays	ball	score	game
Document 1	3	0	5	0	2	2
Document 2	0	7	0	1	0	3

Each cell says how often the column word appears in the row document: "team" appears 3 times in document 1 and 0 times in document 2; "coach" appears 7 times in document 2. A newcomer paper becomes one more row of counts; the system compares its frequency pattern against every stored row and flags it if the pattern matches an existing document. Sense-check: the two rows have very different fingerprints — document 1 leans on team/plays, document 2 on coach — so they are different papers; two copies of the same paper would produce near-identical rows.

This is the document-term matrix of the standard texts: documents as rows, terms as columns, counts as values. Because any single document uses only a small fraction of the vocabulary, most cells are zero, which is why document data is usually stored as a sparse matrix — only the non-zero entries are kept.

3.5.3 Transaction Data

Transaction data is a record-form representation of transactions. Suppose we are in a shopping mall and we have a list of what each user purchases: user one is purchasing three items, user two is purchasing two items, and so on. This transaction log can be represented in record (tabular) form, and that representation is called transaction data. The general rule: any transaction that can be represented in a record form is transaction data. This is the setup that later supports market-basket style mining.

Worked example — a mall transaction log. Five shopping trips, each row one user's basket:

TID	Items purchased
1	bread, soda, milk
2	beer, bread
3	beer, soda, diaper, milk
4	beer, bread, diaper, milk
5	soda, diaper, milk

User one purchases three items (bread, soda, milk); user two purchases two items (beer, bread). The log is record data — one row per transaction, a fixed structure — and the same content can be rewritten as a sparse table with one column per product and 0/1 entries saying whether the product was in the basket. Sense-check: if every row had different columns, it would not be a table, which is why "fixed structure per record" is part of the definition.

In the reference texts this is called market basket data: each record is a set of items bought together, usually stored as binary (asymmetric) attributes. Association rule mining — the "customers who buy X also buy Y" family of algorithms from a later module — runs directly on this tabular form.

Recap + bridge. Record data is the table family: data matrix (numeric, rows = objects, columns = attributes), document data (documents as rows, word frequencies as columns), and transaction data (one row per basket). All three are tables, and tables are what most mining algorithms expect. But not every data set fits a table — the next section looks at graph form.

3.6 Graph Data

3.6.1 The World Wide Web as a Graph

Not every data set fits a table. Some physical entities are naturally represented as graphs — nodes connected by links — and a different family of mining algorithms applies to graph data. The reason for categorizing data forms is that each digital form supports its own category of mining algorithms.

Definition. A graph is a collection of nodes (the objects) and links (the relationships between them). The links themselves carry the information: direction, presence, and weight. In the reference treatment, the data objects are mapped to nodes while the relationships among objects are captured by the links.

The World Wide Web consists of millions of web pages. Most pages contain hyperlinks that connect to other pages, so the web can be drawn as a graph: web page one has a hyperlink to web page two, web page one has another hyperlink to web page three, web page three and web page five link to each other, and so on.

A familiar pattern: a Wikipedia page on India has a link to a page listing states; clicking one state leads to a page about that state's districts; click a district and you go deeper. The connections between pages are always there, and the whole web can be represented as graph data.

Worked example — drawing the web as a graph. Five pages with these links: page 1 links to pages 2 and 3; page 3 links to page 5; page 5 links back to page 3; page 2 links to page 4.

Page	In-links (who points to it)	In-degree
1	—	0
2	1	1
3	1, 5	2
4	2	1
5	3	1

The picture is: nodes P1–P5 with directed arrows between them, P3 receiving two arrows. The table form would need one column per page-relationship pair and would still hide the structure; the graph shows it at a glance. Sense-check: the graph and the table describe the same five objects — the graph just makes the relationships (the arrows) the first-class citizen.

3.6.2 Page Ranking on Graph Data

Once the web is a graph, we can run a category of mining algorithm on it — the page ranking algorithm, which helps in searching. Page ranking works on graph data: given a keyword search, the algorithm arranges the pages in an order for that search. The physical entity (web pages) is converted to a digital form (a graph), and on that graph data we apply page-ranking algorithms to decide which pages to show first.

Intuition + analogy. Page ranking is like popularity among people: the more people point at you, the more visible you are — and a point from a popular person counts for more than a point from a nobody. In the web graph, "pointing at" is a hyperlink: a page accumulates importance from the pages that link to it, and links from already-important pages count more. The reference literature formalizes this as PageRank: every link is treated as a vote of confidence, and a page's rank is built from the rank of its incoming voters.

Real-world: page ranking is the mechanism behind web search engines like Google. Every time you search, something must decide the order of results, and that decision is a graph-based mining operation on the link structure of the web.

Pitfalls.

Rank is not keyword match. Page ranking decides the order of results for a keyword search; it is not itself the text matching. Both work together — match the query, then order the matches by link importance.
The arrows have direction. A page that links out gives influence away; a page that is linked to receives it. Counting links without directions gives the wrong answer.
Link spam exists. Because links carry importance, people build fake link farms to inflate their pages' rank — which is why real ranking algorithms add safeguards. The core idea students should keep: importance flows along the graph's links.

3.6.3 Chemical Data as Graphs

Chemical compounds are a second example of graph data. A benzene molecule — a physical entity — can be represented graphically, with atoms as nodes and bonds as links. Once the molecule is in graphical form, we can run mining algorithms on it to extract knowledge, for example about molecular structure and similarity.

Worked example — benzene as a graph. The benzene molecule $C_{6} H_{6}$ is a ring of six carbon atoms, each bonded to two neighbors in the ring and to one hydrogen atom. As a graph:

Nodes: 6 carbon atoms + 6 hydrogen atoms = 12 nodes.
Links: 6 carbon–carbon bonds (the ring) + 6 carbon–hydrogen bonds = 12 bonds.

The ring shape — each carbon connected to two carbons — is visible in the graph but hidden in the chemical formula. Mining over such graphs answers questions like "which substructures appear often across many compounds?" and "is the presence of a substructure linked to a property such as melting point?" — the branch of mining the reference texts call substructure mining. Sense-check: the graph loses no chemical information needed for structure questions, which is why molecular similarity is studied on graphs, not on plain text formulas.

Recap + bridge. Graph data is the form for anything whose relationships are the information: web pages connected by hyperlinks (mined by page ranking), molecules connected by bonds (mined for structure and similarity). Graphs get their own family of algorithms because tables would hide the arrows. One more family of forms remains — ordered data, where the position in time or space is the information.

3.7 Ordered Data

3.7.1 Sequences: Genomes and Transactions

The third form is ordered data: data that can be arranged in a certain order, where the sequence position itself carries meaning.

Definition. In ordered data, the order of the values is part of the information. Remove the order and the data loses meaning. The standard texts distinguish two flavors: sequential data, where each record carries a time stamp, and sequence data, where there are positions in an ordered sequence without time stamps.

A genome sequence is ordered data — the sequence of bases is arranged in a particular order, and the order is the information.
Transaction sequences are ordered: transaction one happens first, transaction two happens second, then D happens, then C and E happen. The fact that one transaction precedes another is part of the data, so a sequence of transactions is ordered data.

Worked example — reading a genome sequence. A genome is a string over four nucleotide symbols — A (adenine), T (thymine), G (guanine), C (cytosine). A short stretch might read:

GGTTCCGCCTTCAGCCCCGCGCC

Position by position: the 1st base is G, the 2nd is G, the 3rd is T, the 4th is T, and so on. Shuffle the letters into GCTCGTCCTGCCTCCGCCACGG and you get a different molecule with different properties — the order is the information. This is why genomic analysis compares subsequences positionally: similarity of genes means similarity of ordered base strings. Sense-check: a genome is like a sentence — the same letters in a different order say something different.

A transaction sequence works the same way: customer X buys item A at time $t_{1}$ , items C and D at time $t_{2}$ , and item E at time $t_{5}$ . The ordering $t_{1} < t_{2} < t_{5}$ is part of the record — "people who buy a DVD player tend to buy DVDs shortly after" is a pattern that only exists because purchase times are ordered. The standard treatment calls this temporal behavior: when working with time-stamped data, measurements close in time tend to be similar, an effect known as temporal autocorrelation.

3.7.2 Spatio-Temporal Data

Spatio-temporal data combines spatial (distance or location) and temporal (time) structure — the two axes shown by the classic chart of average monthly temperature of land and ocean, with months across the top.

The spatial reading: Delhi is much hotter than Beijing, because Delhi is much closer to the equator than Beijing.
The temporal reading: certain months in Beijing are very cold (shown blue on the chart) while other months are hot — May and June are hot.

So the same data can be read along the distance axis and along the time axis, and that combination is spatio-temporal ordered data.

Visual intuition — the land-and-ocean temperature chart. The figure plots month (January to December) on the horizontal axis against average temperature in degrees on the vertical axis, with one curve for land and one for ocean, and each city's curve marked across the year. Two readings of the same chart: spatially, compare curves between cities — Delhi's curve sits far above Beijing's because Delhi is close to the equator (latitude is the distance axis); temporally, follow one city's curve across the months — Beijing dips blue in winter and climbs in May and June. The one-sentence takeaway: the data is not just "temperatures" — each value has a location and a time, and both axes are information.

The reference texts treat the same idea as spatial and temporal data: objects that are physically close tend to be similar (spatial autocorrelation), and measurements close in time tend to be similar (temporal autocorrelation). Weather and climate records are the classic example — temperature and pressure measured at grid points over the globe, month after month.

Worked example — reading the chart both ways. Take three cells of the land/ocean temperature data:

Delhi, May: hot (equator-side latitude, pre-monsoon month).
Beijing, January: very cold (northern latitude, deep winter) — blue on the chart.
Beijing, June: hot (same location, but summer month).

The first row differs from the third by space (Delhi vs Beijing at the same month), and the second from the third by time (January vs June at the same place). A model that ignores either axis confuses "same place, different season" with "same season, different place" — which is why spatio-temporal data is ordered along both axes. Sense-check: each value's meaning depends on where and when it was measured; drop either coordinate and the pattern disappears.

Recap + bridge. Ordered data is the third form: genomes and transaction sequences carry their meaning in the order of their symbols, and spatio-temporal data carries meaning along both a distance axis and a time axis. With all three forms — record, graph, ordered — in place, the session turns to the practical problem that shadows every data set from the real world: its quality is poor, and preprocessing exists to fix it.

3.8 Data Preprocessing and Data Quality

3.8.1 Why Preprocess Data

Raw data from the real world is in poor shape. Converting a physical entity into a digital form takes a lot of effort, and the digital result starts out with very poor quality. Data preprocessing exists to fix this. There are two goals:

Improve the quality of the data, so that mining algorithms can be applied to it.
Modify the data to better fit a specific type of mining technique — reshaping the data for the algorithm, not just cleaning it.

The five tasks of data preprocessing are: data cleaning, data integration, data transformation, data reduction, and data discretization.

Intuition + analogy. Preprocessing is the kitchen before the restaurant opens. Raw vegetables arrive dirty, bruised, and mixed with the wrong boxes; the kitchen cleans them (data cleaning), combines supplies from several farms into one store (data integration), cuts them into the shapes the recipes need (data transformation), trims the volume so the chef is not overwhelmed (data reduction), and sorts them into bins the dishes call for (data discretization). No chef cooks with mud still on the vegetables, and no mining algorithm should run on unprocessed raw data.

In real life, data is almost always incomplete, noisy, and inconsistent. Certain attributes were never included, certain attribute values for particular objects are missing, some values were not captured accurately, or aggregate data was captured when minute-level detail was needed. Before any mining algorithm runs, this dirt has to be cleaned out. The recurring warning: if you pass bad data to a mining algorithm, there is a high chance you get garbage only — the knowledge you mine from bad data is bad.

3.8.2 The Six Data Quality Parameters

Definition. Data quality refers to the overall utility of a data set as a function of its ability to be easily processed and analyzed. It is judged on six parameters: accuracy, completeness, consistency, timeliness, believability, and interpretability.

Accuracy. The physical entity must be captured as accurately as possible. Consider a tsunami detection system: sensors are deployed in the Indian Ocean, they collect data into a server, and a mining algorithm predicts whether a tsunami will occur now or five minutes down the line. Some sensors are bad or not working at all and pour inaccurate data into the server — data you cannot run a mining algorithm on. If the physical entity is not captured accurately into digital form, accuracy fails.

Completeness. The data should have no missing values, and the digital form should represent the physical form as closely as possible. There are two failure directions:

Horizontal missing values: certain objects have some attribute values missing. In the tsunami system, sensors are placed at positions like 0.1, 0.2, 0.3 in the ocean; if a sensor stops working, its field is empty. The data is incomplete because attribute values are missing for some objects.
Vertical missing values: a whole attribute was never captured. Suppose temperature was judged not essential for the tsunami system and was never measured — yet temperature is one of the important characteristics for deciding whether a tsunami is coming. The object was captured, but an essential attribute is missing. Similarly, an oil price prediction system driven only by demand and supply misses events like a war that constrains supply, so predictions go wrong. For gold price prediction, demand and supply matter but so do many other parameters.

Choosing which attributes to capture is the most tedious part of the process — nobody tells you whether to collect 50 attributes, 500, or 2. The decision normally comes from domain knowledge, and if you miss an important attribute, the model built without it will perform poorly.

Consistency. Merged data must agree. If one source represents a date as day-month-year and another as month-day-year, blindly merging the tables creates inconsistency. If height is stored in feet in one table and in meters in another, the merged column is inconsistent until converted to one uniform metric.

Timeliness. The data must be currently relevant. Building Google News means clustering today's news articles so users can browse today's news conveniently — the news from five years or fifty years back is irrelevant. For such applications the data must be as updated as possible.

Believability. How much trust can you have in the data? Sometimes the data itself is not trustworthy, and steps are needed to make it more believable. The standard texts give a real example: a database that once contained errors that were later corrected still loses the trust of users who were harmed by those errors — the data may be accurate now, but nobody believes it.

Interpretability. How well can you interpret the data — the attributes, the objects, the different parts of the data set? If you cannot understand an attribute or an object, interpretability fails. The standard texts' example: a database full of accounting codes that the sales department cannot decode is not interpretable to them, no matter how clean it is.

Worked example — judging one data set on all six parameters. Take the tsunami sensor system:

Parameter	Question it asks	What a failing case looks like
Accuracy	Is the reading the true value?	A dead sensor feeds constant values into the server.
Completeness	Is everything captured?	A sensor stopped working, so its field is empty (horizontal); temperature was never measured (vertical).
Consistency	Do merged sources agree?	Two sensor generations report pressure in different units.
Timeliness	Is the data current?	The server ingests sensor data hours after the event.
Believability	Can we trust it?	The monitoring team learned readings are unreliable and ignores alerts.
Interpretability	Can we read it?	Buoy codes are undocumented; nobody can map a code to a location.

The same system, six different failure modes. Sense-check: a system can pass five parameters and fail on the sixth — accuracy alone does not make data usable.

3.8.3 Data Quality Is Subjective

The bottom line: data has quality if it satisfies the requirements of the intended user. Because the intended user changes, the same data set can be excellent for one person and worthless for another.

Example: an Amazon data set with order number, destination address, destination date, packaging date, and so on. This data set is framed from the delivery point of view. A daily delivery manager wants to know whether the product was delivered or not — questions this data set answers well, so for the delivery manager the data quality is very good. An R&D manager asks different questions — whether this product is good or bad, whether it has certain features — and this same data set cannot answer them, so for the R&D manager the data quality is extremely poor. Same data set, opposite quality verdicts, based purely on the questions being asked.

Intuition. Quality is fitness for use, not a property of the file. A stock market historian finds fifty-year-old price records excellent; a Google News system finds the same records useless. The reference texts make the same point with a customer database: a marketing analyst needing addresses is pleased with a database whose addresses are 80% accurate, while a sales manager, who needs the accurate individual records, finds the same database bad. The data set did not change — the user's needs did.

3.8.4 Bad Data Leads to Bad Mining Results

The takeaway, repeated as a rule of thumb: if you use bad data, you get bad mining results. First we detect problems in the data, then we resolve those problems to improve the data quality, and only then do we mine. The quality problems to detect and resolve are: noise, outliers, missing values, and duplicates.

The order of operations matters. Detection comes first, resolution second, mining last. Skipping detection means the resolution step does not know what it is fixing; skipping resolution means the mining algorithm consumes the problems. Each of the four problems — and how to detect and fix it — is the subject of the next section.

Recap + bridge. Two goals of preprocessing (improve quality, fit the technique) and five tasks (cleaning, integration, transformation, reduction, discretization). Data quality has six parameters — accuracy, completeness, consistency, timeliness, believability, interpretability — and quality is subjective: it depends on the intended user. Bad data in, bad mining out. The next section names the four specific quality problems to detect and resolve.

3.9 Data Quality Problems

3.9.1 Noise

Definition. Noise refers to the modification of the original data — a random error or variance added to the variable. The standard definition matches exactly: noise is a random error or variance in a measured variable.

A sensor in the tsunami detection system malfunctions and every value it passes is skewed by plus two, or it feeds a constant value to the database: that added error is noise. A person's voice distorted while talking on a poor signal is the same idea in another medium — the true signal exists, but a random corruption rides on top of it.

How can we tell that data contains noise? One approach: compute the Fourier transform of the signal. Given a data wave that is essentially two sine waves plus noise, the Fourier transform reveals that it contains only two waves — a blue wave and a green wave — and everything else was noise. We take the frequency of those two waves and remove the rest of the signal; that removal is noise reduction.

Worked example — separating two waves from noise. Suppose the recorded signal is

$s (t) = sin (2 π f_{1} t) + \frac{1}{2} sin (2 π f_{2} t) + noise (t),$

where $f_{1}$ and $f_{2}$ are two fixed frequencies, say $f_{1} = 2$ Hz and $f_{2} = 5$ Hz, and noise is a random jitter added at every sample. A Fourier transform of the signal produces a frequency spectrum — a plot with frequency on the horizontal axis and strength on the vertical axis. The spectrum shows two tall spikes, exactly at $f_{1} = 2$ Hz and $f_{2} = 5$ Hz, and small ragged values everywhere else. The two spikes are the two real waves (the blue wave at 2 Hz, the green wave at 5 Hz); the ragged values are the noise. Removing everything except the two spikes — zeroing the spectrum outside $f_{1}$ and $f_{2}$ — and transforming back recovers the clean signal. Sense-check: the reconstruction has only two frequencies left, which is what "the data was two sine waves plus noise" should look like after reduction.

A critical ordering rule: noise should be removed before detecting outliers. If outliers are detected first, noise points can be mistaken for them. A noisy point and a true outlier can look identical at a glance — both sit far from the bulk — so the cleaning order is fixed: take the noise out first, then look for real anomalies.

3.9.2 Outliers

Definition. An outlier is a legitimate data object whose characteristics are considerably different from most of the data set. Two defining properties:

Outliers are valid points — real, genuine data objects, not errors.
They are considerably different from the rest of the points — their attribute values stand far apart from the majority.

Why are outliers valuable? Consider an intrusion detection system (malware detection). A packet filter checks whether a packet is performing an intrusion into the system. Out of 1000 packets processed in a day, maybe one packet is bad — malicious, an intrusion. That packet is a valid packet; it really happened. But it is an anomaly, an outlier relative to the 999 normal packets. If we treated outliers as noise and deleted them, we would delete exactly the packets the system exists to catch.

Worked example — the one bad packet in a thousand. The intrusion detection system processes 1000 packets today. 999 are normal web traffic; 1 is a malicious packet trying to break in.

View of the bad packet	Consequence
Noise (a mistake to delete)	It is removed in cleaning, and the intrusion is never flagged — the system misses exactly the event it exists to catch.
Outlier (a valid anomaly to keep)	It is preserved as a real, different object; the filter flags it and the intrusion is caught.

The packet really happened and is genuinely different from the other 999 — both properties of an outlier. The reference texts make the same point: unlike noise, outliers may be of interest, and fraud and network intrusion detection are built on finding unusual objects among many normal ones. Sense-check: deleting outliers from this data set deletes the signal, not the error.

Noise can be an outlier and an outlier can be noise — but we should not treat outliers as noise. The two categories are handled differently. The standard texts say it directly: it is important to distinguish noise from outliers, because outliers can be legitimate data objects or values.

Intuition + analogy. In a village of cars, one double-decker bus is an outlier: it is a perfectly genuine vehicle, and it is dramatically different from everything else. Removing it because it "looks wrong" would erase the only interesting thing about the traffic. A dented car that is actually a cardboard prop is noise: it was never a real vehicle. The bus stays for analysis; the prop is removed. The category — not the appearance — decides what to do.

3.9.3 Missing Values

Missing values enter a data set in several ways:

Information was not collected. Certain people refused to give the information, or for certain people the information was not valid, so their rows have gaps.
An attribute itself is missing. The attribute was important for building the system, but nobody captured it while converting the physical entity into the digital form.
Collection-side failures. Equipment was not working, there was inconsistency in how the data was noted down, or the reason for recording a value was not understood at collection time.

The standard texts add two more routes: data that was not considered important at the time of entry, and data that was deleted because it was inconsistent with other recorded data. Whatever the route, missing values have to be dealt with before mining — the methods for that come in the next section.

3.9.4 Duplicate Data

Duplicates arrive mostly through careless merging. Two tables are merged, and many rows are common to both — the same objects appear in both tables. A blind merge repeats those rows, so the merged data set contains redundant tuples. Duplicate data lowers data quality, so duplicates must be identified and removed.

Intuition. Merging tables is like combining two class lists by taping them together: students on both lists get counted twice. The standard texts see the same pattern in daily life — people receive duplicate mailings because they appear in a database several times under slightly different names. The process of finding and merging such rows is called deduplication, and it is genuinely tricky: the two rows for the same person usually disagree on some values (which to keep?), and two different people can share a name (which must not be merged).

3.9.5 Incomplete versus Incorrect Data and Disguised Missing Values

Two different failures are easy to confuse:

Incomplete data means the data was not captured — a missing attribute value or a missing attribute.
Incorrect data means the value is present, but the value is wrong.

The tricky case is a disguised missing value: the real value is missing, but a temporary or fake value was pushed in because the collection method was wrong. Example: many websites ask for your date of birth with a preset default date of 1 January 1980. Most people do not want to share their real date of birth and simply click OK. The row is complete — the field is filled — but the stored date is incorrect, and mining on it produces wrong knowledge. The data was not missing; the data that was fed was wrong. The field looks filled and passes a simple completeness check, which is exactly why disguised missing values are hard to catch.

Worked example — the 1 January 1980 birthday. A site's registration form opens with the date-of-birth field already set to 1 January 1980. Users who do not want to share their real birth date click OK and move on. The database now stores a birth date for every user — no empty cells, no warning flags. But for the users who clicked through, the stored date is not their birthday at all:

Completeness check: passes (every cell is filled).
Accuracy check: fails (the value is fabricated).

The value was never missing — a wrong value was fed in because the collection method made the fake value easy. The standard texts call exactly this scenario disguised missing data: users submit incorrect values for mandatory fields rather than personal information. Sense-check: a completeness report says the column is fine, while any mining on age, cohorts, or demographics is quietly wrong.

3.9.6 How to Detect Incorrect Data

Identifying incorrect data is hard, in large part subjective, and relies heavily on domain knowledge. The practical tool is a set of checks — parsers and scanners that encode what the domain says is possible.

Q: How do we identify incorrect data from correct data?

A: This is genuinely hard, because a disguised missing value looks filled — the value is there, but in real terms it is missing. The answer is to build preprocessing checks that catch these values and then fix them. Most of this is subjective and a lot of domain knowledge goes into it, but concrete rules work: salary can never be negative, so a value like minus 10 is flagged as incorrect; age can be defined within a bracket, say from 0 or 1 up to 150 or 200, so if someone pushes age minus 20 or plus 2000, a parser or scanner finds the out-of-range values and we remove or fix those tuples. We can write many such parsers based on domain knowledge. What we cannot do is remove all noise automatically — that remains a very hard task.

Pitfalls in detecting incorrect data.

Do not trust filled cells. A filled field is not proof the value is real — that is the whole point of disguised missing values.
Domain rules must be encoded, not assumed. "Salary is never negative" and "age is between 0 and 150" are checks someone has to write; no algorithm invents them from the data.
Out-of-range is only the easy part. Values inside the legal range can still be wrong, which is why the standard texts call incorrectness "in large part subjective" — the parsers catch what the domain knowledge can express.

Recap + bridge. Four quality problems: noise (random error added to values — reduce it before outlier detection), outliers (valid points that differ a lot — keep them, especially for intrusion detection), missing values (information that never made it in), and duplicates (rows repeated by careless merging). Incomplete (never captured) differs from incorrect (wrong value), and the disguised missing value — a fake value that looks filled, like the 1 January 1980 birthday — is the hardest of all. The next section shows how to handle the missing-value problem itself.

3.10 Handling Missing Values

3.10.1 Ignore the Tuples That Have Missing Values

Real-world data is dirty, so data cleaning is the first preprocessing task: the raw data has incorrect values, incomplete values (an occupation field missing here and there), noisy values (salary equal to minus 10), and inconsistencies (date in two formats, ratings as 1-2-3 in one table and A-B-C in another, merged blindly). The question is how to repair missing values. There are six standard approaches.

The simplest method: drop the tuples that have missing values from the data set, and build the model on what remains. Sometimes we are forced into this. The problem: if a lot of tuples have missing values, we lose a considerable part of the data set, and a mining model trained on the shrunken set may no longer be representative.

Scope — when ignoring tuples is safe, and when it breaks. Ignoring works when the missing values are few and scattered — a few incomplete rows out of thousands, and the rest still represents the population. It breaks when the missingness is widespread or systematic: if every second row has a gap, the shrunken set is a biased sample of the original, and the model built on it is not representative. The standard texts add a warning in the same spirit: even a partially specified row still contains some information, and when many rows are affected, dropping them can make reliable analysis impossible.

3.10.2 Fill the Missing Values Manually

Go tuple by tuple, attribute by attribute, find each missing value, and fill it in by hand, based on domain knowledge or more information already captured. The problem: this is very tedious, and for large data sets it is sometimes infeasible. That is why automatic fill methods exist.

3.10.3 Fill with a Global Constant

If domain knowledge says the whole table comes from one source, fill every missing cell of an attribute with one global value. Example: a table has a country attribute, and many tuples have the country missing. If the domain expert or the data collection team confirms that the entire data set came from India, set a global constant — country equals India — and replace every missing country value with India.

Global constants also work in the data integration context: suppose three tables are to be merged, one belonging to India, one to the USA, one to China. Fill the missing country values in each table with its own global constant (India, USA, China) before merging, and the merged table carries the relevant information.

Worked example — the global constant in action. Table A has 1000 rows; 80 rows have an empty country column. The collection team confirms every row came from India. The global constant is "India":

Set $country = India$ for the 80 empty cells.
Result: 1000 rows, all with country filled.

Now add the integration case: table B (all rows from the USA) has 40 empty country cells, table C (all rows from China) has 60. Fill each table with its own constant first — India, USA, China — then merge all three tables. The merged table has no country gaps, and every country value is correct because the constant matched the table's origin. Sense-check: filling India into a USA-only table would have introduced errors, which is why the constant is chosen per table, before merging, and only when the team is sure of the table's origin.

3.10.4 Fill with Local Constants

A local constant is a fixed value applied to a group of tuples rather than the whole table. Suppose certain tuples are known to come from a particular state, and the state information is missing for some attributes of those objects. Fill the state for those tuples locally. The same idea works for any grouping — a group of five attributes here, a group of fifty-three there — each group gets its own constant.

Intuition — global vs local constant. The global constant makes one claim about the whole table ("all rows are from India"). The local constant makes a narrower claim about one group ("these rows are from Karnataka"). Narrower claims are safer whenever you have the grouping: a local constant can be right where a global constant would be wrong, and it never throws away the group information the table already carries.

3.10.5 Fill with a Measure of Central Tendency

Fill the missing value of an attribute with a measure of central tendency of that attribute — the mean or the median. If the salary field is missing for certain objects, and the average salary in the country or in the relevant group is some value X, fill the missing salaries with that average.

Worked example — filling with the mean. Five employees, one missing salary:

Employee	Salary (thousands)
E1	30
E2	40
E3	?
E4	50
E5	40

The mean of the four known salaries is

$\frac{30 + 40 + 50 + 40}{4} = \frac{160}{4} = 40$

so the missing cell is filled with 40 (thousand). The median of {30, 40, 40, 50} is also 40, so both central-tendency choices agree here. When they disagree, the choice matters: the mean follows extreme values, while the median resists them. Sense-check: the filled value sits inside the observed range and matches the general level of the column, which is what "central tendency" promises — though it also flattens the row's individuality.

The standard texts confirm the method: use a measure of central tendency for the attribute, such as the mean or median, to fill in the missing value, and for better results use the mean or median of the samples belonging to the same class as the tuple with the missing value.

3.10.6 Fill with Probabilistic Methods

A probabilistic fill uses other attributes to estimate the missing one. The basic form uses conditional probability: given the known attributes of an object, what value is most likely for the missing attribute?

Formalize — the conditional statement. Simple example with an age and salary table: for some objects the salary field is missing. The rule — if a person's age is less than 20, fill the salary as 0 — is a conditional statement:

$P (Salary = 0 ∣ Age < 20)$

Read as: the probability that the salary is 0, given that the age is below 20. The bar $∣$ means "given". The fill uses the most likely value: because the conditional probability of salary 0 is high within the age-under-20 group, 0 is the value chosen for those tuples. The reasoning: in India, a person under 20 is normally not earning, so assuming a salary of zero might be wrong but is most likely not wrong. A group of attributes can be used together this way to fill missing attribute values. More advanced forms of conditional probability exist; this is the basic example of the idea.

The standard texts describe the same idea as filling with the most probable value: determine it from the data using other attributes, for example with a decision tree built to predict the missing attribute. The simple conditional-rule version in this session is the same principle in its most basic form — use what is known about the object to guess the missing piece.

Scope and assumptions of probabilistic filling. The conditional fill assumes the other attributes are reliable and informative about the missing one: age predicts salary, but it would not predict eye color. It also assumes the underlying probability statement holds — if the group's reality is different (say, most under-20s do earn), the "most likely" value is wrong. The professor's caveat carries the point: the fill "might be wrong but is most likely not wrong" — it is an estimate, not a fact.

3.10.7 Student Questions and Answers

Q: If we fill a missing value with a default value, and the default is high — wouldn't the high value make the point look genuine?

A: A high value does not make the point genuine. Consider measuring the temperature of Goa in January: today 20, yesterday 18, tomorrow 22 — the January range is roughly 18 to 25. Suppose on one day the temperature reads 55. It is the highest value in the data, but a sudden very hot day in January is not possible, so this is most likely noise, not an outlier, and it should be identified as noise and removed. Whether a value is noise or an outlier is not decided by how high it is.

Q: When we fill missing data with a default, how do we set the default?

A: There are two ways to look at it. First, we fill missing values so that the mining algorithm can run: certain mathematical operations are impossible on an empty field, so we put a substitute in. Second, we want the substitute to be the closest value the attribute could reasonably have — filling a high value that does not represent the real-world scenario may skew the mining in a wrong direction.

Q: In the country example, we fill country as India because most of the data comes from India — but not necessarily all of it?

A: Correct, and that is why most of these decisions are based on domain knowledge. If the domain expert or the data collection team confirms the entire data set came from India, the global constant India is safe. When that certainty is not available, we fall back on the per-table global constants before integration: fill India, USA, and China in their own tables first, then merge.

Q: Can we drop the attribute or drop the tuples that have missing values?

A: Yes. Dropping the attribute entirely is one way of handling a missing attribute, and dropping the tuples is the ignore-the-tuple method — the simplest tool, used when the loss is acceptable or when no reasonable fill exists.

Exam note. The six methods for handling missing values: ignore the tuples, fill manually, fill with a global constant, fill with local constants, fill with a measure of central tendency (mean or median), and fill with probabilistic methods (conditional probability, most probable value). The two examinable judgment calls: when a global constant is safe (domain knowledge confirms the source), and what makes a fill substitute sensible (close to the realistic value, not arbitrarily high). The conditional form to be able to read and write is $P (Salary = 0 ∣ Age < 20)$ — probability of the missing value given the known attribute.

3.11 Handling Noisy Data

3.11.1 Where Noise Comes From

Noise is random error or variance added to the variable. It enters the data through a handful of common channels: a faulty instrument, a data entry problem, a transmission problem, a technology limitation, or an inconsistency in the naming convention used during collection. Each channel corrupts the recorded values relative to the true ones.

Worked example — one channel, one corruption. A faulty thermometer in a weather station reads 2 degrees too warm on every sample: the true 20° is recorded as 22°, 21° as 23°, 25° as 27°. The recorded series has the true signal underneath, shifted by a fixed error — that fixed shift is noise from a faulty instrument. The same pattern appears as mistyped digits (data entry), garbled packets (transmission), coarse sensors (technology limitation), and values logged under different names for the same thing (naming conventions). Sense-check: in every channel the recorded value differs from the true value in a way the data set itself does not announce.

3.11.2 Data Smoothing

The standard way to reduce the impact of noise is data smoothing — the methods for smoothing a noisy variable (binning, clustering, regression-style fitting, and related techniques) are the tools that dampen the random error so the underlying signal survives. The reminder from the noise discussion applies here: noise should be removed before outlier detection, because noise points that survive will otherwise masquerade as outliers.

Purpose — what smoothing is for. Smoothing replaces noisy values with values closer to their local neighborhood, on the assumption that the true signal varies gently while noise jumps around. The standard texts list three families: binning (smoothing within small groups of sorted values), clustering (values that fall together in a cluster are smoothed toward the cluster, and points that belong to no cluster may be noise), and regression (fit a smooth function to the data and replace values with the fitted line).

Worked example — smoothing by bin means. Nine price values, sorted smallest to largest:

4, 8, 15, 21, 21, 24, 25, 28, 34

Split into three bins of depth 3 (three values per bin):

Bin 1: {4, 8, 15} — mean $\frac{4 + 8 + 15}{3} = \frac{27}{3} = 9$
Bin 2: {21, 21, 24} — mean $\frac{21 + 21 + 24}{3} = \frac{66}{3} = 22$
Bin 3: {25, 28, 34} — mean $\frac{25 + 28 + 34}{3} = \frac{87}{3} = 29$

Replace each value by its bin's mean: the smoothed series is 9, 9, 9, 22, 22, 22, 29, 29, 29. The jumpy 4 and the high 34 are pulled toward their neighbors — that is the noise reduction. (Variations smooth by bin medians instead — 8, 21, 28 here — or by bin boundaries, replacing each value with the nearest of its bin's min or max.) Sense-check: the smoothed series has the same general level as the original but much less wobble, which is exactly the goal; the cost is that individual real variations inside a bin are flattened too.

Binning is local smoothing: each value is adjusted using only its own neighborhood. The standard texts add the practical note that the same binning machinery reappears later as a discretization tool — one technique, two preprocessing jobs.

3.11.3 Noise versus Outlier: Why the Highest Value Does Not Decide

A high value alone never tells you whether a point is noise or an outlier — the distinction depends on whether the point is genuine.

Noise is a corruption of a value that should have been different: the sensor was skewed, the entry was mistyped, the transmission garbled it. Removing it restores the truth. The Goa January temperature of 55, in a month where the range is 18 to 25, is the highest value but is not genuine — it is noise and should be removed.
Outliers are genuine, valid data points that are considerably different from the rest. In the intrusion detection system, thousands of packets flow through the network; one or two are malicious. Those packets are valid — they really occurred — but they differ considerably from the rest of the data, so they are outliers, not noise.

In an outlier detection application, a very high point is highly likely to be an outlier — and you want to detect it — but it might also be noise. Different applications treat the same extreme value in different ways, which is why the category decision is part of the preprocessing design, not an automatic consequence of the value.

Worked example — the same value, two verdicts. Temperature 55 in a January range of 18 to 25:

Application	What 55 means	Verdict
Goa climate data (January)	A sudden very hot January day is not possible; the sensor misread.	Noise — remove.
Intrusion detection (network packets)	An extreme packet really occurred and looks nothing like normal traffic.	Outlier — flag and keep.

The value did not change; the application did. This is why the standard treatment distinguishes the two: outliers can be legitimate, while noise is a corruption to be removed — and why the design decision (which category, which handling) belongs to the preprocessing step. Sense-check: "55 is high" was never the argument; "55 is genuine" is the question.

3.11.4 Real-world: Data Preprocessing Libraries

Real-world: the preprocessing steps described here are standard and implemented in mainstream machine learning libraries. In scikit-learn (sklearn), the sklearn.preprocessing module provides functions for cleaning and transforming data — searching "sklearn data preprocessing" surfaces the documentation for these standard techniques, including noise and missing-value handling. These libraries encode the same steps: detecting problems in the data and resolving them before modeling.

Recap + bridge. Noise enters through faulty instruments, bad entry, transmission, technology limits, and naming inconsistencies; data smoothing — binning, clustering, regression fitting — dampens it; and noise must be removed before outliers are detected, because noise points can masquerade as outliers. What decides noise versus outlier is genuineness, not size. The session closes with the shape of the whole preprocessing story: detect the problems, resolve them, then mine.

Exam Guidance Summary

This session contained no direct exam-specific announcements — no mark distribution, no question-type preview, and no list of what is excluded from the exam. The study guidance it did offer: reference books are provided at the end of every module presentation, and students are expected to read beyond the presentation.

For the material itself, the distinctions most worth practicing are the ones the session drilled repeatedly:

The four attribute types and their allowed operations — nominal (2: $=$ , $\neq =$ ), ordinal (4: adds $<$ , $>$ ), interval (6: adds $+$ , $-$ ), ratio (8: adds $\times$ , $\div$ ). The 2–4–6–8 ladder is the most examinable single fact of the session.
Discrete versus continuous attributes — binary and ternary as special cases of discrete; real-valued as continuous; the two axes are independent.
The three record forms — data matrix, document data, transaction data — plus graph form and ordered form as the other two data-set representations.
The six data quality parameters — accuracy, completeness, consistency, timeliness, believability, interpretability — and the point that quality is subjective ("satisfies the requirements of the intended user").
Noise versus outliers — both are unusual values, but noise is a corruption to remove and an outlier is a valid point to keep; the category is decided by genuineness, not by size.
Incomplete versus incorrect data — missing (never captured) versus wrong (captured badly), including the disguised missing value that looks filled.
The six methods for handling missing values — ignore tuples, fill manually, global constant, local constants, central tendency (mean/median), probabilistic (conditional probability).
The two judgment questions the content supports: classify an attribute and name its legal operations; and say why a global constant is or is not safe for a given table.

Key Industry Applications

Web search. Page ranking algorithms operate on the web-as-graph representation to order pages for a keyword search; this is the mechanism behind search engines such as Google. Every query's result ordering is a graph mining operation on link structure.
News aggregation. Google News clusters today's news articles for convenient browsing — a timeliness-driven application where old news is irrelevant and the data must be as current as possible.
Intrusion detection. Packet filters flag the one-in-a-thousand malicious packet as an anomaly; outliers here are the signal, not the noise. Deleting them as "errors" would delete exactly the events the system exists to catch.
Tsunami detection. Sensor networks in the ocean feed mining algorithms that predict tsunamis minutes ahead; bad sensors and missing attributes degrade accuracy and completeness, two of the six quality parameters.
Plagiarism detection. Research-paper repositories represented as document-frequency matrices detect copied submissions by matching a newcomer's word-frequency pattern against stored documents.
Market analytics. Transaction logs from shopping malls, represented as transaction data, support market-basket style analysis — the tabular foundation for "customers who buy X also buy Y" patterns.
Retail logistics. Order, address, packaging, and delivery data answer delivery questions well but not product R&D questions — the same data set scores excellent for a delivery manager and poor for an R&D manager, because data quality is application-dependent.
Economic forecasting. Oil price and gold price prediction depend on capturing factors beyond demand and supply (for example, supply shocks from conflict); missing those attributes degrades predictions — a vertical-missing-value failure.
Genomics and climate science. Genome sequences are ordered data mined position by position for gene similarity, and land-ocean temperature records are spatio-temporal data read along both the distance and time axes.
Machine learning libraries. scikit-learn's data preprocessing module implements these cleaning and transformation steps in code — detecting problems in the data and resolving them before modeling.

DM Lecture 3 notes · Data and Data Preprocessing

Data Mining· postgraduate· 2026-08-05

Sections Breakdown

1What Is Data

Data mining works on digital representations of physical entities; data is a collection of objects (rows) and attributes (columns), where objects are also called records, points, samples, entities, or instances, and the set of values an attribute can hold shapes what we can do with it.

2The Four Attribute Types

Attributes come in four types — nominal (labels, only same/different), ordinal (order, no equal gaps), interval (equal gaps, arbitrary zero), and ratio (true zero, meaningful ratios) — forming a cumulative ladder from weakest to strongest.

3Mathematical Operations on Attributes

Four families of operations (distinctness, order, addition, multiplication) give eight operations total; nominal allows 2, ordinal 4, interval 6, ratio 8, and attributes should be treated at their highest possible level to maximize information extraction.

4Discrete and Continuous Attributes

A second, independent categorization: discrete attributes have a fixed countable set of values (binary and ternary are special cases), continuous attributes take real-number values with infinitely many possibilities; the two axes (four types vs discrete/continuous) combine freely.

5Record Data

Record data is the tabular form — fixed number of attributes per record — in three varieties: data matrix (rows objects, columns attributes, numeric), document data (documents as rows, word frequencies as columns, for plagiarism detection), and transaction data (one row per purchase basket, for market-basket mining).

6Graph Data

Graph data represents objects as nodes and relationships as links; the web (pages as nodes, hyperlinks as links) supports page-ranking algorithms for search, and chemical compounds (atoms as nodes, bonds as links) support molecular structure and similarity mining.

7Ordered Data

Ordered data is the form where sequence position itself carries meaning: genome sequences (ordered nucleotide bases), transaction sequences (ordered purchase times), and spatio-temporal data (values carrying both a distance axis and a time axis, as in the land-ocean monthly temperature chart).

8Data Preprocessing and Data Quality

Preprocessing has two goals (improve data quality, modify data to fit a mining technique) and five tasks (cleaning, integration, transformation, reduction, discretization); data quality is judged on six parameters — accuracy, completeness, consistency, timeliness, believability, interpretability — and is subjective: a data set satisfies whoever's requirements it meets.

9Data Quality Problems

The four quality problems are noise (random error added to values — reduced via the Fourier transform and removed before outlier detection), outliers (valid points considerably different from the rest — kept, e.g., for intrusion detection), missing values, and duplicates from careless merging; incomplete data differs from incorrect data, and the disguised missing value (fake value that looks filled) is the hardest to catch.

10Handling Missing Values

Six standard approaches to missing values: ignore the tuples, fill manually, fill with a global constant, fill with local constants, fill with a measure of central tendency (mean or median), and fill with probabilistic methods (conditional probability / most probable value); most decisions are based on domain knowledge.

11Handling Noisy Data

Noise enters through faulty instruments, data entry, transmission, technology limits, and naming inconsistencies; data smoothing (binning, clustering, regression) dampens it, noise must be removed before outlier detection, and a high value alone never decides noise versus outlier — genuineness does.

12Exam Guidance Summary

No mark distribution or question-type preview was announced; reference books are provided at the end of every module presentation; the practice list is the four attribute types and the 2–4–6–8 operation ladder, discrete vs continuous, the three record forms, the six quality parameters, noise vs outliers, incomplete vs incorrect data, and the six missing-value methods.

13Key Industry Applications

Consolidated real-world map of the session: web search (page ranking / Google), news aggregation (Google News), intrusion detection (outlier packets), tsunami detection (sensor accuracy and completeness), plagiarism detection (document-frequency matrices), market analytics (transaction data), retail logistics (delivery vs R&D quality), economic forecasting (missing supply-shock attributes), genomics and climate science (ordered data), and machine learning libraries (sklearn.preprocessing).

Postgraduate students in Data Mining

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

What Is Data

Must-know: Data = objects (rows) + attributes (columns); object synonyms: record, point, sample, entity, instance; attribute synonyms: variable, field, characteristic, dimension, feature; the passbook example shows a physical entity digitized into a fixed-structure table.

⚠️ Top pitfall: Treating attribute and attribute value as the same thing — the attribute is the property, the value varies from object to object; digit-shaped labels (IDs) are symbols, not numbers.

Self-check: A table has 5 rows and 3 columns. How many objects and how many attributes does it contain?

Connects to: 3.2

The Four Attribute Types

Must-know: Nominal: values are symbols/identifiers, only = and != (zip codes, IDs, eye color). Ordinal: order meaningful, gaps not (grades, hardness, good/better/best). Interval: equal fixed gaps, arbitrary zero (calendar dates, Celsius). Ratio: true zero, ratios meaningful (distance, age, counts, Kelvin). Cumulative ladder: each higher type keeps lower operations.

$\frac{2000 km}{1000 km} = 2$

⚠️ Top pitfall: Treating digit-shaped labels (zip codes, IDs) as numbers; saying Celsius 'twice as hot'; assuming grade gaps are equal.

Self-check: Why is temperature interval on Celsius but ratio on Kelvin?

Connects to: 3.3, 3.4

Mathematical Operations on Attributes

Must-know: Eight operations in four families: distinctness (=, !=), order (<, >), addition (+, -), multiplication (x, /). Nominal 2, ordinal 4, interval 6, ratio 8 — cumulative. Rule: treat every attribute at its highest possible level; downgrading (distance as nominal) cuts operations and blocks information extraction; upgrading a nominal attribute (adding names) is not possible.

$nominal: 2 = {=, \neq =} \to ordinal: 4 \to interval: 6 \to ratio: 8$

⚠️ Top pitfall: Downgrading a ratio attribute to nominal cuts eight operations down to two; assuming the ladder works in reverse.

Self-check: How many operations does an interval attribute support, and which ones are they?

Connects to: 3.2, 3.4

Discrete and Continuous Attributes

Must-know: Discrete = fixed countable value set (cheat/refund binary, marital status ternary); continuous = real-number values, infinite (temperature, height, income). The two categorizations are independent: counts are discrete and ratio; nominal/ordinal usually discrete, interval/ratio usually continuous; a continuous binary attribute is hard to imagine.

⚠️ Top pitfall: Assuming discrete/continuous coincides with the four types; thinking a continuous attribute can be listed exhaustively.

Self-check: Is the number of children in a family discrete or continuous, and which of the four types is it?

Connects to: 3.2, 3.5

Record Data

Must-know: Three record forms: data matrix (rows = objects, columns = attributes, numeric — supports matrix operations), document data (documents as rows, unique words as columns, cells = word frequencies — the document-term matrix, usually sparse), transaction data (one row per transaction/basket, items as attributes — market basket data). Plagiarism detection: newcomer's frequency pattern matched against stored rows.

⚠️ Top pitfall: Forgetting that document-term matrices are mostly zeros (sparse) or that transaction rows must share a fixed structure.

Self-check: In the plagiarism example, what sits in the cells of the document-frequency table?

Connects to: 3.1, 3.6

Graph Data

Must-know: Graph = nodes + links; direction and weight live on links. Web as graph: pages nodes, hyperlinks links; page ranking orders pages for a keyword search using link structure — the mechanism behind Google. Benzene C6H6: 12 atoms as nodes, 12 bonds as links; graph mining answers structure and similarity questions (substructure mining, e.g., linked to melting point).

⚠️ Top pitfall: Treating rank as keyword match; ignoring link direction; counting links without importance.

Self-check: In the web-as-graph picture, what are the nodes and what are the links?

Connects to: 3.5, 3.7

Ordered Data

Must-know: Ordered data: position carries the information. Genomes: strings over A, T, G, C where order defines the molecule. Transaction sequences: purchase order (t1 < t2 < t5) is part of the record; patterns like 'DVD players lead to DVD purchases' only exist with time ordering. Spatio-temporal: land-ocean monthly temperature chart read spatially (Delhi hotter than Beijing — latitude) and temporally (Beijing cold in winter, hot in May/June).

⚠️ Top pitfall: Shuffling ordered values as if order did not matter; reading spatio-temporal data along only one axis.

Self-check: Give the two readings of the land-ocean monthly temperature chart.

Connects to: 3.5, 3.6

Data Preprocessing and Data Quality

Must-know: Two goals: improve quality, fit the technique. Five tasks: cleaning, integration, transformation, reduction, discretization. Six quality parameters: accuracy, completeness (horizontal vs vertical missing values), consistency, timeliness, believability, interpretability. Quality is subjective: the Amazon data set is excellent for a delivery manager, poor for an R&D manager. Bad data in, bad mining out; detect problems first, then resolve, then mine.

⚠️ Top pitfall: Judging quality without a user; treating a filled table as accurate; mining before cleaning.

Self-check: Name the six data quality parameters and give the two failure directions of completeness.

Connects to: 3.9, 3.10

Data Quality Problems

Must-know: Noise = random error/variance added to the variable; Fourier transform reveals the true waves (two spikes at f1, f2) and the rest is noise — remove it. Ordering rule: remove noise before detecting outliers. Outlier = valid point, considerably different (1000 packets, 1 malicious — keep it). Missing values enter via non-collection, missing attributes, collection failures. Duplicates come from careless merging. Incomplete (not captured) vs incorrect (wrong value); disguised missing value: 1 January 1980 default birthday — filled but fake. Detection: domain-knowledge parsers (salary never negative, age in a bracket); cannot remove all noise automatically.

$s (t) = sin (2 π f_{1} t) + \frac{1}{2} sin (2 π f_{2} t) + noise (t)$

⚠️ Top pitfall: Deleting outliers as if they were noise (deletes the intrusion signal); trusting filled cells (disguised missing values); detecting outliers before removing noise.

Self-check: Why is the 1 January 1980 birthday a disguised missing value rather than just incorrect data?

Connects to: 3.8, 3.10, 3.11

Handling Missing Values

Must-know: Six methods: ignore tuples (loses data, may lose representativeness), fill manually (tedious), global constant (safe only when domain knowledge confirms the table's source; per-table constants before merging), local constants (per group), central tendency (mean or median), probabilistic (conditional probability — most probable value). Reading and writing the conditional form P(Salary = 0 | Age < 20) is examinable. A high default does not make a point genuine (Goa January 55 is noise, not an outlier); the substitute should be the closest reasonable value so mining is not skewed.

$P (Salary = 0 ∣ Age < 20)$

⚠️ Top pitfall: Using a global constant without confirming the table's origin; filling with arbitrarily high values that skew mining; assuming a filled default value is genuine.

Self-check: Why is the global constant India safe in the country example, and when does it stop being safe?

Connects to: 3.9, 3.11

Handling Noisy Data

Must-know: Noise channels: faulty instrument, data entry, transmission, technology limitation, naming convention. Smoothing: binning (bin means/medians/boundaries — worked: 4,8,15,21,21,24,25,28,34 → bins of 3 → means 9, 22, 29), clustering, regression. Remove noise before detecting outliers. Noise vs outlier: noise is a corruption to remove (Goa January 55, range 18–25); an outlier is genuine and different (malicious packet among 999 normal ones); the category is a preprocessing design decision, not a consequence of the value. sklearn.preprocessing encodes these steps.

⚠️ Top pitfall: Detecting outliers before removing noise; deciding noise vs outlier by value size alone; smoothing away genuine outliers.

Self-check: In the binning example, what are the three bin means for the sorted prices 4, 8, 15, 21, 21, 24, 25, 28, 34?

Connects to: 3.9, 3.10

Exam Guidance Summary

Must-know: Study guidance: reference books at the end of every module presentation; read beyond the presentation. Practice the distinctions drilled repeatedly: four attribute types with allowed operations (nominal 2, ordinal 4, interval 6, ratio 8); discrete vs continuous; the three record forms; six data quality parameters; noise vs outliers; incomplete vs incorrect data; six methods for handling missing values; when a global constant is safe.

Self-check: Which operation families are legal for an interval attribute?

Connects to: 3.2, 3.3, 3.4, 3.5, 3.8, 3.9, 3.10

Key Industry Applications

Must-know: Named applications per concept: page ranking orders search results via web-graph links (Google); Google News is timeliness-driven; intrusion detection depends on keeping outliers (1 in 1000 packets); tsunami detection ties bad sensors to accuracy and missing attributes to completeness; plagiarism detection matches document-frequency patterns; mall transaction logs become market-basket data; the Amazon data set is quality-good for delivery questions and quality-poor for R&D questions; oil and gold price prediction need attributes beyond demand and supply; genomes and temperature charts are ordered data; sklearn.preprocessing encodes the standard cleaning steps.

Self-check: Which application depends on treating outliers as signal rather than noise?

Connects to: 3.6, 3.7, 3.8, 3.9, 3.11

Previous LectureLecture 02 · Data Mining Fundamentals

Next LectureLecture 04 · Data Preprocessing: Noise, Integration, Transformation, and Reduction

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

3.1 What Is Data

3.1.1 From Physical Entity to Digital Form

3.1.2 Objects and Attributes

3.1.3 Attribute Values

3.2 The Four Attribute Types

3.2.1 Nominal Attributes

3.2.2 Ordinal Attributes

3.2.3 Interval Attributes

3.2.4 Ratio Attributes

3.3 Mathematical Operations on Attributes

3.3.1 The Eight Operations

3.3.2 Operations Allowed for Each Attribute Type

3.3.3 Treat Attributes at Their Highest Level

3.3.4 Student Questions and Answers

3.4 Discrete and Continuous Attributes

3.4.1 Discrete Attributes

3.4.2 Continuous Attributes

3.4.3 Student Questions and Answers

3.5 Record Data

3.5.1 Data Matrix

3.5.2 Document Data

3.5.3 Transaction Data

3.6 Graph Data

3.6.1 The World Wide Web as a Graph

3.6.2 Page Ranking on Graph Data

3.6.3 Chemical Data as Graphs

3.7 Ordered Data

3.7.1 Sequences: Genomes and Transactions

3.7.2 Spatio-Temporal Data

3.8 Data Preprocessing and Data Quality

3.8.1 Why Preprocess Data

3.8.2 The Six Data Quality Parameters

3.8.3 Data Quality Is Subjective

3.8.4 Bad Data Leads to Bad Mining Results

3.9 Data Quality Problems

3.9.1 Noise

3.9.2 Outliers

3.9.3 Missing Values

3.9.4 Duplicate Data

3.9.5 Incomplete versus Incorrect Data and Disguised Missing Values

3.9.6 How to Detect Incorrect Data

3.10 Handling Missing Values

3.10.1 Ignore the Tuples That Have Missing Values

3.10.2 Fill the Missing Values Manually

3.10.3 Fill with a Global Constant

3.10.4 Fill with Local Constants

3.10.5 Fill with a Measure of Central Tendency

3.10.6 Fill with Probabilistic Methods

3.10.7 Student Questions and Answers

3.11 Handling Noisy Data

3.11.1 Where Noise Comes From

3.11.2 Data Smoothing

3.11.3 Noise versus Outlier: Why the Highest Value Does Not Decide

3.11.4 Real-world: Data Preprocessing Libraries

Exam Guidance Summary

Key Industry Applications

DM Lecture 3 notes · Data and Data Preprocessing

Sections Breakdown

Exam Revision Notes

What Is Data

The Four Attribute Types

Mathematical Operations on Attributes

Discrete and Continuous Attributes

Record Data

Graph Data

Ordered Data

Data Preprocessing and Data Quality

Data Quality Problems

Handling Missing Values

Handling Noisy Data

Exam Guidance Summary

Key Industry Applications

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat