—

Data Management for Machine Learning

Data Ingestion, Validation, and Data Quality

Published: 2026-08-07

Level: postgraduate

Audience: Postgraduate students in Data Management for Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures in the course. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Data quality and the data doctor — covered in Lecture 1 (Foundations of Data and Data Representation)
Data models and NoSQL stores — covered in Lecture 1 (Foundations of Data and Data Representation)
ETL versus ELT — covered in Lectures 2, 3, 5, and 6 (Query Paradigms; Data Pre-Processing; The Modern Data Stack; Data Pipelines)
Data governance — covered in Lecture 2 (Query Paradigms, Storage Architectures, and Data Pipelines)
Star and snowflake schemas — covered in Lectures 2 and 3 (Query Paradigms; Data Pre-Processing, Data Architecture, and Warehouse Schemas)
Data cleaning and handling missing data — covered in Lecture 3 (Data Pre-Processing, Data Architecture, and Warehouse Schemas)
Message queues, SOA, REST, and RPC — covered in Lecture 4 (Data Pipelines, Big Data Systems, and Feature Engineering)
Five-point summary and box plots — covered in Lecture 5 (The Modern Data Stack and Data Pipelines)
Data ingestion and integration, CDC — covered in Lectures 5 and 7 (The Modern Data Stack; The Machine Learning Lifecycle)
Outliers — good or bad — covered in Lecture 6 (Data Pipelines, Outliers, and the Machine Learning Lifecycle)
Data drift and out-of-order pipelines — covered in Lectures 6 and 7 (Data Pipelines, Outliers, and the Machine Learning Lifecycle; The Machine Learning Lifecycle)
Data leakage — covered in Lecture 7 (The Machine Learning Lifecycle)
First-, second-, and third-party data — covered in Lecture 7 (The Machine Learning Lifecycle)
APIs, webhooks, gRPC, and data systems — covered in Lecture 7 (The Machine Learning Lifecycle)
The model drift feedback loop — covered in Lecture 7 (The Machine Learning Lifecycle)

8.1 Midterm Examination Blueprint and Preparation

8.1.1 Question Format: Case Studies and Scenarios

The midterm questions are going to be case studies and scenarios. You will be given a scenario, and from that scenario you must answer. The repeated advice for this course is that computer science is common sense: you already know the terms and concepts, and the exam tests whether you know how to apply that knowledge toward a question, toward a case study. A question may hide a concept such as data leakage, ETL, warehouse, model, ingestion, or quality, and you need to relate the concepts to the questions. The expected outcome is simple: your knowledge mapping to the case study is what is expected.

To help you prepare, some sample case study questions and quiz questions are shared, so you know how to prepare for the midterm. Some students already know the concepts and some are doing it practically, but when it comes to academic examination, answers are expected in a certain format, with answer keys in a certain format. If you do it right, you get a good mark. Do not expect the same pattern to appear — the shared questions give you an idea of the question style, not the exact paper. The exam is your application of your concepts.

Hook — why the format matters. How would you answer a question that never names the concept it is testing? A case study describes a situation — "a retail chain's monthly sales file is missing months and contains negative values" — and expects you to recognize which data management idea applies, then use it. The midterm is deliberately written this way: the concept stays hidden inside the story.

How to read a case study. The skill is a two-step mapping:

Spot the trigger words. Every concept has tell-tale words: leakage (future or target information), ETL (extract, transform, load), warehouse (central analytics store), model (a trained predictor), ingestion (data entering the pipeline), quality (dimensions like accuracy or completeness).
Map the concept to the question. Ask: which unit does this scenario belong to, which check or stage applies here, and what would a data engineer do?

A table makes this mapping explicit when you answer — and the examiners reward a well-presented answer.

Scenario signal	Concept it hides	Typical question
Data arrives in wrong shape, wrong dates, negative values	Ingestion validation	What checks do you apply before and after?
Test data or future data reaches training	Data leakage	Why is the model over-optimistic?
Data is transformed before or after loading	ETL versus ELT	Which model fits this scale?
Reports drawn from a central analytics store	Warehouse (star or snowflake schema)	Where does the report data come from?
Raw data moving through stages toward modeling	Data lifecycle / pipeline	Where does quality get lost?

Exam note: expect case-study style questions where a scenario is given and you must identify which data management concepts apply — ingestion, validation, quality, ETL versus ELT, warehouse, data leakage, and the data lifecycle. The sample papers show the style of the question, not the exact paper.

8.1.2 Syllabus Scope for the Midterm

When asked about the syllabus for the midterms, the answer was: everything from unit 1 to unit 8, up to whatever is covered in today's material, which is contact session 8. Most of the focus should go on units 6, 7, and 8 — the data management lifecycle, the warehouse, and big data. Two contact sessions were clarified: session 7 concludes first, then session 8 is covered, and if time runs short an extra session may be conducted later. The material will be shared regardless.

This time the questions lean more toward data ingestion: what you do before, what you do after, what kind of validation you do, and what type of quality checks. The quality dimensions were already discussed, and one data quality exercise was already done in class. So predominantly everything from unit 1 to unit 8, with more focus on the lifecycle of the data stack — how data comes, how data goes, whatever it goes through, the stages of the data management lifecycle, the pipeline — and then mostly on validation and quality.

Q: What is the syllabus for the midterm? How much is covered?

A: The midterm covers everything from unit 1 to unit 8, up to the end of today's material. The focus should be on units 6, 7, and 8 — the data management lifecycle, the data stack, how data comes in and how it goes out, the stages of the pipeline, and especially data ingestion, validation, and quality. Units 6 and 7 are the most important.

Q: Are we covering unit 8 today, or just concluding unit 7?

A: We will conclude contact session 7 first and then cover contact session 8 as much as possible today. The material will be shared, and if needed, an extra session can be conducted later.

8.1.3 How to Present Answers

There is strong, repeated advice on how to write the exam. Do not just use the automated system and type your answers; write as much as possible by hand, the way you normally write in undergraduate exams. Bold and underline key terms, use bullet points, and if something requires it, give examples and draw diagrams to make the answer presentable. Visualize nicely, because when the paper is corrected, a well-presented answer earns good marks. A table makes answers easier to grade.

Do not use ChatGPT or any AI tools while writing the exam. You may be smart, but the examiners know how to catch such answers: if you know how to use AI tools, they know how to use them too. What is actually being tested is your knowledge mapping to the case study.

Pitfall — the AI trap. Using an AI assistant during the exam turns your submission into a copy of someone else's reasoning. The examiners deliberately design case-study answers that cannot be pattern-matched from a chatbot — and the paper is graded on your mapping of concepts to the scenario. A generated answer reads as polished but generic; it fails exactly where the marks live.

Pitfall — the "just type it" trap. Typing plain paragraphs into the answer box loses marks even when the content is right, because the examiners grade presentation. The same facts written as bullets with underlined key terms score better.

Exam note: write by hand, bold and underline key terms, use bullets, give examples, draw diagrams and tables. Never copy-paste; relate the concepts and write your own interpretation. The answer is scored on how well your knowledge maps to the case study, not on how many lines you fill.

8.1.4 Final Advice Before the Exam

The final advice is only three items: read the question twice; understand the concepts — especially the screenshot notes shared in the WhatsApp group, the different stages of the data lifecycle, the different data quality issues, ETL versus ELT, big data characteristics, and pre-ingestion versus post-ingestion, what you do before ingestion and after; then relate, write, and do not copy paste. Wherever possible, give examples, your interpretation, put some tables, and put some diagrams to score a good mark. The course will continue after the midterm with more details of data profiling, but what has been covered up to data profiling is enough preparation for the midterm.

The three-step exam routine. Compress the advice into a repeatable procedure:

Read the question twice. The first read tells you the story; the second read tells you what it is asking. Case studies hide the concept, so the second pass is where you underline trigger words.
Understand the concept behind it. Recall the relevant stage of the data lifecycle, the quality dimension, or the check type — pre-ingestion (presence, range, format, data type) versus post-ingestion (consistency, business rules, statistics).
Relate, then write. Map the concept to the scenario in your own words, add a small example or a table, and only then put pen to paper. Do not copy-paste any prepared text.

Pitfall — the single-read trap. Students who read the scenario once usually answer a question that was never asked: they describe the technology in general instead of applying it to the case. Reading twice is not a ritual; it is the step that finds the hidden concept.

Pitfall — the "everything is examinable" panic. The syllabus is units 1–8, but the weight is deliberately lopsided: lifecycle (units 6–7), ingestion, and validation dominate. Deep preparation on those pays better than skimming everything equally.

Exam note: read the question twice. Understand the stages, data quality issues, ETL versus ELT, big data characteristics, and pre- versus post-ingestion validation. Relate concepts to the case, write your own words, and add examples, tables, and diagrams. Everything up to data profiling is enough preparation — the deeper profiling work comes after the midterm.

8.2 Data Sources and Data Collection

8.2.1 What a Data Source Is

This section continues from where the previous contact session ended: data collection and data ingestion. The starting question is simple — what type of data comes in? Is it raw data, structured data? When you work with any data as a data engineer, you need to know: what is the source, where has it come from, who is creating it, who is the creator, what is the location, and what is the data format. This is exactly where validation comes into the picture.

Hook — you cannot manage data you cannot describe. Before a single check runs, you need a five-point answer for any dataset: source (which system produced it), creator (which team or device), origin (where in the organization or outside it), location (which server, store, or cloud region holds it), and format (CSV, JSON, Parquet, audio, video). If you cannot name all five, you cannot design its validation either.

Real-world: most systems in reality do not follow the standard. That is why there are so many APIs, pipelines, and validation checks in practice. When data is created, some transformation happens — for example, taking analog data as it is and converting to digital data, speech to text, text to speech. NLP, natural language processing, is the tool applied there; there are many tools available that can help with text-to-speech and speech-to-text conversion. Similarly, videos can be converted to streams. These transformations were seen in the previous class.

Scope — the "clean source" assumption is a myth. Textbook examples assume a tidy source system that exports clean tables. In practice the source is messy: an analog sensor that must be digitized, a recorded lecture that must be transcribed, a video that must be chunked into a stream. Every transformation between the source and your pipeline is a place where format and quality can change — which is exactly why validation checks exist at the entry point.

8.2.2 How Data Is Categorized

Data is categorized in many ways: is it analog data or digital data, internal data or external data, primary data or secondary data, core data or business data. You need to know where it is coming from and what type of data format it uses. There are also first-party data, second-party data, and third-party data — who is the creator, where is it coming from, what are the different data types. In the previous class, the categories covered included first party, second party, and third party data with the creator, origin, and data types for each.

The categorization matters because different data types carry different characteristics. A question may test exactly this: given data that is highly heterogeneous and arriving at high speed, how do you map it, and what characteristics belong to what data type.

Category axis	Options	What it tells you
Form	Analog vs. digital	Whether conversion (digitization) is needed before ingestion
Boundary	Internal vs. external	Who owns it and which access rules apply
Collection	Primary vs. secondary	Whether it was collected for your purpose or reused from another purpose
Business role	Core vs. business	Whether it is essential to operations or supports decision-making
Ownership (first/second/third party)	Your own, a direct partner's, an aggregator's	Who the creator is, where it comes from, and what access terms apply

First-, second-, and third-party data. The ownership axis deserves its own close look because the exam tests who creates the data and what you may do with it:

First-party data — collected directly by your organization from its own customers or systems (your app's click logs, your sales records). You own it, you know its provenance, and you control its quality.
Second-party data — someone else's first-party data shared directly with you through an agreement (a partner retailer sharing its transaction feed). The data contract (section 8.7) governs it.
Third-party data — bought or aggregated from a marketplace or broker (demographic overlays, credit bureau attributes). You have the least control over its quality and timeliness.

The same question applies to all three: who is the creator, where is it coming from, and what data types does it carry?

Worked example — mapping characteristics to a data type. Suppose a question gives you this: "Highly heterogeneous data arriving at high speed from many devices — which characteristics belong to what data type?"

Step 1 — Identify the workload. "High speed" and "many devices" point to telemetry: sensors, mobile events, or IoT logs. The data is semi-structured (JSON event payloads), time-stamped, and continuous.

Step 2 — Map the characteristics. High velocity maps to a streaming platform (Kafka, Kinesis — see section 8.5); heterogeneity of format maps to schema-on-read stores (NoSQL document stores — section 8.3); the time stamps make it time-series data.

Step 3 — State the storage answer. Time-series database or event stream with a TTL policy, plus validation at the entry point.

Sense-check: every characteristic named in the question (heterogeneous, high speed, time-stamped) now has a named tool and a named reason. That is the mapping the question expects.

Exam note: expect questions on the characteristics of data, the different types of data, and what characteristics belong to which data type — for example, how to map data that is highly heterogeneous and arriving at high speed. The exam questions are contributed by three professors, so some will test exactly this kind of characteristic mapping. Answer by pairing each characteristic with a tool or store type, and name the reason.

8.2.3 Format Problems: Malformed and Unexpected Data

Sometimes the data is not formatted correctly — malformatting. Sometimes you have a long text, too long. Sometimes people give only initials — "IN" alone — and that may be okay. Sometimes they have a long name, and this creates a real data quality problem. The example used in class: a railway station in Wales with a very long name — Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch — versus a very small name like "LB", which stands for that same long name. There is also a railway station in Chennai with a long name. These real examples illustrate why format problems matter when ingesting data: names, addresses, and identifiers arrive in every possible shape, and the pipeline must handle them.

Worked example — the longest and shortest station names. One real station name is the 58-letter Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch (a village on the island of Anglesey, Wales). In railway systems it is abbreviated to "LB". A Chennai station carries another long name. Now consider a pipeline that ingests station codes:

A field defined as exactly 2 letters receives "LB" — passes a naive format check.
The same field receives the full 58-letter name — a format check that caps length at, say, 20 characters would reject it as malformed.
Some records carry "IN" as a person's initials — valid, but indistinguishable from a country code "IN" to a machine.

The point is not that one answer is right. The point is that the pipeline must decide — enforce a length rule, expand abbreviations against a lookup table, or flag ambiguous tokens for human review. No single decision is correct for every system, but leaving the decision unmade guarantees inconsistent data.

Sense-check: every shape of "name" data — long, short, ambiguous — has now been given a concrete handling path, which is exactly what a format check must specify.

The teaching point: there are a lot of challenges here, and this is why pre-ingestion validation, validation checks, and quality checks matter. Pay attention — this is where a couple of questions come in the examination.

Pitfall — validating what is easy, not what is risky. A format check that only tests "is this a string?" catches nothing — the danger lives in boundary cases: length limits, abbreviation tables, encoding (Unicode names with accents), and tokens that mean different things in different columns.

Pitfall — hard-coding one format. If the pipeline assumes every station name fits 20 characters, the 58-letter name silently truncates and corrupts downstream joins. Decide explicitly: reject, truncate with a flag, or map through a lookup table.

8.2.4 Behavioral Data and Collaborative Filtering

A lot of data comes from system logs and user behavior. With a system log viewer, you look at specific keywords and extract what you need from system error logs — where it says error, where there is a warning — and then the behavior: user behavior. This is one of the important things in the traditional system. There is something called collaborative filtering: most recommendation systems work based on the behavior of the user — what they click, similar patterns, where they scroll, where they zoom, how many people ignore the pop-up, are they choosing the suggestion or not. All those things feed behavioral analytics. At the end of the day, you have so much data coming from this.

Collaborative filtering — the technique behind most recommendation engines: it recommends an item to a user because other similar users behaved toward it the same way. "Similar" is measured from behavior, not from stated preferences. The raw materials are exactly the behavioral signals the class described: clicks, scrolls, zooms, pop-up interactions, and whether a suggested item was chosen. Every such event is a record; every record is data for ingestion.

Scope — behavioral data only tells you what happened, not why. A user who ignores a pop-up may not dislike the offer; they may have been driving. Collaborative filtering models behavior as preference and is blind to the reason — an important limit when you interpret what the data means, and a reason to keep the raw events (not just the derived signals) in the pipeline.

Real-world: system logs (viewed with a tool like the Windows System Viewer) plus click-stream behavior power recommendation engines through collaborative filtering; there is a plan to show RSS feeds, real-time feeds, and analytics on an e-commerce platform like Flipkart after the midterm. Log lines carry severity labels — error, warning, info — and the ingestion layer filters and routes them, while click-stream events stream continuously into the analytics platform.

8.3 Databases and Data Stores for Ingestion

8.3.1 Relational Databases and ACID

Irrespective of the system, most systems have some data model, and it follows SQL, NoSQL, or something else, maybe a graph database. Data gets fed from SQL relational databases, which you already know. Relational databases have the advantage of the ACID property: atomicity, consistency, integrity, durability, and they are very easy to work with.

ACID — the four guarantees a relational (SQL) database gives a transaction:

Atomicity — the transaction is all-or-nothing: if any step fails, the whole transaction rolls back, so no half-written row survives.
Consistency — every transaction moves the database from one valid state to another, obeying constraints like foreign keys and unique indexes.
Integrity (often called isolation) — concurrent transactions do not see each other's partial work, so two simultaneous updates cannot corrupt one another.
Durability — once a transaction commits, its effect survives crashes and power loss because it is written to durable storage.

These guarantees make relational databases easy to work with: applications can rely on the data being correct and complete after every operation. That reliability is why so many source systems are SQL databases and why ingestion pipelines must handle their output with care.

8.3.2 NoSQL, Document, and Columnar Stores

When you want more unstructured data, you can use NoSQL — for example, a key-value store. Data can also be held as a JSON object in a document database. There are also many column databases — columnar databases store data by columns instead of rows. Columnar databases are very useful for IoT data. Real-world: DynamoDB (Amazon), Bigtable (Google), and Apache Cassandra are all column-based databases. MongoDB is a document database, and there is a practical plan to set up a Mongo cluster and a MySQL cluster after the midterm. Elasticsearch is useful for keyword and search databases.

Intuition — rows versus columns. A row-oriented table keeps all fields of one record together: record 1's fields, then record 2's fields, and so on. A columnar database keeps all values of one field together: the whole first column, then the whole second column. Think of a spreadsheet: row storage reads a whole row in one go (great for transaction lookups), while columnar storage reads one column across millions of rows without touching the others (great for analytics over IoT readings like "average temperature per sensor").

Where each fits:

Key-value store (e.g., DynamoDB) — look up one value by its key, at massive scale and low latency.
Document database (e.g., MongoDB) — store self-contained JSON documents; each document may have a different shape, so heterogeneous records need no fixed schema.
Columnar / column-family store (e.g., Bigtable, Cassandra) — wide tables organized by column; ideal for IoT and time-stamped sensor data where analytics read entire columns.

8.3.3 Search Databases for Text

Text search involves searching the entire body of a text for keywords and phrases, matching on exact or semantically similar matches. When the requirement is text search, the recommendation is a search database rather than a relational one. The answer pattern is short: use search databases, use NoSQL databases, something like Elasticsearch. That is the whole answer — two lines, simple.

Why SQL is the wrong answer for text search. A SQL database can search text with a LIKE clause, but it scans records one by one and cannot rank results by relevance. A search database builds an inverted index — a map from every word to the documents containing it — so a 20-keyword query is answered by intersecting word lists in milliseconds, with relevance ranking, fuzzy matching, and synonyms. That is why the answer pattern is short and deliberate: search databases, NoSQL, Elasticsearch.

Real-world: log analysis also calls for search databases. You have very big transaction logs, customer click logs, network logs, real-time monitoring, and anomaly detection — in these cases do not say "I use a SQL database"; that is not good. Elasticsearch-type tools handle keyword matching across the whole body of text.

8.3.4 Time-Series, Graph, and Other Stores

One important thing is temporal data — we deal with a lot of temporal data, time series data. Whenever you have high velocity data and you are also interested in the time, a time-series database fits. The train journey from Amsterdam to Luxembourg provided a vivid example: lots of temporal data was carried — the location data over time as the train moved, the temperature changing, weather changes at different times, rain at certain times. All those changes are temporal data.

Graph query language is one of the very popular options. If you have lots of graphs, use Neo4j or another graph database.

Intuition — what makes a time-series database different. Ordinary databases store state — the current value of things. A time-series database stores change — a stream of (time, value) pairs per sensor or entity, with special handling for high write rates, time-based queries ("between 9:00 and 10:00"), downsampling, and retention. In the Amsterdam–Luxembourg example the entity is the train and the series are its position, cabin temperature, and the weather at each moment; each is a (timestamp, value) pair. High velocity plus time-of-interest is the signature that says: time-series database.

8.3.5 Recommending a Database for a Scenario

These keywords are important for the exam. A plausible scenario: you are a data engineer building a solution and a data pipeline, and your duty is to understand the keywords given by the user and recommend what type of database — SQL, NoSQL, key-value, or something else — with examples of tools. In that situation, apply search databases or Elasticsearch instead of relational databases.

Store type	Best when	Tool examples
Relational (SQL)	Structured transactions, ACID guarantees needed	MySQL, PostgreSQL, Oracle
Key-value	Single-key lookups at huge scale	DynamoDB, Redis
Document (NoSQL)	Heterogeneous JSON records, flexible schema	MongoDB
Columnar / column-family	Analytics over wide tables, IoT sensor data	Bigtable, Cassandra, DynamoDB
Search	Keyword and full-text search across bodies of text	Elasticsearch, Solr
Time-series	High-velocity time-stamped data	InfluxDB, TimescaleDB
Graph	Connected data, relationships and traversal	Neo4j

Worked example — the data-engineer scenario. "A customer wants to find all the issues related to a specific product, using 20 keywords to search. What is your recommended data pipeline, and what type of database do you recommend?"

Step 1 — Read the keywords. "20 keywords", "find issues", "related to a specific product": this is full-text search over a large body of issue records, not a look-up of one row.

Step 2 — Match the store. Text search over the entire body of text → search database. The answer pattern from the lecture: search databases — a NoSQL solution such as Elasticsearch.

Step 3 — Name the pipeline. The issues land in the search index via ingestion (API or message queue), and the application queries the index with the 20 keywords. The same logic extends to log analysis: transaction logs, click logs, network logs, real-time monitoring, and anomaly detection are all search-database use cases — a SQL database is not the right tool there.

Sense-check: every keyword in the question ("20 keywords", "find issues", "related to") maps to a named store and a named reason; the answer ends in a concrete tool, not a generic phrase.

Q: As a data engineer, if a customer wants to find all issues related to a specific product using 20 search keywords, what kind of database do you recommend?

A: Recommend a search database — a NoSQL solution such as Elasticsearch. Text search involves searching the entire body of text for keywords and phrases, matching exact or semantically similar terms, so relational SQL is not the right tool. The same logic applies to log analysis: transaction logs, click logs, network logs, real-time monitoring, and anomaly detection are all search-database use cases.

Exam note: expect scenario questions asking you to recommend a database type with tool examples such as Elasticsearch. The answering pattern is short: name the store type, name a tool, and give the one-line reason tied to the question's keywords. When the trigger word is "search", the answer is a search database, not SQL.

Pitfall — answering with the tool, not the reasoning. "Use Elasticsearch" alone is a weak answer. The examiners want the mapping: the requirement is full-text search over the whole body → search databases → such as Elasticsearch → because SQL cannot rank and scan large text bodies efficiently.

Pitfall — SQL reflex. For structured transaction data, SQL is right. For text, logs, time series, or graphs, reflexively saying "SQL database" loses the marks — the question is testing whether you can choose the store by the data's characteristics.

8.4.1 Calling APIs: REST, RPC, and GraphQL

There are lots of ways to get data. You can use APIs — there are lots and lots of APIs available; you just call the API and it gives data back. That may be a RESTful service, a web service, or a simple RPC call. You can use open source libraries; many SaaS solutions and open source solutions are available.

Predominantly, we use REST API. When the systems are complex, we use a GraphQL API. Graph query language is very, very popular when you have lots of graphs, with Neo4j or other graph databases. There are other API patterns — the broker pattern and consumer patterns were studied earlier, and they are used for data services. Fundamentally it is a simple client-server mechanism: you call the server, and the server provides the data back. RPC is used too — gRPC is Google's RPC framework. An everyday example: even the class has used such a call to locate someone via a mapping service; the same mechanisms can be integrated into your own applications.

The API call — a five-step client-server mechanism. Behind every "get me the data" there is the same walk:

Client builds a request — the URL of the resource, the HTTP method (GET for reading), and any query parameters or keys.
Request travels over the network to the server.
Server authenticates and authorizes the call (API key, token) and executes the operation.
Server returns a response — typically structured data (JSON) plus a status code (200 for success, 401 for a bad key, 429 when rate-limited).
Client parses and stores the payload for the next pipeline stage.

The everyday mapping example the class has used works exactly this way: your phone is the client, the mapping service is the server, and the response is location data — the same call your applications can make programmatically.

API style	Best when	Notes
REST	The common case — simple, stateless resource access over HTTP	The predominant choice; every major SaaS exposes one
GraphQL	Complex systems needing flexible queries over many related objects	Ask for exactly the fields you need in one request
RPC / gRPC	Fast, typed service-to-service calls inside a system	gRPC is Google's RPC framework; efficient binary transport
Message broker patterns	Loose coupling between producers and consumers	Broker and consumer patterns studied in an earlier contact session

Worked example — pulling data from a system that does not follow standard formats. The customer says: "Our legacy system exports nothing standard; we need the data anyway."

Step 1 — Look for an API. Nearly every system exposes one: RESTful services, web services, or at least a simple RPC call. Call it, and it gives data back.

Step 2 — Escalate by complexity. If the system is complex and graph-heavy, GraphQL is very popular, together with graph databases such as Neo4j.

Step 3 — State the mechanism. Fundamentally it is a client-server mechanism: you call the server and it provides the data back.

Sense-check: the answer gives a first choice (API), an escalation path (GraphQL for complex systems), and the underlying mechanism — exactly the structure an exam answer needs.

Q: How do we pull data from a system that does not follow standard formats?

A: You call an API — there are lots of APIs available, RESTful services or web services, or a simple RPC call. For complex systems with lots of graphs, GraphQL is very popular, together with graph databases such as Neo4j. It is basically a client-server mechanism: you call the server and it provides the data back.

Data sharing can streamline the data pipeline within the organization, because you know what data each pipeline needs. Suppose you work on retail banking, somebody works on credit cards, somebody works on insurance. What data needs to be shared for the retail pipeline? What data needs to be shared for the credit card pipeline? Only that segment of data is shared. This is done using role-based access control, RBAC. Based on the role you are doing and what type of data you want, access is given accordingly — you can access a particular table, even a particular row, all controlled at the database level.

Worked example — sharing by segment in a bank. Three business pipelines run off one enterprise data platform: retail banking, credit cards, and insurance.

The retail pipeline needs customer demographics and account balances — but not card transaction histories.
The credit card pipeline needs card transactions and limits — but not insurance claims.
The insurance pipeline needs claims and policies — but not retail balances.

RBAC assigns each pipeline's service account a role; each role has grants on exactly the tables (and even the rows) it needs. A user in the credit card role who queries the insurance claims table is refused at the database level — the control is enforced by the database, not by politeness.

Sense-check: every pipeline gets its own segment, the sharing is explicit and auditable, and access control is enforced at the database level — the three properties the lecture emphasizes.

Real-world: RBAC at the database level controls row- and table-level access so each business pipeline sees only its own segment of data.

Scope — RBAC governs access, not quality. RBAC decides who may read which rows; it does not decide whether the rows are correct, current, or consistent. That is the job of validation (sections 8.8–8.9) and the data quality dimensions (section 8.11). Keep the two concerns separate when you design the pipeline.

8.4.3 Third-Party and Open Data Sources

There are a lot of third-party data sources. Real-world: the Indian government publishes open data at data.gov.in, NASA has a data portal with many datasets, and many other third-party data sources exist. There is also data available in the cloud that you can access directly.

Why third-party data needs extra care. Unlike your own first-party data, you did not create it, you do not control its quality, and its format may change without notice. That is why third-party sources are a prime place for the ingestion contract (section 8.7) and for pre-ingestion validation (section 8.8): the checks you run on your own data are the same checks that protect you from someone else's.

8.4.4 Message Queues and Event Streams

There are a lot of message queues and streamings. The MQ series: there will be a bus, there will be a queue. A producer puts data into the message queue, and subscribers and publishers subscribe — like Instagram: a musician publishes a new song, you subscribe, and you are able to get it. That is the message queue model.

Streaming covers data that is always being produced — a backup running, snapshot replication running, something continuously running. Real events are tracked: how many people are coming, how many people are listening, how many people are responding. We use an event queue for this.

Intuition — the Instagram song model. A musician publishes a new song; you subscribe; the moment it is released you get it. The musician is the producer, the platform is the message queue, and every subscriber is a consumer. The producer does not know or care how many subscribers exist — it publishes once, and the queue delivers to all who subscribed. That decoupling is the whole point: producer and consumer never need to be available at the same instant.

Message queue versus event queue. A message queue moves discrete messages between producers and consumers (a song release, an order confirmation). An event queue handles continuously produced streaming data — a backup running, snapshot replication running, live audience counters ("how many are coming, how many are listening, how many are responding"). Same family, different rhythm: discrete deliveries versus an endless stream.

Real-world: message queues follow the producer-subscriber model (the Instagram song-publishing analogy), while event queues handle continuously produced streaming data such as live audience events. Both feed ingestion: the queue is where the pipeline attaches to consume what was produced.

8.5 Data Ingestion: Batch, Streaming, and Ingestion Methods

8.5.1 What Data Ingestion Is

This is flagged as the key material — roughly 50 percent of the exam answers come from this. Modern data integration begins with data ingestion. Data engineers use data transformation pipelines to better handle the scale and complexity of business demands for data. Lots of data pipelines operate continuously across the organization, without direct involvement of a development team. The majority of the data pipeline captures data in batch, as well as through SQL queries or routines. Data ingestion is where the effort in a modern data platform gets started.

Hook — the pipeline's front door. Everything downstream — storage, transformation, modeling, dashboards — consumes what ingestion delivers. If ingestion is built badly, nothing downstream can repair it cheaply: the data is already missing, late, duplicated, or malformed. That is why the material on this topic is worth about half the exam answers.

Exam note: this ingestion material is the key to about half of the exam answers. Questions may come from data ingestion, from the importance of data catalog and governance, and from data transformation. This all connects back to the CRISP methodology studied earlier in the life cycle — everything is the same, basically: there is a data pumping in, the data comes, it goes through stages, and it is used for modeling. Whenever you answer an ingestion question, you are answering a life-cycle question.

8.5.2 Batch Ingestion: Time-Based and Size-Based

A key design decision: do you go for time-interval-based ingestion or size-based ingestion? Are you going to do a full snapshot or a differential? Do you want a batch insert or a batch update? For example, a large financial data provider ingests by region and by asset type on a schedule: asset data from 8 to 10, other transaction data from 11 to 1, then cash flow and other things, long-term mutual funds. That is size-based and time-wise ingestion. A mainframe may flood the data at certain times — for example, around Christmas or during big stock market activity, a lot of data is pushed between 9 and 10. Either you go by time interval or by size.

The two scheduling decisions. Batch ingestion runs at fixed windows and moves bulk data:

Time-interval or size-based? Time-interval: run every night at 2 AM. Size-based: every time 100 MB of events accumulate, cut them into a file for the data lake. The financial provider's schedule is both: asset data 8–10, transaction data 11–1, then cash flow and mutual fund data — each window timed to its source.
Full snapshot or differential? A full snapshot copies the entire current state of the source each run — simple and complete, heavy on network. A differential (incremental) pull copies only the rows that changed since the last read — light on traffic, but you must track watermarks and handle deletes.
Batch insert or batch update? Insert adds new rows; update modifies existing ones. Columnar stores are built for big bulk inserts and slow down badly on many small in-place updates — know which pattern your destination prefers.

A mainframe floods data at specific moments — around Christmas or during heavy stock market activity, a large push lands between 9 and 10. The design question is the same: cut the flow by time or by size.

Worked example — the Salesforce-to-warehouse pipeline. A classic end-to-end batch ingestion scenario: data sits in Salesforce.com, and the users only see Tableau.

Step 1 — Extract. Go to Salesforce and use the API or web services it exposes; push the data out.

Step 2 — Transport and translate. From your application, pull the data, write some SQL or equivalent to translate it, and load it into the data warehouse.

Step 3 — Model. The warehouse is built using a star schema or snowflake schema (both studied when data warehouse was covered). One part of data ingestion is done: reading from the legacy or third-party database into the warehouse.

Step 4 — Serve. Now pull from the warehouse: the person who wants to know something gets a customized Tableau report, a customized Power BI report, customized reports.

Sense-check: each stage names its tool (Salesforce API → application → warehouse → Tableau/Power BI), and the warehouse schema (star/snowflake) explains how the report queries are answered. That four-stage walk is the model answer for "describe a batch ingestion pipeline".

Worked example — Twitter feed for sentiment analysis. APIs are available to get the feed — you have the Twitter handle, the access control, the API keys. This has been used with Twitter feeds, Facebook, and LinkedIn. Then you can do real-time analysis: suppose a public leader is tweeting — what type of response is coming, somebody else in the back end may be responding. That is sentiment analysis.

Step 1 — authenticate with the API keys and access control; Step 2 — pull the feed for the handle; Step 3 — analyze the responses as they arrive to gauge sentiment; Step 4 — feed the result into real-time analysis and reporting.

Sense-check: the pipeline runs from authentication through collection to analysis — and the same access-key pattern applies to any social platform feed.

Real-world: Salesforce-to-warehouse pipelines feeding Tableau and Power BI reports are a common batch ingestion pattern; Twitter and social feeds pulled through APIs with access keys feed real-time sentiment analysis. AWS tools can be used for these decisions: for a petabyte-scale data migration, or for compute and storage for hybrid workloads, depending on the data you can load batch-wise or full, or use batch data migration to migrate the data to a new database.

The freshness cost of batch. Batch systems are, by nature, a day behind. Typically there is an 8-hour difference, 12-hour difference, or 24-hour difference — one day behind, always. We use replication and mirroring to handle this: a replica database keeps a near-current copy so reads can run against it, and mirroring reduces the gap between what the source knows and what the warehouse shows.

8.5.3 The Batch War Story

Worked example — the State Street overnight job. A war story from working at State Street, a financial services company: a batch program was written that starts at 8 PM and runs until 4 AM. The engineer was on 24-by-7 production support, carrying a beeper — the olden days' pager, a small pager that vibrates — plus a phone, a VPN, and an RSA security token wherever he went. The job starts extracting, platforming, and loading at 8 PM and completes at 4 AM. From 4:30 onwards, customers can run a report. But the data is always 8 hours, 12 hours, or 24 hours behind — batch systems are always a day behind.

Step 1 — The schedule is fixed. 8 PM start, 4 AM completion, eight hours of extract–platform–load work while the trading day is over.

Step 2 — The support is real. Someone carries the pager, phone, VPN, and RSA token at all times, because a failed overnight run means no morning reports.

Step 3 — The freshness ceiling. Reports are available from 4:30, but they describe yesterday's data. A customer who asks "what happened at noon?" cannot get a real-time answer from this pipeline — the answer is hours or a day old.

Sense-check: the story shows the three facts about batch in one picture: fixed window, dedicated operations support, and inherent staleness. Batch is chosen for its simplicity and cost, and its price is that the data is always a day behind — the reason replication and mirroring exist.

8.5.4 Streaming, Replication, and Late Arrival Data

Streaming handles continuously produced data. An example from the automotive domain: a car is driven by somebody, and you want to find out whether people are wearing the seatbelt, what the head position is, what the eye position is, what the phase is — AI can be used for that. One M Tech project reviewed by the professor did exactly this for Daimler, the car company, simulating how a human expects things: a human knows a shadow; a self-driving Tesla sees everything as an object — even the shadow is an object. Since everything is an image, the model must handle the shadow in real time, which requires lots and lots of data streaming.

Intuition — batch answers yesterday's question; streaming answers now's. A camera in a car streams frames continuously; the model must decide about the seatbelt, head position, eye position, and the phase of the drive in real time. Even a shadow is an object to a self-driving car — it cannot wait for an overnight batch to learn that the shadow moved. When the decision is time-critical, the data path must be streaming.

Late arrival data is a real streaming problem. A typical example: buying a train ticket in Amsterdam with a debit card. The email came before the SMS — interesting, because the SMS should come first since the phone number is registered. Good or bad, you don't know; as long as you know you get it, seriously. Immediately the email came while the phone was being configured, then a call also came asking "are you the one who did it?", and only after five minutes the SMS arrived, then the balance. If the data is not handled correctly, you have a problem.

Multiple delivery is another observed issue: some banks repeatedly send the same message, and the same message repeatedly floats around. Duplicate messages confuse customers into thinking they performed the transaction twice. Event ordering can also get disturbed.

Q: Why do bank apps send the same confirmation message multiple times?

A: Event ordering can get disturbed. When a change data capture (CDC) fires because there is a change in data, and another change happens for the same record, two events are fired and both are put into an asynchronous queue — and there is no predictability about which will come out first. That is why out-of-order delivery happens. Duplicate delivery also occurs: some banks send the same message twice, which confuses customers into thinking they paid twice.

Why out-of-order happens. Change data capture reads the database's change log and emits one event per change. Two rapid changes to the same record produce two events; in an asynchronous queue with multiple workers, event 2 can be consumed before event 1. The events themselves are fine — their order is not. Streams must handle duplicates (at-least-once delivery) and reordering, exactly as the train-ticket story shows: email, then a verification call, then the SMS, then the balance — the events arrived, but not in the order anyone expected.

Real-world: CDC-based event systems using asynchronous queues can deliver events out of order; banks such as some Indian banks have been observed sending duplicated confirmation messages.

8.5.5 Time to Live (TTL)

Time to live is one of the very critical parameters. In networking, a network packet — a TCP/IP packet or a UDP datagram — carries a TTL that decides how long the packet may travel, often around 30, 40, or 38 seconds; the network routers decrement it. In streaming, TTL is how long you preserve the event record. An extremely short TTL, milliseconds or seconds, might cause most messages to disappear before processing. A very long TTL may create a backlog, and some intruder — a hacker — may come and mess with your data, and many unprocessed messages result in a long wait time.

TTL — the same word, two worlds. In networking, a packet (a TCP/IP packet or a UDP datagram) carries a time-to-live: the number of hops or seconds the packet may travel before routers discard it, often around 30, 40, or 38 seconds; each router decrements the counter. It prevents a packet from looping forever. In streaming, TTL is the maximum retention time of an event record: how long an unacknowledged event is preserved before it expires and automatically disappears. An extremely short TTL — milliseconds or seconds — can make most messages disappear before they are processed. A very long TTL — several weeks or months — creates a backlog of many unprocessed messages, long wait times, and a wider window for intruders to reach the queue. The right TTL is a deliberate balance.

Real-world: Google Cloud retention is up to seven days; Amazon Kinesis data streams are up to 365 days. How long it is, is your choice.

Q: What is "time to live" (TTL)?

A: In networking, a packet such as a TCP/IP packet or a UDP datagram carries a TTL that decides how long it can travel — often around 30 or 40 seconds. In streaming, TTL is how long the event record is preserved. An extremely short TTL, milliseconds or seconds, can make most messages disappear before processing. A very long TTL can create a backlog, let intruders or hackers mess with your data, and leave many unprocessed messages waiting. For reference, Google Cloud retention is up to seven days, and Amazon Kinesis data streams go up to 365 days.

The TTL balance. Too short: legitimate messages expire before a worker reads them — silent data loss. Too long: the queue grows without bound, processing falls behind, and the data sits exposed. The lecture's rule of thumb: pick the shortest TTL that your slowest legitimate consumer can tolerate. For a nightly batch consumer, a few days; for a near-real-time system, minutes. The platform limits — 7 days on Google Cloud, 365 days on Amazon Kinesis — set the ceiling, not the target.

How you handle errors in streaming matters: a stream too large to ingest post-ETL, not able to ingest — that is why you need a proper CDC mechanism, change data capture, and proper queue handling: how you handle the queue, and then the error handling mechanism. Events that fail (too large, expired TTL, unknown topic) should be rerouted to a dead-letter queue so one bad event never blocks the rest of the stream.

8.5.6 Push versus Pull Ingestion

Do you want to push or do you want to pull? Both ways are available. Based on the subscriber model, a subscriber reads a message from a topic. RabbitMQ supports a push subscription. Pull subscriptions also exist: you can pull whatever you want, based on the permissions available — only the message whatever you want to pull, only pull. Instead of being pushed, you get what you want.

The professor's analogy — Subway. At Subway, the sandwich is pushed to you: whatever they have in their sandwich is pushed, and you have to taste it — whether you like jalapeno in it, whether you like the layers in it, the sandwich is pushed and you take it. Rather, you have been given an option: you choose your bread, you choose the ingredients — extra cheese if you want, fewer vegetables if you don't. That is pull — pull only what you want, with your own flexibility, cost savings, and your taste.

Where the analogy maps and where it breaks. Push = the producer decides what you receive (Subway assembles the sandwich; you eat it as made). Pull = the consumer decides what to fetch (you build the sandwich: your bread, your ingredients, your portion). The analogy breaks in one respect: a Subway sandwich has no deadlines — but in streaming, a pull consumer that waits too long lets the TTL expire and loses messages. Choice has a price: with pull, the consumer carries the responsibility to read in time.

Dimension	Push	Pull
Who decides what you receive	Producer (the queue writes to you)	Consumer (you fetch what you want)
Delivery model	Subscriber model — RabbitMQ supports push subscriptions	You pull based on the permissions available
Flexibility	You take what comes	You get only what you want — flexibility and cost savings
Responsibility for timeliness	Producer-side delivery	Consumer must read before TTL expires
Typical tools	Pub/Sub, RabbitMQ push	Kafka and Kinesis pull subscriptions

Exam note: when the question says "push or pull", answer with the Subway sandwich analogy: push is the sandwich made for you — you take what comes; pull is building your own sandwich — you choose your bread and ingredients, extra cheese if you want, fewer vegetables if you don't. Then add the one-line technical mapping: RabbitMQ supports push subscriptions; pull subscriptions let you fetch only what your permissions allow.

8.5.7 Ways to Ingest Data

The ways to ingest data: direct DB connection; CDC mechanism; APIs; message queues; streaming; object storage; file export — you just export the file from an object store; shell and secure shell; secure FTP; secure copy; webhooks; and other mechanisms available. These are the ways through which you can make the ingestion happen.

The ingestion toolbox. Ten named channels, each for a situation:

Direct DB connection — query the source database over JDBC or ODBC; the classic pull for relational systems.
CDC mechanism — change data capture: read the source's change log and emit an event per change; the streaming-friendly replacement for repeated full pulls.
APIs — RESTful or web service calls returning structured data.
Message queues — producers publish, subscribers consume (the Instagram model).
Streaming — continuous event ingestion for time-critical data.
Object storage — read files directly from a bucket like Amazon S3.
File export — the source exports a file you then import.
Shell and secure shell — scripted remote access for ad-hoc pulls.
Secure FTP and secure copy (SFTP/SCP) — encrypted file transfer.
Webhooks — the source calls your endpoint when something happens (push-style event delivery).

The exam answer pattern: name the channel, name the scenario that calls for it, and name a tool where one exists.

Real-world: common ingestion channels include JDBC direct connections, CDC, REST APIs, message queues like Kafka, streaming platforms, object storage, file export, SFTP/SCP, and webhooks.

8.6 ETL versus ELT and Ingestion Tools

8.6.1 Why ETL Is Not Enough

Whenever there is complexity, it is going to take time. When there are a lot of changes, it takes time. Maintenance and rework take time. The old ETL process is not really, really good for these demands, so you need to go with a new model. That is where the ELT model comes in — you make it flexible.

Hook — the pipeline that must be rewritten for every question. In classic ETL the transformation logic lives in the pipeline code. Every new source field, every changed business rule, every new report means editing, testing, and redeploying that code — maintenance and rework take time. When complexity and change are the norm, the pipeline becomes the bottleneck. The answer is to move the transformation somewhere it can scale without a code deployment: the ELT model.

Exam note: ETL versus ELT is a likely question — possibly not asked exactly as "ETL versus ELT", but indirectly. Be ready to explain when you use ELT instead of ETL: the trigger words are scale, complexity, frequent changes, and heavy rework.

8.6.2 The ELT Model

The contrast: ETL (extract, transform, load) transforms the data before loading it into the target. ELT (extract, load, transform) loads first and transforms later, in the target platform. The ELT model is more flexible for modern scale and complexity, because the transformation happens inside a scalable warehouse platform instead of in the pipeline. That flexibility is the core reason to choose ELT when there are lots of changes and heavy rework in the old ETL process.

	ETL	ELT
Order	Extract → Transform → Load	Extract → Load → Transform
Where transformation runs	In the pipeline, before the data lands	Inside the target platform (the warehouse)
Load timing	Load only after transformation succeeds	Load first, transform later on demand
Scale behavior	Pipeline capacity limits the work	Warehouse compute scales independently
Change handling	Every rule change needs pipeline rework	New rules are queries against loaded data
Typical era	Traditional warehousing, smaller data	Modern cloud platforms, big data

When to pick which. Use ETL when the target store cannot transform data well (a simple database or a compliance need to clean data before it ever enters). Use ELT when you have large volumes, a scalable warehouse, and frequent business changes — the transformation runs where the compute lives, and rework becomes a query change, not a code deployment. One-line rule from the lecture: ELT exists to make the process flexible under modern scale and complexity.

8.6.3 Tools: Hand-Coded, ETL Suites, and Modern Platforms

Tools for the transformation pipeline range from manual hand coding to enterprise suites. A lot of manual hand coding was done in practice — a lot of SQL programs, C programs. ETL tools: Informatica, IBM Data Stage, Oracle Warehouse Builder, ODI — Oracle Data Integrator. SAP also has something: SAP HANA has a lot of things built in.

Worked example — the same file, two toolkits. A customer file with 10,000 rows needs cleaning before reporting.

Hand-coded: a SQL program (or, classically, a C program) reads the file, applies the transform rules, and writes the cleaned output. Total control, but every rule change is a code change — the maintenance cost that ETL critics describe.
ETL suite: the same logic is configured in Informatica, IBM Data Stage, or Oracle Data Integrator (ODI) as data flows with mapping rules, schedulers, and error handling built in. Less hand code, faster to change, but the suite itself is the platform.
Modern platform: SAP HANA has much of this built in; cloud warehouses transform in place, which is the ELT direction.

Sense-check: the same business logic exists in all three, but the cost of changing it drops as you move from hand code to suite to platform — exactly the flexibility argument for ELT.

Real-world: Informatica, IBM Data Stage, Oracle ODI, and SAP HANA are classic ETL platforms; modern ingestion also targets cloud platforms.

8.6.4 Source and Destination Types

Data teams have moved away beyond the walled garden of the enterprise data — there is a lot of third-party data, a lot of unchecked data. Common data source types: Apache Kafka can give data; JDBC can give data; Oracle change data capture systems can give replication views and data; some HTTP clients can view the HTTP data; HDFS, the Hadoop Distributed File System, where data is distributed on multiple systems. These are predominantly the data source types.

Destinations can be, again, Kafka; again JDBC, back to the system; a Snowflake schema — it goes to a data warehouse model; Amazon S3 container; it may go to Databricks, the warehouse.

Scope — leaving the walled garden. Classic ETL assumed controlled enterprise sources. Modern pipelines draw from a lot of third-party and unchecked data — which is why the ingestion contract (section 8.7) and pre-ingestion validation (section 8.8) matter: you no longer control the source, so you must control the entry.

Role	Types
Sources	Apache Kafka, JDBC, Oracle CDC (replication views), HTTP clients, HDFS (Hadoop Distributed File System, data distributed on multiple systems)
Destinations	Kafka, JDBC (back to a system), Snowflake schema warehouse model, Amazon S3, Databricks

Why the same names repeat on both sides. Kafka, JDBC, and object stores appear as sources and destinations because pipelines chain: one pipeline's destination is the next pipeline's source. Data flows from system to system through the same interfaces, which is why a stable contract and stable schema matter at every hop — a change at hop one propagates to all downstream hops.

8.7 Data Contracts, Lineage, and Governance

8.7.1 What a Data Contract Is

A data contract is a written agreement between the owner of the source system and the team ingesting data from that system for use in the data pipeline. The contract should state: what kind of data has been extracted and via which method — let it be a full extract or the incremental extract; how often we have to extract the data; and who the person and the team are — the contacts of both the source system and the ingestion team. The data contract should be stored in a well-known and easy-to-find location, such as a GitHub repo or an internal documentation site.

If you work in an IT services or IT product company, you know about SLA — service level agreement — and the underpinning contract. The data contract is the new business mode you are learning today: how long I can use it, how often I can use it, how the data is coming.

What a data contract states. Four clauses, per the lecture:

What — which data is extracted (which tables, fields, or events).
How — the method: full extract or incremental extract.
How often — the schedule and timing of extraction.
Who — the contacts: the person and team on the source side and on the ingestion side.

It is stored in a well-known, easy-to-find location — a GitHub repo or an internal documentation site — so both sides can always consult the current terms.

The SLA parallel. In IT, an SLA (service level agreement) governs the service you receive while you receive it. The data contract is the same idea moved to data: how long the data can be used, how often it comes, and how it arrives — agreed between the source system and the consumer before the pipeline runs.

The Applied Materials story — what happens without a contract. At Applied Materials, a semiconductor equipment company, contracts were signed with ASM, Intel, and Samsung. They do not provide all their data — only that of factory automation, because the focus was the manufacturing execution system. There is some equipment-level data that will not be given. If the tools collect data beyond the contract and it gets used, that is totally a data violation, leakage — data leakage. That is why you need to make sure the data contract is properly committed, signed, and agreed: the contract defines the boundary, and crossing it is not a technical bug but a legal and data-governance violation.

Q: When you connect MCP tools to Codex or Claude, you get an authentication key with an expiry date. Is that a data contract?

A: Exactly — it is almost like a service level agreement. The key is valid for a chosen period, say 24 hours or six months. In IT, an SLA runs while you receive the service; similarly, between the source system and the consumer, the parties sign a data contract stating what data is provided and for how long. The expiry date is the contract's term: access ends when the term ends.

When you use LinkedIn, Facebook, and other products, they ask a lot of tick, tick, tick boxes: would you like this? Would you like this? Can I use this for advertisement? Whenever you work with Gen AI tools, they ask: would you like to subscribe to this? How long? All those things you are giving — that is basically a data contract. It is like a service level agreement: in IT, until you are getting the service, you are doing SLA; similarly, between the source systems and the target systems, whoever is the consumer, you are signing a data contract.

Q: When apps ask "would you like this? Can we use this for advertisement?", what are they doing?

A: That is a data contract in disguise — you are giving consent for how long and how your data can be used. Every time a social network or Gen AI tool asks you to subscribe or accept, you are signing an agreement about data usage, exactly like the tick boxes on LinkedIn or Facebook. You are the source-system owner in that moment, and the app is the consumer.

Real-world: consent tick-boxes on social networks, subscription prompts in Gen AI tools, and API keys with expiry dates are everyday data contracts between a data provider and a data consumer.

Exam note: data contracts and consent agreements are the same mechanism at different scales — a written or ticked agreement about what data flows, for how long, and under what terms. When a scenario mentions "we started using vendor data we were not supposed to use", the hidden concept is the data contract, and the violation is data leakage (section 8.12).

8.7.3 Data Lineage

Data lineage is the documentation and visualization of data's journey through an organization: tracking where data originates, how it transforms as it moves through pipelines, and where it ultimately surfaces for end consumers. Producing data lineage requires: identifying the data assets; tracking those assets from ingestion sources; documenting all those sources; mapping the path of data as it moved through various pipelines and transforms; and finally pinpointing where the data was being served up in dashboards and reports.

You can visualize where the data originated; it is a kind of flow — where the data was modified, where it was transformed. At one time it was customer data, later it became integrated data, sometime it has changed its shape. We as humans have our transformations; similarly, data moves through multiple stages and towards a pipeline. That is data lineage.

The five steps of producing lineage. The lecture gives the recipe exactly:

Identify the data assets — the tables, files, and reports that matter.
Track the assets from ingestion sources — where each asset first entered the organization.
Document all those sources — who owns them and what they contain.
Map the path — how the data moved through the pipelines and transforms, changing shape at each stage.
Pinpoint the serving point — where the data is surfaced in dashboards and reports.

The result is a flow diagram: at one time the record was customer data; later it became integrated data; somewhere it changed its shape. As humans have their own transformations, data moves through multiple stages toward the pipeline.

Scope — lineage tells you the path, not the quality. Lineage shows where data came from and how it changed, but a lineage map alone does not say whether any stage corrupted the data. Combine lineage (which stage touched it?) with validation (what did that stage do?) to answer "which pipeline step introduced this error?" — that combination is how incidents get traced back to their source.

8.7.4 Data Governance and Regulatory Standards

Data governance is a data management discipline that focuses on the quality, security, and availability of data. That is where you have all the standards: the regulatory standards like GDPR, HIPAA, PII, PCIDSS, DRP, DPD — all those provide regulatory compliance. Whenever you are working on data engineering, this is a key area; there could be questions related to this.

Data governance — the discipline that keeps data high quality (accurate and fit for use), secure (protected from unauthorized access and breach), and available (reachable by those entitled to it). The regulatory standards attach legal teeth to these goals:

GDPR — the EU's General Data Protection Regulation: consent, right to erasure, data portability.
HIPAA — US health data privacy rules for protected health information.
PII — personally identifiable information: any data that identifies an individual; protected under many regimes.
PCIDSS — the Payment Card Industry Data Security Standard for card data.
DRP / DPD — disaster recovery planning and data protection directives in the governance toolkit.

In practice, governance means: catalog the data, enforce access (RBAC from section 8.4), document lineage (section 8.7.3), and apply validation so quality survives the pipeline.

Exam note: questions may come from the importance of data catalog and governance, and from what you do in data transformation. The governance answer pattern: quality + security + availability, then name the standards (GDPR, HIPAA, PII, PCIDSS, DRP, DPD) and say what each regulates.

8.8 Pre-Ingestion Validation

8.8.1 What Pre-Validation Does

Pre-validation, pre-ingestion validation, happens at data entry points — forms, APIs, ETL. Its focus is cleanliness, schema compliance, and security. Examples: reject malformed reports, wrong date format; block SQL injection attempts; ensure mandatory fields are present. Pre-validation focuses on input validation: are the schemas matching fine, are there no nulls in the important fields, do we have values in the key columns?

Mandatory fields must be present — for example, if there is a country column, the country column must be present; a date of birth column must be present. Pre-validation happens before injection, at the data entry points, and the focus is sanitization and schema compliance: rejecting non-numeric input in a numeric field, blocking SQL injection attempts, alerting admins when mandatory fields are missing.

Hook — the bouncer at the door. Pre-validation is the bouncer who checks every guest before they enter the party: is the invitation (schema) valid, is the name on the list (presence), is the guest who they claim to be (type), is the age in range? The checks run at the data entry points — forms, APIs, ETL jobs — before the data enters the system. Its three focus areas: cleanliness (no malformed records), schema compliance (fields match expectations), and security (no SQL injection, no hostile input).

Exam note: the check-level detail here is high-value. Expect to be asked what type of checks you would write for given data — whether all the fields are there, how you would handle it, each record should have a field or not, the must columns, the range of values, whether negative values come, what you do, what type of format, and data type checks. Practise writing the checks as a list against a concrete scenario.

8.8.2 The Four Core Checks

The four pre-ingestion checks are very, very important — definitely, this could be a question in your examination. They are: presence checks, range checks, format checks, and data type checks.

Presence check — is every required field present in each record? Range check — do the values fall within the allowed range? Format check — is the format confirmed, such as the date format? Data type check — is the value numeric or non-numeric, as the field expects?

The four checks as a procedure. Each check asks one question and answers it by a rule:

#	Check	Question it answers	Rule written for a system
1	Presence	Is every required field present in each record?	Reject or flag any record missing a mandatory column (e.g., country, date of birth)
2	Range	Do the values fall within the allowed range?	Reject negative sales, temperatures outside −50…+60, ages above 150
3	Format	Is the format confirmed, such as the date format?	Enforce YYYY-MM-DD dates, fixed-length codes, phone patterns
4	Data type	Is the value numeric or non-numeric, as the field expects?	Reject "abc" in a numeric field; alert on letters where digits are expected

The four checks run in sequence on every record at the entry point, and each failure routes to its action: reject, flag, or alert the admin.

Worked example — checking one record. A customer record arrives: country = "", age = "two", birth_date = "04-2020-15", salary = -5000.

Step 1 — Presence. country is empty. The country column must be present → reject or flag.

Step 2 — Data type. age holds the word "two", not a number → reject (a data type failure).

Step 3 — Format. birth_date is "04-2020-15" — a date where the year sits in the month position → reject (format failure: expected YYYY-MM-DD).

Step 4 — Range. salary = -5000 is below zero → reject (range failure).

Sense-check: every check in the four-check list fired on exactly one record, and each failure names the check and the fix — the pattern exam answers should follow.

Q: What are the four pre-ingestion checks?

A: Presence checks — is every mandatory field present, like country or date of birth. Range checks — no negative values where negatives are impossible. Format checks — dates in the right format. And data type checks — numeric fields contain numbers.

Q: Which check ensures that a start date is before an end date?

A: A consistency check, not a range check. Range is about the span of values — for example, if blood pressure comes in as -2, that is a range problem. Start date before end date is about the relationship between two values, so it is consistency.

Why the confusion happens. "Between these two numbers" sounds like a range, but range checks test one value against fixed limits. Start-before-end compares two values against each other — a relationship between fields, which is a consistency question. The quiz answer was confirmed in class: some students first said range, but the correct answer is consistency. A range check would catch a blood pressure reading of -2; it will not catch a start date that comes after its end date.

Pitfall — range versus consistency. This exact quiz question is the classic trap: "which check ensures a start date is before an end date?" The wrong answer, "range", sounds plausible because both deal with numbers. Range = one value within fixed limits (−2 blood pressure is a range problem). Consistency = the relationship between two or more values (start before end, invoice total matching line items).

Pitfall — writing checks as prose, not rules. "Check the dates" is not a check. A real check is a rule with a trigger and an action: "if birth_date does not match YYYY-MM-DD, reject with code F3". The exam rewards rules, not descriptions.

8.8.3 The Pre-Ingestion Checklist: Freshness, Risk, and More

Before ingesting the data, ask the whole list: What is the freshness of the data — how new are we getting, old data or new data? What risks are associated with it — they are giving like 20 GB data but your warehouse table cannot load that 20 GB at a time; that is a risk, and all those things need to be considered. What was the velocity at which it is coming through? What was the quality? I need to know the lineage of it — where is the data going, where is the source, where is the target, where is the flow. What is the format? Is it reusable — can it be used? What use case? What volume is expected? How often it is updated, that is freshness.

The eight pre-ingestion questions. Before data enters the pipeline, a data engineer answers:

Freshness — how new is the data; how often is it updated?
Risk — what can go wrong when it lands? (A 20 GB feed into a warehouse table that cannot take 20 GB at a time is a risk.)
Velocity — at what speed does it arrive?
Quality — does it meet the quality dimensions (section 8.11)?
Lineage — where is the source, where is the target, where is the flow?
Format — what shape does the data take (CSV, JSON, Parquet)?
Reusability — can the same data serve multiple use cases?
Volume and use case — how much data is expected, and what is it for?

These questions are asked before the data arrives, which is why this is called pre-ingestion.

Exam note: related to this list you can address questions in the exam — you have to apply your own thought process. Freshness, risk, velocity, quality, lineage, format, reusability, volume: these are the pre-ingestion questions a data engineer answers before the data enters the pipeline. When a scenario gives you details about a feed, check them against this list.

8.8.4 The Water Analogy and Prevention

The professor's analogy: before ingesting the data, there is a water body, a lake, a river. Before the river data comes into our home, don't you do some kind of checks? Making sure that water is of the quality — any chemicals mixed, is it hard water, is it drinkable, everything needs to be tested. The same applies to data: test the quality before it enters.

The fish story (told in full in the stories section) makes the same point: prevention is better than cure — apply it before the data enters the system. Please make a note of this.

The professor's river-water analogy. The river flows past your home; the lake holds the water; but before that water comes into your house you test it — any chemicals mixed, is it hard water, is it drinkable — everything needs to be tested. The same applies to data: the lake and river are the sources, the home is the pipeline, and the testing happens before the water enters. The three fishes story says the same thing in story form: prevention is better than cure — apply it before the data enters the system. Make a note of this; it is the philosophical anchor of the whole validation chapter.

Recap + bridge. Pre-ingestion validation is the bouncer and the water test: four core checks (presence, range, format, data type) at the entry point, plus the eight-question checklist (freshness, risk, velocity, quality, lineage, format, reusability, volume). The relationship checks like start-before-end are consistency — and consistency is where post-ingestion validation takes over, which is the next section.

8.9 Post-Ingestion Validation

8.9.1 What Post-Validation Does

Post-ingestion validation happens after data is stored or ingested into pipelines or a warehouse. Its focus: business rules, statistical checks, and consistency. Examples: verifying sales totals match ledger entries; ensuring time series data has no missing months; confirming referential integrity across tables.

Pre-validation is about the data at the door; post-validation is pipeline verification — business rules, anomaly detection, referential integrity.

Hook — two guards, two questions. Pre-validation asks "was this record well-formed at the door?" Post-validation asks "did the pipeline deliver the right outcome?" A file of well-formed records can still fail downstream: five rows silently dropped, totals that no longer match the ledger, a month that never arrived in the time series. Post-validation is the second guard, checking what actually happened after the data landed.

Q: What checks do we do in post-ingestion validation?

A: Consistency checks, business rule verification, and statistical checks — whether the data is skewed left or right, balanced or not, and whether governance, auditing, and business rules are satisfied. The presence, range, format, and data type checks were already done before ingestion.

Why not the same checks again? Because you already checked presence, range, format, and data type before ingestion. Here you need to make sure that consistency is consistently coming — like the professor's story about a class that was always out of balance and was somehow managed, and finally a story was told; the consistency of telling the story is always there, the consistency of giving input is always there. That is what post-ingestion means: the relationship between what arrived, what was expected, and what the business rules require.

Stage	Focus	Example checks
Pre-ingestion	The record itself, at the door	Presence, range, format, data type; SQL-injection blocking
Post-ingestion	The pipeline outcome, after landing	Consistency, business rules, statistical checks, referential integrity

8.9.2 Worked Example: N Records In, N minus X Out

The setup: there is a system one (the source) sending data to a newer system, system two (the destination). The data passes some pre-ingestion checks. Suppose the source puts in N number of such elements or records, and the destination receives N minus X number of records in the post-ingestion system:

\[ \text{records received} = N - X \]

where \(N\) is the number of records sent from the source system and \(X\) is the number of records that went missing or were rejected in the pipeline. Then what we normally do is go and check what happens to those X records. Is there any problem? If there is a valid reason for which they were rejected in the ingestion pipeline, we need to go and check what those reasons are — talk to the team, understand what is happening. The why behind it is very important. That is exactly where anomaly detection comes in.

Worked example — counting the loss. System one sends \(N = 10{,}000\) records. The destination reports \(\text{records received} = N - X = 9{,}980\).

Step 1 — Compute the gap. \(X = N - \text{received} = 10{,}000 - 9{,}980 = 20\). Twenty records did not arrive or were rejected in the pipeline.

Step 2 — Investigate the X, never ignore it. Pull the rejected records and their error codes from the queue or dead-letter log: ten failed a range check (negative amounts), six were duplicates of already-loaded rows, four were rejected as malformed by the pre-checks.

Step 3 — Decide by reason. Valid reason (rejected by a designed rule) → confirm the rule is correct and close the case. Suspicious reason (records vanished without a code) → anomaly detection: this is where the "why" matters, talk to the team, and understand what is happening.

Sense-check: the formula produced the exact number of missing records, and every one of the \(X\) records was traced to a reason — a designed rejection or an anomaly worth investigating.

8.9.3 Worked Example: The 10,000-Record File with Header

A war story from US data warehousing, where a lot of ETL process was done. Suppose there are 10,000 transaction records. With the records there is also something called an acknowledgment file, and a header file. For every file — for example, a transaction detail file for a big chipmaker customer, the transaction detail and the header for that, saying "transaction detail [customer] header" — the sender sends the file and also sends the summary of the file.

The program was written in C. A tool called Sync Sort was used to sort the files. Unix commands were applied to find the number of records: the sender says 10,000 records — check it with WC -l (the line count command). The count came back as 10K5 (10,005) — one more than expected. Then the first record and the last record were pulled: the header has the first record and the last record, so the values were matched. That is pre-validation: ensure that whatever was sent is correct or not — am I getting 10,000 records, is the header one correct, is the final one correct? After that, the data was loaded.

Post-ingestion: there is an audit table. In the audit table, for that date and time, a query runs to ensure all these 10,000 records went into the system — how many records were updated, how many records were deleted, those things. That is post-validation: you do it before taking the data into the system, and again in the pipeline when it comes.

Worked example — the 10,005 mystery. The sender promises a transaction detail file with 10,000 records, plus an acknowledgment file and a header file naming the first and last records.

Step 1 — Count. Run WC -l (the Unix line-count command) after sorting with Sync Sort. The count comes back as 10,005 — "10K5" — one more than the promised 10,000.

Step 2 — Check the boundaries. Pull the first record and the last record. The header file declares which record should be first and which should be last; match the values. If both match, the file is accepted despite the count anomaly.

Step 3 — Load. With the boundary check passed, the data is loaded into the warehouse.

Step 4 — Audit after loading. The audit table records date and time; a query runs to confirm all 10,000 records went in — how many were updated, how many were deleted.

Sense-check: the count said "one more than expected", the boundary match said "the file is what the sender claims", and the audit table confirmed the outcome after loading. The lesson: pre-validation protects the door (count + header + first/last record), and post-validation protects the outcome (audit table).

8.9.4 Leakage Checks, Model Integrity, and Skewness Checks

Post-ingestion also covers leakage checks — where are the leakages? Model integrity: if there is a problem, there is an integrity problem in the data model. Whichever data model you are preparing — whether it is a star schema, or a snowflake schema, or just the ER model, or whatever data model — ensure no target leakages, and enable the audit features. (The spoken wording reads "SAS schema"; this is almost certainly an automatic-speech-recognition error for the star schema, which is also mentioned explicitly later as "we studied star schema".)

Leakage and model integrity. Post-ingestion asks two more questions. First, leakage checks: where are the leakages — did data flow where it should not (beyond a contract, section 8.7; or future/target data reaching training, section 8.12)? Second, model integrity: is the data model itself sound? Whichever model you prepare — a star schema, a snowflake schema, or just the ER model — the pipeline must ensure no target leakages and must enable the audit features so that anomalies leave a trail. The spoken wording in the recording sounds like "SAS schema", but the context is unambiguous: the course covered the star schema and the snowflake schema as the two warehouse models, and star schema is what was studied earlier in the course.

Skewness checks: is the data towards this side or that side? There is a median, there is a mean value. Is it on the left-hand side or right-hand side? We need to find the imbalance in the data.

Skewness in the post-ingestion toolkit. A statistical check on the landed data: is the distribution towards this side or that side — where is the mean relative to the median, is the tail on the left-hand side or the right-hand side? The goal is to find the imbalance in the data, because an imbalanced or skewed column biases mean-based models downstream (developed fully in section 8.13).

8.9.5 Post-Ingestion Checks in Summary

The post-ingestion checks are: consistency checks, business rule verification, and statistical checks. Statistical checks look at whether the data is skewed to the left-hand side or the right-hand side, skewed or not skewed, balanced or not balanced — and as per governance, as per auditing, as per the business rule. The other checks are not important here because presence, range, format, and data type were already done at pre-ingestion.

Recap + bridge. Pre-ingestion = the four checks at the door (presence, range, format, data type). Post-ingestion = consistency, business rules, and statistical checks after landing (with leakage checks and model integrity), because the first four are already done. The audit table (10,000-record story) is the concrete mechanism, and anomaly detection is where the \(X\) missing records get explained. Next, the retail case study puts both stages on one dataset.

8.10 Worked Case Study: Retail Monthly Sales Data

8.10.1 The Scenario

The case study: a retail company imports monthly sales data from 200 stores. Some records have missing months; others show negative sales values. When the report is coming, this is happening — when they are importing the data. Import means we are ingesting the data from the real-time sales data. A question of this type may be expected — possibly with a table, possibly with a bit more detail. The kind of question you may get: what do you do as a data engineer, what type of pre-data ingestion validation techniques do you apply?

Hook — the ten-mark question. This case study is the model answer for the whole validation chapter: one scenario, both validation stages, worth about ten marks if you know it properly. The scenario itself is deliberately small — 200 stores, monthly sales — but it hides all four pre-checks and all three post-checks. Master this case, and the case-study questions write themselves.

Exam note: this style of question has come before, and you may expect a question like it. The scenario may come with a table or more detail; apply the concepts you have learned. If a table appears, work through its rows the way the examples below do.

8.10.2 Pre-Ingestion Handling

The answer builds on the four checks, and the whole process is automated — you are going to introduce a system that applies the checks before the data.

Presence check: the month is the mandate. It is monthly sales data, so a month record has to be there for every month. You ensure the presence of each month's record; missing months should be rejected or flagged.

Range check: block negative sales values if they come. Sales cannot be negative.

Format check: the month format is correct — year and month, e.g. YYYY-MM.

Data type check: whether the value is numeric, because sometimes it shows different values.

The students' own answers added the practical dimension: check the month format is correct; filter out the negative sales values; for missing months, either ignore them or flag them. One student pointed out that the month is very critical for the data because it is monthly sales — so if records have months missing, there is nothing we can do at pre-ingestion; skip those records, do not ingest them. For negative sales values, there are two options: skip them, or keep them and handle them later — keep them aside, and later during transformation, in the data cleaning stage, think of imputing some mean values.

The automated gate. The system applies the four checks to every incoming record before anything enters the warehouse:

Check	Rule on this data	What it catches
Presence	Every record must carry its month	Missing months
Range	Sales ≥ 0	Negative sales values
Format	Month matches YYYY-MM	Wrong date shapes like "15-2024-03"
Data type	Sales value must be numeric	Text, blanks, or symbols in the sales column

Because the process is automated, a bad record is rejected or flagged in milliseconds without human review — but the decisions below must be made first.

Worked example — running the gate on real rows. A sample of the import file:

Store	Month	Sales
S01	2024-01	560
S01	2024-02	-30
S02	2024-01	(missing)
S03	2024-03	"five hundred"

Step 1 — Presence. Row 3 has no month at all → skip it, do not ingest (the student's point: for monthly sales the month is the mandate, and at pre-ingestion there is nothing to fix — the month simply is not there).

Step 2 — Range. Row 2 shows −30 sales → two options: skip it, or keep it aside and handle it later — during transformation, in the data cleaning stage, impute a mean value.

Step 3 — Format. Row 3's month is missing entirely; any month that appears must match YYYY-MM.

Step 4 — Data type. Row 4 holds the text "five hundred" in a numeric field → reject.

Sense-check: every row in the sample maps to exactly one check and one action (skip, reject, or keep-aside-and-impute-later) — the structure the exam expects.

8.10.3 Post-Ingestion Handling

Post-ingestion validation: consistency check — ensure the monthly total aligns to the ledger total. Like say monthly total is this:

\[ 12 \times 560 = 6720 \]

The spoken comparison total in the recording is garbled — "12 into 560" — but the arithmetic is straightforward: twelve months at an average of 560 per month gives a yearly total of 6,720, which the consistency check compares against the ledger's figure. So the monthly total computed from the ingested data must equal the ledger total.

Business rules validation: sales cannot be negative; a missing month must be interpolated or flagged per review. Statistical checks: detect anomalies — direct anomalies in the data. That is it — worth ten marks if you know it properly.

Worked example — the consistency arithmetic. After ingestion, compute the monthly total from the data that actually landed.

Step 1 — Compute the sum. Twelve monthly records of 560 each:

\[ 12 \times 560 = 6720 \]

Step 2 — Compare with the ledger. The accounting ledger's total for the same period must match 6,720. If the ingested total differs, a record was lost, duplicated, or altered in the pipeline — that is a consistency failure, and it sends you back to the \(N-X\) investigation from section 8.9.

Step 3 — Run the other two post-checks. Business rules: sales cannot be negative (already blocked pre-ingestion, but re-verified on the landed data). Statistical checks: detect direct anomalies — a store whose sales jump 40× in one month, or a month with near-zero activity.

Sense-check: the number 6,720 is compared to an independent source (the ledger), which is the whole idea of a consistency check — two records of the same fact must agree.

Q: A retail company imports monthly sales data from 200 stores. Some records have missing months, others show negative sales values. What do you do as a data engineer?

A: At pre-ingestion: presence check, because the month is mandatory for monthly sales — missing months should be rejected or flagged; range check to block negative sales values; format check for the year-month format; data type check that the sales value is numeric. At post-ingestion: consistency check that the monthly total aligns to the ledger total, business rules such as sales cannot be negative, and statistical checks to detect anomalies. Missing months can be skipped at ingestion and interpolated later during transformation, or flagged for review.

8.10.4 How to Write This Answer in an Exam

Pre-ingestion: block the missing months, reject negative values, enforce numeric and data type format. Post-ingestion: verify the total with consistency checks, apply anomaly detection. Any tools can be used — third-party tools, or you can write your own. The answer structure mirrors the checks: for each stage, name the check, name what it catches in this data, and state what the system does (reject, flag, block).

The three-column answer pattern. For each stage of the pipeline, write three things:

Stage	Check	What it catches here	System action
Pre	Presence	Missing months	Skip / flag
Pre	Range	Negative sales values	Block
Pre	Format	Wrong year-month shape	Reject
Pre	Data type	Text in the numeric sales field	Reject
Post	Consistency	Monthly total vs ledger total	Alert on mismatch
Post	Business rules	Negative sales survived ingestion	Block / correct
Post	Statistical	Anomalies in the landed data	Flag for review

The tools are open: third-party validation tools or your own code — the checks matter, not the vendor.

Recap + bridge. One scenario, two stages: the four pre-checks (presence, range, format, data type) at the door, the three post-checks (consistency with the ledger total \(12 \times 560 = 6720\), business rules, statistical anomalies) after landing. This case is the bridge between the mechanics of validation and the why: every check exists to protect one of the data quality dimensions, which is the next topic.

8.11 Data Quality Dimensions

8.11.1 The Core Dimensions

Data quality dimensions are the measurements, the attributes of data, which you can individually assess and interpret and improve. The core dimensions recited in class: accuracy — values reflect reality; completeness — no missing mandatory fields; consistency — the same data across systems; validity — matches schema and rules; timeliness — data is up to date. There are more than seven data quality dimensions in total.

What we are trying to do in pre-ingestion and post-ingestion, our primary goal, is automatically checking whether the data is following the data quality dimensions. All these dimensions are important.

Hook — the scorecard behind every check. Every validation check from the last three sections is really measuring one of these dimensions. The range check measures accuracy (a negative sale does not reflect reality); the presence check measures completeness (a missing month is a missing mandatory field); the format check measures validity (schema and rules). The dimensions are the vocabulary that lets you say which quality you are protecting and why the check exists.

The five core dimensions. Each is a measurement of data that can be individually assessed, interpreted, and improved:

Accuracy — values reflect reality (the recorded temperature matches the real temperature).
Completeness — no missing mandatory fields (every required column filled for every record).
Consistency — the same data across systems (a customer's address agrees between CRM and billing).
Validity — matches the schema and rules (dates in YYYY-MM-DD, codes from the allowed list).
Timeliness — data is up to date (the report reflects the latest events, not last month's).

There are more than seven dimensions in total; the five above are the core set the class recites.

Q: What are the data quality dimensions?

A: Accuracy — values reflect reality; completeness — no missing mandatory fields; consistency — the same data across systems; validity — matches schema and rules; timeliness — data is up to date. There are more than seven quality dimensions in total.

8.11.2 The Extended Dimensions

The other dimensions seen earlier in the course: timeliness is very important — without getting the timely data, you may not be able to get valid data. Consistency — this data is consistent. Completeness — the data is complete. Accuracy — the data is accurate. Uniqueness of the data — Raja Kumar versus R. Kumar is the example of duplicate identities. Integrity. Traceability — will I be able to place the data back? Accessibility — can you reach the data? So the quality dimensions include accuracy, completeness, consistency, timeliness, validity, uniqueness, integrity, lineage, reliability, and accessibility.

Data should also be informative enough — enough information that can be used for modeling. Coverage — how much is the coverage, does it give you enough coverage? Completeness and coverage, real inputs — is it really real? Accuracy. Real-time data should be unbiased — data should be unbiased, it should have integrity. We need a feedback loop to ensure that the data follows the correct quality dimension. Good data is not a result of the model itself; it is a problem of the feedback loop. Good data has consistent labels — consistency.

The extended set. Beyond the core five, the course adds:

Uniqueness — no duplicate identities. The lecture's example: "Raja Kumar" versus "R. Kumar" — the same person stored twice, split across two spellings; a uniqueness check catches duplicates that names alone cannot.
Integrity — the data model stays sound; referential relationships hold.
Traceability — will I be able to place the data back? Can every record be traced to its origin and its path (lineage, section 8.7)?
Accessibility — can you reach the data? Authorization, latency, and location all count.
Lineage — the documented journey of the data (section 8.7).
Reliability — the data can be trusted over time.
Coverage — how much of the population or feature space does the data span — does it give you enough coverage for the use case?
Informativeness — enough information to be useful for modeling.

Two deeper points close the set: timeliness first — without timely data you may not be able to get valid data at all, because the world moved on; and bias — data should be unbiased and have integrity, because a biased sample produces a biased model (section 8.14).

The feedback loop. Good data is not a result of the model itself; it is a problem of the feedback loop. The pipeline must feed its validation results back into the process — every rejected record, every consistency failure, every drift signal — so the data continues to follow the correct quality dimensions. Good data has consistent labels: consistency extends even to how the training set is labeled over time.

8.11.3 Benefits of Pre- and Post-Ingestion Validation

There are multiple strategic benefits of doing pre- and post-ingestion validation. First, it prevents the pipeline failures and the alert fatigue. Second, it reduces the data cleansing cost which you have to perform downstream. Third, it builds the trustworthy BI dashboards — which is the end result for us. In class this benefits list was prepared with the help of Microsoft Copilot.

The three benefits. Why do pre- and post-ingestion validation at all?

It prevents pipeline failures and alert fatigue. Bad data that never enters the pipeline cannot crash it downstream — and when failures are rare, alerts stay meaningful instead of being ignored as background noise.
It reduces downstream data cleansing cost. Cleaning is cheaper at the door than after the data has propagated through warehouses, reports, and models. Catch it early, fix it once.
It builds trustworthy BI dashboards — the end result. The dashboards executives read are only as good as the data under them; validation is what makes "the numbers agree with the ledger" true.

The benefits list was prepared in class with the help of Microsoft Copilot, and it answers the "why do this at all" question that case studies love.

Exam note: the techniques are data type checks, range checks, presence checks, format checks, consistency checks — and the benefits above answer "why do this at all". When a scenario asks why validation matters, cite the three benefits: fewer pipeline failures and less alert fatigue, lower downstream cleansing cost, and trustworthy BI dashboards.

8.11.4 Validation Tools in the Industry

The tools which are used for validation: Great Expectations — a data validation tool, an enterprise-scale automation tool, precisely for the semantic validation and the structural validation. SAP pre-validation tool — for the data hygiene. And Definition pre-validation — for the migration readiness of the pipelines.

See the words: like when we talk about hygiene of life, hygiene in your dress, hygiene in your water — we are talking about hygiene for the data as well. Data hygiene leads to master data — that is how they build the master data management.

The named tools. Three tools from the industry toolkit:

Great Expectations — a data validation tool at enterprise scale, automating both semantic validation (does the value mean what it should — an amount field holding amounts) and structural validation (does the record match the schema — columns, types, order).
SAP pre-validation tool — focused on data hygiene: keeping the data clean, the way hygiene keeps a body, a dress, or water clean.
Definition pre-validation — focused on migration readiness: confirming the pipeline can absorb the data before the move.

Hygiene → master data. Data hygiene is the process of keeping data clean; clean data becomes master data — the single, trusted reference for an entity (customer, product, supplier) — and that is how master data management (MDM) is built. The hygiene vocabulary is the same for data as for life: cleanliness of the water, the dress, and the record.

Real-world: Great Expectations (semantic and structural validation at enterprise scale), SAP pre-validation (data hygiene), and Definition pre-validation (migration readiness of pipelines) are the named validation tools.

Recap + bridge. Quality dimensions are the scorecard (accuracy, completeness, consistency, validity, timeliness, plus uniqueness, integrity, traceability, accessibility, lineage, reliability, coverage); validation enforces them; the benefits are fewer failures, lower cleansing cost, and trustworthy dashboards; tools like Great Expectations automate the checks. But validation protects the pipeline — one failure mode it cannot repair is when the data itself is contaminated by design: data leakage, the next topic.

8.12 Data Leakage

8.12.1 Definition and Causes

Data leakage, also called target leakage, is a problem affecting several stages of the machine learning life cycle, from data collection to model evaluation. The definition: data leakage is when information from outside the training data set — like often the test set or future data — improperly influences the model. In supervised learning, data leakage is the unintentional introduction of information about the target that shouldn't be made available; it is also called contamination.

Causes: using the target variable in feature engineering; including future time stamps in training.

Hook — the model that learned the answer key. Imagine studying for an exam with the answer key in the same notebook you train with. Your practice scores are perfect — but the real exam, without the key, is a disaster. That is data leakage: information that will not exist at prediction time (the test set, future data, the target itself) reaches the model during training, so it looks brilliant in evaluation and fails in production.

Definition. Data leakage (target leakage, contamination) is when information from outside the training data set — often the test set or future data — improperly influences the model. In supervised learning it is the unintentional introduction of information about the target that should not have been made available.

The two classic causes:

Using the target variable in feature engineering — a feature built from the very thing you are trying to predict.
Including future time stamps in training — training on data from a time the prediction point has not reached yet.

The professor's plain-language version: think of a pipeline with a water filter — if there is a leakage in the pipeline, water is not coming correctly to your target pipe filter. Information available at that time — and unavailable at the prediction time — when needed; if it is not available at prediction time but was used in training, data has leaked, data is not present. Information unavailable at the prediction time.

An everyday analogy — the LinkedIn profile. You read someone's LinkedIn profile and build assumptions about them from extra, unrelated information. That extra information is a kind of leak: sometimes so much data is good, but some unwanted data, unnecessary data — leaking privacy data — is not. Only the needed data should be there; unwanted data, or data that gives a different meaning, is called data leakage or contaminated data.

Why it is dangerous. Training on contaminated data leads to overly optimistic expectations about the model performance. The model appears excellent on the contaminated training set and evaluation set, but the moment real prediction arrives — without the leaked information — performance collapses. Don't expect the model to do as well in production as it did on your contaminated training set.

Real-world: leakage connects to the model train-test split — why do we set training at 80 percent? The whole goal is avoiding overfitting and underfitting. When leakage happens, you have too much data or too little data, mostly overfitting — unnecessary data is floating. The remedy is proper train-test split enforcement, proper auditing at the source level, proper target variables, and correct timestamps.

8.12.2 Detection and Prevention

The checks for leakage: strict train-test split enforcement; audit feature sources; cross validation with leakage detection.

The prevention routine. Three checks run before a model is trusted:

Strict train-test split enforcement — the split must happen before any preprocessing or feature engineering, so no statistic computed on the full dataset (a mean, a scaling factor) leaks into the training view.
Audit feature sources — trace every feature to its origin (lineage, section 8.7): is any feature derived from the target, or from a timestamp that would not exist at prediction time?
Cross validation with leakage detection — evaluate with cross-validation and inspect for implausible performance: a suspiciously high score is itself a leakage symptom worth investigating.

The remedy list from the lecture: proper train-test split enforcement, proper auditing at the source level, proper target variables, and correct timestamps.

Exam note: data leakage was flagged with a scenario question in the quiz — "data leakage occurs when future or target information contaminates training data." Recognize the scenario form: a description of a pipeline where test data or future data reaches training is always answered with leakage.

8.12.3 Worked Example: Predicting GDP

The target being a function of features: predicting a country's GDP based on area, population, and geographic region. Gross domestic product, GDP, is the monetary measure of all finished goods. If you do not do a proper analysis of each attribute and relation, it may create a problem. If the population column is incorrect — maybe the population data was captured wrongly — then the derived feature, GDP, is wrong, and that error is carried forward as a leakage mark everywhere, creating problems down the line. The fact that GDP is wrong is exactly what is called data leakage:

\[ \text{GDP} = f(\text{area}, \text{population}, \text{geographic region}) \]

The spoken wording in the recording is garbled here — "target being a function of two features, population GDP" — but the feature names given are area, population, and geographic region, so the reconstruction above is the natural one: the target GDP is a function of those three inputs.

Worked example — the wrong population column. A model predicts GDP as a function of area, population, and geographic region: \(\text{GDP} = f(\text{area}, \text{population}, \text{geographic region})\). GDP is the monetary measure of all finished goods produced, and the derived feature most sensitive to error here is population.

Step 1 — The dataset. Two countries: Country A with population recorded correctly at 140,000,000 and Country B with its population captured wrongly — say 1,400,000,000 instead of 140,000,000 (a tenfold transcription error in the source system).

Step 2 — The derived feature. The pipeline engineers a per-capita feature \(\frac{\text{GDP}}{\text{population}}\) (or any population-scaled feature). For Country B the feature is wrong by a factor of ten — and the error propagates into every record and every model that consumes it.

Step 3 — The leakage mark. Because the wrong feature is built from a wrong source column, the contamination travels downstream: training, validation, and future scoring all use the polluted feature. The model's performance numbers look fine — the error is silent — but the predictions for Country B are wrong by construction.

Step 4 — The fix. Audit the feature source (population came from a faulty extract), correct the timestamped extract, re-run validation. The fact that GDP was wrong because its input was wrong is exactly what is called data leakage: error carried forward as a contamination mark everywhere.

Sense-check: one bad column, one derived feature, and the contamination reaches every downstream consumer — which is why the lecture says the error is "carried forward as a leakage mark everywhere, creating problems down the line".

8.12.4 The Paratha and Chutney Analogy

The professor's food analogy: idli with chutney is a good combination of breakfast. Aloo paratha — the paratha with nicely boiled, melted aloo — is also very tasty. But in case the aloo was bad — too much bad, the aloo was not boiled correctly, maybe very hard, maybe salty — it is going to collapse the entire taste of the aloo paratha. The same with idli: the idli is very good, but by mistake the chutney was bad — maybe the coconut was not right — and you poured the chutney without tasting it completely on the idli; the whole idli becomes messy. That is called data leakage, that is data contamination: whenever you build data in a pipeline, you depend on some data; if some dependency goes wrong, the target function creates problems. The whole dish is gone — and not only gone: you would have enjoyed the idli alone with some puri or sambar, but because you put the chutney on, it became useless.

The idli-and-chutney analogy. Idli with chutney is a good combination; aloo paratha with nicely boiled, melted aloo is very tasty. But if the aloo was bad — not boiled correctly, very hard, too salty — it collapses the entire taste of the paratha. The same with idli: the idli is very good, but by mistake the chutney was bad — the coconut was not right — and you poured it without tasting; the whole idli becomes messy. That is data leakage, that is data contamination: whenever you build data in a pipeline, you depend on some data; if some dependency goes wrong, the target function creates problems. The whole dish is gone — and not only gone: you would have enjoyed the idli with puri or sambar, but because you put the chutney on, it became useless. The dependency is the pipeline feature; the ruined dish is the target.

8.12.5 Student Confusion and Clarification

One student was confused by the word "leakage" itself — misheard as "liquor". The clarification: leakage means the data is not correctly available; it is spread out, built over, not properly representing everything. Another student remained confused about the term, and the full re-explanation was given: leakage is like erroneous data — the data that shouldn't be available is made available, and when you train on that contaminated data, it leads to overly optimistic expectations about model performance.

A student then added a further point on the same line: if we have wrong data, data leakage may have happened; another scenario is the test data also getting introduced during the training phase itself — that is also one of the data leakages. The professor agreed completely: yes, you are testing basically on something bad, then it is going to be bad only.

Q: What is meant by "liquor" here — I mean "leakage"?

A: Leakage means the data is not correctly available — it is spread out, built over, not properly representing everything. Wait for the example. In short: data leakage is erroneous data — information that should not be available being made available. Training on that contaminated data leads to overly optimistic expectations about model performance.

The mishearing. The word sounded like "liquor" to one student. It is leakage — and the meaning is about data escaping its proper boundary: data that should not be available is made available, or data that is not correctly representing everything spreads through the pipeline. The term has nothing to do with drink; it is about the leak in the pipe.

Q: I am confused about data leakage. Also, if the test data gets introduced during the training phase, is that leakage too?

A: Yes — leakage is information unavailable at prediction time that still reached the model, like using the target variable in feature engineering or including future timestamps in training. If test data enters training, you are testing on something bad, so results will be bad. Both are contamination, and they cause overly optimistic performance expectations.

The student's contribution confirmed. A student added: if we have wrong data, data leakage may have happened — and another scenario is the test data getting introduced during the training phase itself, which is also one of the data leakages. The professor agreed completely: yes, you are testing basically on something bad, then it is going to be bad only. The point is not just the definition — it is that any information unavailable at prediction time that still reached the model is contamination, whether it is the target, the test set, or future data.

Recap + bridge. Leakage = information unavailable at prediction time that still reached the model (target in features, future timestamps, test data in training). It causes over-optimistic expectations and failure in production; prevention is the train-test split, feature-source audits, and leakage-aware cross-validation. The idli-and-chutney story is the exam memory hook. Next: the distribution problems that validation and leakage both interact with — skewness and data imbalance.

8.13 Skewness and Data Distribution Issues

8.13.1 Positive and Negative Skew

Skewness is asymmetry in the data distribution. Positive skew means a long right tail; negative skew means a long left tail. The distribution tail is longer on the right-hand side — that is positive skewness. The professor's phrasing: is the data towards this side or that side — the median versus the mean — left-hand side or right-hand side; we need to find the imbalance in the data.

Hook — why a "balanced-looking" column can still be lopsided. A column of incomes where most people earn 30–50 thousand and a few earn millions has a mean dragged far right of the median. The data is skewed — asymmetric — and every mean-based calculation downstream inherits that drag. Spotting the direction of the tail is the first step to fixing it.

The formal measure is the skewness coefficient:

\[ \gamma_1 = \frac{1}{n}\sum_{i=1}^{n}\left(\frac{x_i - \bar{x}}{s}\right)^3 \]

The lecture describes skewness verbally — asymmetry in the distribution, detected with statistical tests, histograms, and box plots — and the formula above is the standard moment-based definition reconstructed from that description. Here \(\gamma_1\) is the skewness coefficient, \(x_i\) is the \(i\)-th observed value, \(\bar{x}\) is the sample mean, \(s\) is the sample standard deviation, and \(n\) is the number of observations.

Reading the coefficient. The formula is the average of the standardized cubed deviations: each value \(x_i\) is compared to the mean \(\bar{x}\), divided by the standard deviation \(s\) (so the deviations are dimensionless — measured in standard units), cubed (so the sign survives but big deviations are amplified), and averaged over all \(n\) observations.

\(\gamma_1 = 0\) — a symmetric distribution; the mean and median coincide.
\(\gamma_1 > 0\) — positive skew: a long right tail; the mean sits to the right of the median.
\(\gamma_1 < 0\) — negative skew: a long left tail; the mean sits to the left of the median.

Why the cube? Squaring would lose the direction of the tail (everything would come out positive); the cube keeps the sign while giving more weight to the distant outliers that define the tail.

Worked example — computing skewness on five incomes. Incomes in thousands: \(10, 12, 14, 15, 100\).

Step 1 — The mean. \(\bar{x} = \frac{10+12+14+15+100}{5} = \frac{151}{5} = 30.2\).

Step 2 — The standard deviation. Deviations from the mean: \(-20.2, -18.2, -16.2, -15.2, 69.8\). Squared: \(408.04 + 331.24 + 262.44 + 231.04 + 4872.04 = 6104.8\). Sample variance \(\frac{6104.8}{4} = 1526.2\), so \(s \approx 39.07\).

Step 3 — The standardized cubed deviations.

\[ \begin{aligned} &\left(\frac{10 - 30.2}{39.07}\right)^3 \approx (-0.517)^3 \approx -0.138\\ &\left(\frac{12 - 30.2}{39.07}\right)^3 \approx (-0.466)^3 \approx -0.101\\ &\left(\frac{14 - 30.2}{39.07}\right)^3 \approx (-0.415)^3 \approx -0.071\\ &\left(\frac{15 - 30.2}{39.07}\right)^3 \approx (-0.389)^3 \approx -0.059\\ &\left(\frac{100 - 30.2}{39.07}\right)^3 \approx (1.787)^3 \approx 5.704 \end{aligned} \]

Step 4 — Average. \(\gamma_1 = \frac{-0.138 - 0.101 - 0.071 - 0.059 + 5.704}{5} \approx \frac{5.335}{5} \approx 1.07\).

Sense-check: \(\gamma_1 \approx 1.07 > 0\) — a positive value with a long right tail, which matches the data: four modest incomes and one huge outlier dragging the tail right. Exactly what "positive skew" means.

8.13.2 Detecting Skewness

Detection methods: statistical tests and the skewness coefficient; visualization — draw histograms and box plots. Post-midterm there is a plan to do a proper statistical analysis — box plot distributions, five-point summary (minimum, first quartile, median, third quartile, maximum) — all those things will help you understand the data, because many students may have seen them in an introduction to data science but benefit from a refresher.

The box plot as a skew meter. A box plot draws the five-point summary — minimum, first quartile, median, third quartile, maximum — as a box with whiskers. In a symmetric distribution the median sits in the middle of the box and the whiskers balance. In positive skew, the median is pushed to the left of the box, the box is short on the left, and the right whisker stretches far (and outliers beyond 1.5 × the interquartile range appear as dots). A histogram shows the same story: the bulk of bars on the left with a long flat tail to the right.

8.13.3 Impact and Fixes

The impact of skewness: it biases mean-based models. Skewed data requires normalization or transformation — for example, log transform or Box-Cox transform — before modeling. So you do a lot of normalization and standardization techniques to handle the skewness of data.

Scope — when skew hurts and when it does not. Skew biases mean-based models — linear regression, logistic regression, anything that computes an arithmetic mean or a Gaussian likelihood. Tree-based models (decision trees, random forests) split on thresholds and are largely insensitive to monotone skew. And some targets are supposed to be skewed (income, house prices, network delays): the fix is to transform the feature, not to pretend the world is symmetric.

Assumption that fails: the normal assumption. Many statistical tools assume the data is symmetric (bell-shaped); with skew, confidence intervals and p-values computed under that assumption are wrong, because the tail drags the mean away from the typical value.

Worked example — the log transform fixing the tail. Revisit the incomes \(10, 12, 14, 15, 100\) (thousands).

Step 1 — Compute the base-10 logs: \(\log_{10}(10) = 1.000\), \(\log_{10}(12) \approx 1.079\), \(\log_{10}(14) \approx 1.146\), \(\log_{10}(15) \approx 1.176\), \(\log_{10}(100) = 2.000\).

Step 2 — The transformed set: \(1.000, 1.079, 1.146, 1.176, 2.000\). Mean \(\approx 1.28\); the gap between the largest value (2.000) and the rest is now a fraction, not a factor of ten.

Step 3 — Compare the skew. Before the transform \(\gamma_1 \approx 1.07\) (long right tail). After the log transform, the outlier 100 → 2 sits close to its neighbors, so the distribution is far more symmetric — a skewness that is now near zero.

Sense-check: the same five numbers, one monotone transform, and the tail stops dominating the mean — the reason log and Box-Cox transforms are the standard fixes for positive skew.

Q: What is skewness?

A: Skewness is asymmetry in the data distribution. Positive skew means a long right tail; negative skew means a long left tail. You detect it with statistical tests, the skewness coefficient, histograms, and box plots. It biases mean-based models, so you normalize or transform the data, for example with log or Box-Cox.

Recap + bridge. Skewness = asymmetry, measured by \(\gamma_1\) (the average of standardized cubed deviations), read off histograms and box plots, and fixed with log or Box-Cox transforms before mean-based modeling. Skew is one of the common data problems; the next section surveys the full list — cost, noise, outliers, bias — because recognizing "what is wrong with this data" is itself an exam skill.

8.14 Common Data Problems and Types of Bias

8.14.1 Costly, Noisy, Low-Predictive Data

The common problems with data: data is so costly — getting balanced, high-quality data can be expensive. Data is noisy — a little noise, a little error, and instrument error can cause a lot of problems; images can be blurry, incomplete. Data can have low predictive power. Data may be outdated.

Hook — the four costs of real data. Every dataset you touch in industry pays one of four prices: cost (balanced, high-quality data is expensive to collect and label), noise (a little instrument error can corrupt a lot of downstream signal — a blurry image, an incomplete record), low predictive power (the features simply do not separate the classes), and staleness (data that described the world yesterday no longer describes it today — the doorway to drift, section 8.15). Recognizing which price you are paying is the first step of data engineering.

8.14.2 Outliers

Outliers are data points that come under the other than the lower quartile or above the upper quartile — data that is not around the mean but outside the interquartile range. Outliers are one of the common data problems.

Outliers — the interquartile-range definition. An outlier is a point outside the central block of the data: below the lower quartile or above the upper quartile — more precisely, outside the interquartile range (IQR), the span between the first quartile \(Q_1\) and the third quartile \(Q_3\). A common rule of thumb flags a value as an outlier when it is more than \(1.5 \times \text{IQR}\) beyond a quartile — that is the fence your box plot draws.

Worked example — finding outliers in a column. Sales figures: \(10, 12, 14, 15, 100\).

Step 1 — Sort and find quartiles. Sorted: \(10, 12, 14, 15, 100\). \(Q_1 = 12\), median \(= 14\), \(Q_3 = 15\).

Step 2 — The interquartile range. \(\text{IQR} = Q_3 - Q_1 = 15 - 12 = 3\).

Step 3 — The fences. Lower fence: \(Q_1 - 1.5 \times \text{IQR} = 12 - 4.5 = 7.5\). Upper fence: \(Q_3 + 1.5 \times \text{IQR} = 15 + 4.5 = 19.5\).

Step 4 — The verdict. The value 100 is above the upper fence of 19.5 → outlier. The values 10–15 all sit inside the fences → not outliers. Note that 100 is also exactly the value that drove the positive skew in section 8.13.

Sense-check: one rule, one fence computation, and the extreme value is flagged objectively — the box plot shows it as a dot beyond the whisker.

Scope — outliers are findings, not always errors. An outlier can be a data-entry error (a stray decimal point) — or a genuine extreme event (a store's one-day record sale). Never auto-delete outliers: investigate first (is it real?), then decide between keeping, winsorizing, or removing based on what the data is for.

8.14.3 The Many Kinds of Bias

Data is so much bias-prone that all AI should follow fairness, accountability, and trustworthiness. The different types of bias listed: omitted variable bias; sponsorship or funding bias — some research papers are biased as well; some experiments are biased; label bias — when they label it a certain way; selection bias — why they selected some students, when I am selecting the sample I may not select it correctly; implicit bias — something will have their own thing; stereotype bias — they always think these people are good, this system is a reliable system — if a big industrial company is giving the data, which is correct — those are all bias; experimental bias; reporting bias — reporting by us; collection bias — data is not selected in a representative fashion. Labeling by us: they may assign the digit as six or nine — a classic ambiguity. These are the biases to look for when assessing data.

Fairness, accountability, trustworthiness. Data is so bias-prone that all AI should follow three principles: fairness (decisions do not systematically disadvantage a group), accountability (someone answers for the model's behavior), and trustworthiness (the model's outputs can be relied on). Bias anywhere in the data chain — collection, labeling, sampling — undermines all three.

The bias catalogue. Know the list and one example for each — the exam asks "what is wrong with this data":

Bias	What it is	One example
Omitted variable	A relevant cause is left out, so the model assigns its effect to other features	Predicting credit risk without income; income's effect lands on location
Sponsorship / funding	Research or experiments tilted by who paid for them	A sponsored study reports only favorable results
Label bias	Labels assigned a certain way by the labeller	The digit "6" versus "9" — rotated handwriting that either labeller resolves differently
Selection bias	The sample is not representative because only some people are included	A study that calls on only the same few students — the sample excludes the rest
Implicit bias	Unstated assumptions in the people building the data	A designer's assumptions about users leak into feature choices
Stereotype bias	Beliefs about groups applied to individuals	"A big industrial company gives correct data" — trust by reputation, not by check
Experimental bias	The experiment's design favors one outcome	A test group chosen because it is easy to reach, not because it is representative
Reporting bias	What gets reported is not what happened	Only successful projects enter the dataset; failures vanish
Collection bias	Data is not selected in a representative fashion	Sensor readings only from urban areas; rural behavior missing

The lecture's in-class example of selection bias is worth remembering: when a teacher keeps calling on the same few students, it might look like bias — but the intent is the opposite, wanting other students to speak too. Selection bias in a real study means the sample is not representative because only some people are included.

Exam note: bias is a recurring theme; know the list of bias types and one example for each, because data quality questions often test "what is wrong with this data." The answer pattern: name the bias, define it in one line, and give the example from the table — selection, label, reporting, collection, sponsorship, stereotype, implicit, experimental, omitted variable.

8.14.4 Missing Features and Unexpected Values

Missing features: how are you going to deal with them? Unexpected feature values: whenever you pinpoint unexpected feature values, how you handle them. These are practical data engineering problems that show up in validation design.

Handling missing features and unexpected values. Missing features have a menu: drop the record, drop the feature, impute a statistic (mean, median, mode), or build a model to predict the missing value — each with its own bias risk (dropping rows with missing income silently removes the poor, skewing the sample). Unexpected values — a category that was never in the allowed list, a date in the wrong era — should be rejected or flagged by the validation rules you designed in section 8.8, then investigated: an unexpected value is often the first visible symptom of an upstream change. These are not trivia; they are the decisions written into the validation checks.

8.15 Model Drift and Training-Serving Skew

8.15.1 Why Production Models Decay

Once you build a model and deploy it in production, the model usually performs well for some time. Then, sometimes you need to do this tuning again — model quality decays. The professor's vehicle analogy: your vehicle — you depend on the vehicle, you put petrol, and after two years, whatever data you have, you may be outdated. You may need to do something else to your bike — a new tyre, probably replace a tyre or change the tyre pressure for some roads. Going with the same data may lead into performance problems. The data becomes outdated.

Machine learning in production is quite different from machine learning in a competition. The biggest challenge is the dataset: sometimes you give a dataset which is always static, sometimes the datasets are all dynamic. You need to choose the speed of decay — like our teeth, how they decay. Data will decay. Most model decay is drift. ML models in production can experience reduced performance over time — you may say something, but the model may do something else.

Hook — the model that was right yesterday. The model performed well at launch. Months later, without any code change, its predictions drift off: you say one thing, but the model does something else. Nothing in the model changed — the data changed. That is why machine learning in production is different from machine learning in a competition: in a competition the dataset is fixed and static; in production the datasets are all dynamic.

The professor's vehicle analogy. You depend on your vehicle; you put petrol; but after two years, whatever data you have, you may be outdated. You may need a new tyre, or to change the tyre pressure for some roads. Going with the same data leads to performance problems — the data becomes outdated. Choose the speed of decay — like our teeth, how they decay. Data will decay; most model decay is drift.

Q: Why does model performance decay in production?

A: Because of data drift or concept drift. In data drift, the production data a model receives for scoring has diverged from the data used to train, tune, and evaluate it — the discrepancy is usually schema skew or distribution skew. Concept drift means the interpretation or labels change. You need retraining, retuning, and re-modeling again and again.

8.15.2 Data Drift and Schema Skew

Data drift: a skew grows between training data and serving data. In data drift, the production data that a model receives for scoring has diverged from the dataset that was used to train, tune, and evaluate the model. The discrepancies between training and serving data can usually be classified as schema skew or distribution skew.

Schema skew occurs when training data and serving data don't conform to the same schema. You built some schema — a data warehouse schema — but it is not satisfying your needs; there is a deviation between the training and serving data, an inconsistent feature; your model has to be updated.

Data drift, schema skew. Data drift is the general name: the production data a model receives for scoring has diverged from the dataset used to train, tune, and evaluate it. The discrepancies are usually one of two kinds:

Schema skew — training and serving data do not conform to the same schema. The feature list changed shape: a new field was added upstream, a field renamed, a value type flipped from string to integer. You built a schema (a warehouse schema) that no longer satisfies the needs; there is a deviation between training and serving data, an inconsistent feature — and the model must be updated to match.
Distribution skew — the schema matches, but the values have moved (section 8.15.3).

8.15.3 Distribution Skew and Concept Drift

Distribution skew: the data is not on the right distribution — we didn't have that. There are new trends and patterns that have come; we don't capture the changes — for example, the price of real estate. You did some code five years ago, three years ago, and now you are coming with the same logic; it does not work, because they may have added something else for the real estate predictions.

Concept drift: your interpretation may change. Day one you had some understanding; after several more contact sessions you may have a different understanding. The mapping — how you are doing a mapping based on your knowledge, based on the concept — there is some concept called the concept hierarchy. We all have some understanding of a concept, of a hierarchy, a parent-child hierarchy — this is how it is. Based on that hierarchy you work on it. Sometimes there will be a drift — what we understand about something may be right, may not be right. That is where concept drift comes. Labels change: one may label a group one way today and another way later — that is concept drift; the concept hierarchy goes out of the box.

Distribution skew, concept drift. Two different failures, both called "drift":

Distribution skew — the data is not on the right distribution; new trends and patterns have come and we do not capture the changes. The real-estate example: you wrote code five years ago, three years ago, and now you run the same logic on current prices; it does not work, because the market added something else to the price structure. The schema is identical; the values moved.
Concept drift — your interpretation changes. The mapping between input and target shifts even though the input looks the same. Day one you had one understanding; after several more sessions you have a different one. We all carry a concept hierarchy — a parent-child understanding of categories — and we work from it; drift happens when what we understand may be right or may not be right. Labels change: one may label a group one way today and another way later — that is concept drift; the concept hierarchy goes out of the box.

Real-world: concept drift example — a model trained on older conventions for how people are categorized stops matching new usage; the mapping between input and target changes even though the input looks the same.

	Schema skew	Distribution skew	Concept drift
What changed	The schema (fields, types)	The input values' distribution	The input-to-target meaning
Detection	Schema comparison	Histogram/PSI (section 8.16)	Label/outcome analysis
Example	A field renamed upstream	Real-estate prices moved	"Luxury" now means a different price band

8.15.4 Training-Serving Skew and How to Avoid It

Training-serving skew: there could be a discrepancy between how data is handled in the training pipeline and the serving pipeline. Whatever is deployed, when it goes for real time it may be different — maybe the queue parameter needs to be tuned, the model needs to be retrained. Avoid training-serving skew. Ideally, data engineers should reuse the same feature engineering code — whatever feature engineering code you used before, you need to retrain again and again, use the same feature vector during training and deployment serving. Same thing with the computation resources — sometimes the computation resources may have a problem, how we use randomness.

The professor's mental model: this is the original data — this is how it has to be; but in the training and testing it is what it is. That gap between expectation and reality is the drift problem.

Training-serving skew — the same code must serve both sides. The discrepancy between how data is handled in the training pipeline and how it is handled in the serving pipeline. Whatever is deployed, when it goes real-time it may be different: a queue parameter needs tuning, the model needs retraining. The prevention rules:

Reuse the same feature engineering code — whatever feature engineering code you used before, retrain again and again with it; do not hand-write the serving features a second time.
Use the same feature vector during training and deployment serving — one vector definition, one code path.
Control the environment differences — computation resources may differ between training and serving, and so does how randomness is used (seeds, sampling): those differences must be made explicit, not accidental.

The professor's mental model: this is the original data — this is how it has to be; but in the training and testing it is what it is. The gap between expectation and reality is the drift problem.

Recap + bridge. Production models decay because data decays: data drift (schema skew or distribution skew) and concept drift (labels and interpretation change), plus training-serving skew when the two pipelines diverge. The remedies are retraining, retuning, re-modeling — and reusing the same feature engineering code in both pipelines. But "the model drifted" needs a number: the next section gives the metrics that measure drift — PSI, KL divergence, and entropy.

8.16 Drift Detection: PSI, KL Divergence, and Entropy

8.16.1 Population Stability Index (PSI)

There are metrics available to detect drift — model-based approach, statistical approach, algorithm-based approach — and one of the first is the population stability index, PSI. The definition read in class: the population stability index is a statistical tool used to measure how much the distribution of data changes between two datasets, usually between a training dataset and a new dataset. A high PSI score indicates a significant change in population distribution, while a low PSI score indicates stability. Importance — model stability: it helps check whether a model trained on historical data remains valid when applied to new data; it detects data drifts; a high PSI score can indicate that the new data distribution has drifted significantly from the past, which may require recalibrating the model.

Garbage in, garbage out — this is the message repeated when discussing drift: when you have a training dataset and a new dataset, and you run any machine learning model with training data and test data, then when new data comes you must ask what sort of distribution the training had — low to high, medium to high — and compare it with the new data. Are we correct or wrong? Do I have it right? PSI is the tool for that.

The standard PSI formula:

\[ \text{PSI} = \sum_{i=1}^{n} (A_i - E_i) \times \ln\left(\frac{A_i}{E_i}\right) \]

The lecture describes PSI verbally — measuring how much the distribution of data changes between two datasets, often via histograms — without writing the formula; the form above is the standard reconstruction. Here \(A_i\) is the actual share of observations in bin \(i\) of the new dataset, \(E_i\) is the expected share in bin \(i\) of the training dataset, and \(n\) is the number of bins. PSI essentially does some histogram comparison between the two datasets.

Hook — the model is healthy if the population still looks like training. A credit model was trained on last year's applicants. This year's applicants arrive. Before trusting the scores, ask: does the new population still look like the training population? PSI answers with one number — and "garbage in, garbage out" is the message to remember: if the new data's distribution has drifted away from training's, the model's outputs on that data are garbage until it is recalibrated.

Reading the PSI formula. Bucket both datasets into the same \(n\) histogram bins (by value ranges or by score bands). For each bin \(i\):

\(E_i\) — expected share of observations in bin \(i\) (from the training dataset);
\(A_i\) — actual share in bin \(i\) (from the new dataset);
\((A_i - E_i)\) — how much the share moved;
\(\ln(A_i/E_i)\) — the relative change, amplified by the log.

Every term is non-negative: if the share grew (\(A_i > E_i\)) both factors are positive; if it shrank (\(A_i < E_i\)) both are negative. Sum over bins and you get the drift score. If the two datasets are identical, \(A_i = E_i\) for every bin and \(\text{PSI} = 0\) — perfect stability.

The important part: a high PSI score indicates a significant change in population distribution, which may require recalibrating the model; a low score indicates stability — the model trained on historical data remains valid.

Related explainable-AI tools. SHAP (Shapley) and LIME are two tools that do similar comparisons — they are based on different permutations and combinations, using different metrics of computation and probability to find out the distance between every single data point. They answer a different question (why did this prediction happen?) with the same underlying instinct: compare what changed between data states.

Q: What is the Population Stability Index (PSI)?

A: PSI is a statistical tool that measures how much the distribution of data changes between two datasets — usually a training dataset and a new dataset. A high PSI score indicates a significant change in population distribution, while a low PSI score indicates stability. It checks whether a model trained on historical data remains valid on new data and detects data drift; a high score means recalibration may be needed.

8.16.2 PSI Thresholds and a Worked Example

The threshold board: if the PSI is greater than 0.25, that is significant drift; 0.1 is moderate drift; less than 0.1 means no significant drift.

\[ \text{PSI} > 0.25 \text{ significant drift}, \quad 0.1 < \text{PSI} \leq 0.25 \text{ moderate drift}, \quad \text{PSI} \leq 0.1 \text{ no significant drift} \]

The worked example: training data normally distributed — meaning around 50 validations done — and test data drifted distribution around mean 55. So the training distribution sits around a mean of 50, the new data sits around a mean of 55, and the PSI computation over the histograms tells you whether that shift crosses the threshold. You can try this with training data and test data to understand: run a program right away with some training data and a new dataset — you can get a PSI program from a coding assistant like ChatGPT or Copilot, which will give you the code and a dataset to experiment with. That is how you do it; some code can be shared later.

Worked example — does a mean shift of 5 cross the threshold? Training data is normally distributed with mean 50; the new dataset has drifted to mean 55 (both with standard deviation 10 — the lecture's numbers). Bucket both into five bins: ≤40, 40–50, 50–60, 60–70, >70.

Step 1 — Expected shares \(E_i\) from training (mean 50): using the standard normal table: bin1 ≈ 0.159, bin2 ≈ 0.341, bin3 ≈ 0.341, bin4 ≈ 0.136, bin5 ≈ 0.023.

Step 2 — Actual shares \(A_i\) from the new data (mean 55): bin1 ≈ 0.067, bin2 ≈ 0.242, bin3 ≈ 0.383, bin4 ≈ 0.242, bin5 ≈ 0.067. (The whole histogram slid right: more mass in bins 3–5, less in bins 1–2.)

Step 3 — Per-bin PSI terms \((A_i - E_i)\ln(A_i/E_i)\):

\[ \begin{aligned} &\text{bin 1: } (0.067 - 0.159)\ln\left(\tfrac{0.067}{0.159}\right) = (-0.092)(-0.864) \approx 0.079\\ &\text{bin 2: } (0.242 - 0.341)\ln\left(\tfrac{0.242}{0.341}\right) = (-0.099)(-0.343) \approx 0.034\\ &\text{bin 3: } (0.383 - 0.341)\ln\left(\tfrac{0.383}{0.341}\right) = (0.042)(0.116) \approx 0.005\\ &\text{bin 4: } (0.242 - 0.136)\ln\left(\tfrac{0.242}{0.136}\right) = (0.106)(0.576) \approx 0.061\\ &\text{bin 5: } (0.067 - 0.023)\ln\left(\tfrac{0.067}{0.023}\right) = (0.044)(1.069) \approx 0.047 \end{aligned} \]

Step 4 — Sum. \(\text{PSI} \approx 0.079 + 0.034 + 0.005 + 0.061 + 0.047 \approx 0.23\).

Step 5 — Verdict against the threshold board. \(0.1 < 0.23 \leq 0.25\) → moderate drift. The population has moved enough to matter, not enough to panic — watch it and plan recalibration.

Sense-check: a mean shift of just 5 units out of a standard deviation of 10 produces PSI ≈ 0.23, squarely in the moderate band — exactly what the lecture's threshold board predicts, and a concrete reason why the mean-50/mean-55 example was chosen.

Exam note — the threshold board: \(\text{PSI} > 0.25\) = significant drift; \(0.1 < \text{PSI} \leq 0.25\) = moderate drift; \(\text{PSI} \leq 0.1\) = no significant drift. Questions give you a scenario and a PSI value (or a distribution shift) — place it on the board and state what to do (monitor, recalibrate).

8.16.3 Real-World: Credit Scoring and Customer Segmentation

PSI is used for scoring — for credit scoring, customer segmentation, risk assessment. The professor's credit scoring example: you want to give a loan, similar to a credit score like the CIBIL score. You used some data, created your machine learning code, finished it, calculated the credit score. Now new data comes — when you calculate that new data, you try to find out whether it is correct: will it give a good credit score or not? The credit score program may work, but the data distribution of how it has handled the model's previous data and this data — there may be a difference. PSI helps make sure the credit scoring model still works well for the current population, especially after big changes in the economy or when new rules come. If you find that population distance, you know how helpful this is.

Q: If a credit scoring model keeps predicting wrong, what metric helps?

A: PSI. For credit scoring, customer segmentation, or risk assessment, you compare the distribution of the new data against the training data. If the population has shifted — after big economic changes or new rules — PSI tells you the scoring model no longer fits the current population, and you should recalibrate.

Real-world: in the exam, a scenario may say "we have customer segmentation or credit scoring; the predictions were always wrong — what metric do you use?" The answer: PSI.

Scope — PSI measures the population, not the model. PSI compares input distributions; it tells you the population drifted, not why the model fails on it. Combine it with the drift taxonomy from section 8.15 (schema skew, distribution skew, concept drift) to decide what to do: re-bin, retrain, or re-define the labels.

8.16.4 KL Divergence and Relative Entropy

There is another metric called KL divergence — the Kullback-Leibler divergence. It is a relative entropy. The standard form:

\[ D_{KL}(P \parallel Q) = \sum_{x} P(x) \log\left(\frac{P(x)}{Q(x)}\right) \]

The lecture only names Kullback-Leibler divergence as "relative entropy"; the formula above is the standard reconstruction. Here \(P\) is the true or reference probability distribution, \(Q\) is the approximating distribution, and the sum runs over all values \(x\) the variable can take. It measures how much one probability distribution diverges from another — how much information is lost when \(Q\) is used to approximate \(P\).

Reading KL divergence. For every value \(x\), take the reference probability \(P(x)\), weight the ratio \(\log(P(x)/Q(x))\) by it, and sum. The result answers: if the real distribution is \(P\) but the model believes \(Q\), how much information is lost?

\(D_{KL}(P \parallel Q) \geq 0\), and it is exactly 0 only when \(P = Q\) everywhere.
It is asymmetric: \(D_{KL}(P \parallel Q) \neq D_{KL}(Q \parallel P)\). The direction matters — asking "how wrong is my model's assumption about the world" differs from "how wrong is the world according to my model". That asymmetry is what lets KL detect distribution order switching where a symmetric metric could not.
Relation to PSI: both are weighted log-ratio sums over histogram bins — PSI is a symmetric variant of the same idea; KL is the information-theoretic parent, and entropy (next) is its building block.

8.16.5 Entropy, Information Gain, and Gini Index

What is entropy? The degree of disorder — a measure of impurity. A set with apples and oranges: if there is only one apple and many oranges, that is near-pure; a mixed set shows high entropy. In data science you studied entropy, information gain, and the Gini index. Information gain and Gini index both measure impurity — the exact measure of impurity.

Entropy in information theory:

\[ H(X) = -\sum_{x} p(x) \log_2 p(x) \]

The lecture describes entropy as "degree of disorder" and uses an apple-orange impurity example without writing the formula; this is the standard Shannon entropy. Here \(X\) is a discrete random variable, \(p(x)\) is the probability of each value \(x\), and \(H(X)\) is the expected information content — the disorder — in bits.

The Gini index, which also measures impurity:

\[ G = 1 - \sum_{k} p_k^2 \]

The lecture names the Gini index as a measure of impurity without stating the formula; this is the standard form. Here \(p_k\) is the proportion of observations belonging to class \(k\). A pure set has \(G = 0\); the most mixed set has the highest Gini.

Intuition — the apples and oranges. A bowl with one apple and nine oranges is near-pure: almost everything is an orange, so there is little surprise in predicting "orange". A bowl with five apples and five oranges is maximally mixed: predicting the next fruit is a coin flip — high disorder. Entropy quantifies exactly this: it is the expected surprise, in bits.

Worked example — entropy and Gini on the same bowl. Bowl A: 1 apple, 9 oranges. Bowl B: 5 apples, 5 oranges.

Step 1 — Probabilities. Bowl A: \(p(\text{apple}) = 0.1\), \(p(\text{orange}) = 0.9\). Bowl B: \(p(\text{apple}) = 0.5\), \(p(\text{orange}) = 0.5\).

Step 2 — Entropy (base 2):

\[ \begin{aligned} H_A &= -[0.1\log_2 0.1 + 0.9\log_2 0.9] \approx -[0.1(-3.322) + 0.9(-0.152)] \approx 0.469 \text{ bits}\\ H_B &= -[0.5\log_2 0.5 + 0.5\log_2 0.5] = -[0.5(-1) + 0.5(-1)] = 1.000 \text{ bit} \end{aligned} \]

Step 3 — Gini:

\[ \begin{aligned} G_A &= 1 - (0.1^2 + 0.9^2) = 1 - 0.82 = 0.18\\ G_B &= 1 - (0.5^2 + 0.5^2) = 1 - 0.50 = 0.50 \end{aligned} \]

Step 4 — Read the numbers. Both measures rank the bowls the same way: Bowl A (near-pure) is low — \(H_A \approx 0.47\) bits, \(G_A = 0.18\); Bowl B (mixed) is high — \(H_B = 1\) bit, \(G_B = 0.50\). Information gain is the reduction in entropy you get from a split; Gini is the alternative impurity score decision trees use.

Sense-check: a pure set scores zero on both (1.0/0.0 fruit → \(H = 0\), \(G = 0\)), a maximally mixed set scores the maximum — the two measures agree on ordering, which is why either works for tree splits.

Q: What is entropy?

A: Entropy is the degree of disorder — a measure of impurity. A set with one apple and many oranges has low entropy; a mixed set shows high entropy. Entropy underlies information gain and the Gini index, which also measure impurity.

Recap + bridge. Drift detection has a metric family: PSI (bucketed, with the 0.25/0.1 threshold board and the credit-scoring scenario), KL divergence (asymmetric relative entropy), and the impurity measures entropy and Gini behind every tree split. They all answer "how different are these two distributions" — the numeric backbone of the drift story from section 8.15. Next, the discipline that discovers these problems before modeling: data profiling.

8.17 Data Profiling

8.17.1 What Data Profiling Is

Data profiling means taking the data and understanding what it is — what is the metadata. It is basically like customer profiling: the data profile is a profile about the data. If you describe a person's profile — the CV talks about education, whereabouts, skills, certifications — similarly, when you talk about data profiling, you talk about all the characteristics which represent the data.

Hook — the CV for your data. A CV tells you about a person without meeting them: education, whereabouts, skills, certifications. Data profiling builds the same document for a dataset — before you model it, you read its profile so you know what you are working with. It is the "understand before you use" step: the metadata, the summary statistics, and the relationships inside the data.

CV element	Data profile equivalent
Education	Data types and schema
Whereabouts	Origins, lineage, and sources (section 8.7)
Skills	Value ranges and distributions
Certifications	Key dependencies and relationships between fields

Real-world: profiling is done before deep modeling to understand a dataset — the metadata, the summary statistics, and the relationships inside the data. A fuller expansion of data profiling is planned for the next contact session after the midterm; what has been covered up to this point is enough preparation.

8.17.2 What a Data Profile Contains

A data profile captures: the summary statistics; what data types it has; what values; what distributions; and what the dependencies are in keys — multiple fields that are dependent on each other. That is basically data profiling.

The five parts of a data profile. A profile answers five questions about the dataset:

Summary statistics — means, medians, standard deviations, quartiles: the five-point summary (minimum, first quartile, median, third quartile, maximum) and the skewness story from section 8.13.
Data types — numeric, categorical, datetime, text: what each field is declared as, and whether the values agree with the declaration.
Values — the distinct values a field takes, their frequencies, and the gaps (missing values, unexpected values from section 8.14).
Distributions — histograms and box plots showing shape, tails, and balance (the statistical analysis planned after the midterm).
Key dependencies — relationships between fields: keys that identify records, and multiple fields that depend on each other (the candidate for referential integrity checks from section 8.9).

That is basically data profiling: a profile about the data, so the engineer and the model both know what they are dealing with.

Exam note: data profiling covered up to this point is enough preparation for the midterm; the deeper expansion (box plots, five-point summaries, statistical analysis) comes after the midterm. In an exam answer, profile = metadata + summary statistics + values + distributions + key dependencies.

8.18 Teaching Stories: Attitude, Prevention, and Data Concepts

8.18.1 The Guru and the Sunflower Seeds

This is the motivational story of the contact session, told before the main topic. There was a very famous guru who lived in the Himalayas and knew everything — astronomy, martial arts, physics, chemistry, nature. Two young men walked from South India (in those days there was no train or bus). The guru said his seats were full — he only taught 15 students per batch, and they were extra baggage; maybe next year. The boys begged, promising to do whatever he said. He gave them a test: he went to his hut and brought some seats, and gave one to each. You go east, you go west, he told them; you should not meet each other. Today is the 13th of the sixth month; you come back in July, one month. Don't waste half a year: do this test, grow something, come back with the results. That is all he said.

One month later, the first boy came back with a dull head, carrying only one sunflower, complaining: "I thought you were a great guru, and you do a lot of good exercises — you could have given me hunting, bow and arrow; instead you gave me lousy work of growing plants. Some of the seeds were dropped on the way, some were boiled seeds, corrupt seeds, bad seeds; there is no proper soil here; every day I had to go two miles to get water because the surface is upper — I had to go down and climb back. With all difficulty I got this one sunflower."

A couple of hours later the second boy came like a harvest celebration — one basket in the right hand, one basket in the left hand, one basket on his head — fruits, vegetables, flowers. He said: "Guruji, I loved your assignment. I carefully took your seeds, used my towel, and carefully analyzed which seed is good and which is boiled; I segregated it. You can apply data warehousing, you can apply ETL concepts here — extraction, transformation, loading. I did a lot of transformation — which soil is good for what. You can't eat all the dishes together and put them in one stomach; it is highly impossible. The same thing for plants: you can't put all the trees together in the same soil — they look for different ingredients. So I was looking for the right soil. That is clustering — you study classification and clustering; that person applied the data mining techniques of classification, clustering, and data cleansing — everything. I identified the soils. I was finding difficulty pouring the water, so I started digging a small well near the plant, and I got water. I put a little hut. Every day I was seeing the plant, its growth. I was appreciating nature; I was loving what I was doing."

That is the point of the story in one line: attitude always matters. The class was asked to type the characteristic each boy exhibited.

Q: What attitude was each boy exhibiting?

A: The second boy showed a positive attitude — patient, analytical, systematic, appreciative: he segregated the good seeds from the boiled ones, chose the right soil, dug a well, and watched the plant grow. The first boy showed a negative attitude — complaining about boiled seeds, bad soil, and the two-mile walk for water. The second boy, with the three baskets, was selected; he learned with the guru for many years and became very successful.

The story as a data lesson. Growing the sunflower is an ETL job, and the second boy is a data engineer:

Extraction — collecting the seeds and bringing them home (getting data from the source).
Transformation — segregating the good seeds from the boiled ones, analyzing which soil suits which plant ("you can't put all the trees together in the same soil — they look for different ingredients"): cleaning the seed set is data cleansing; choosing the right soil per plant is clustering and classification.
Loading — planting in the right soil and watering it (loading into the right store).

The first boy skipped the transformation: he dropped seeds on the way, did not separate the boiled seeds, and planted blindly — garbage in, one sick sunflower out. The point: the same systematic, careful process used in data engineering — apply checks before, understand the quality of the input — is also the attitude that succeeds in life. Attitude always matters.

8.18.2 The Three Fishes

The story of the three fishes: there were three fishes, always friends. One fish one day detected that people were going to come and catch them, so it escaped through a tunnel. Another fish saw how it escaped. The third fish said, "no, I won't do anything." When they came, the first fish — which knew they were coming — left before. The other two fishes were caught; but the second fish, which had understood what the first one did, became chill and still, and then it escaped from the pond. The third fish did nothing and got killed.

The moral, applied to data: do you do that before or after? Are you going to handle it, or are you going to prevent it? Prevention is better than cure — apply it before data enters into a system. That is why pre-ingestion validation exists, and why the river-water analogy (test the water quality before the water enters the home) matters.

The three fishes, mapped to data engineering. Three fishes, three postures toward the same threat:

The first fish detected the danger early and escaped through the tunnel — it prevented the problem before it arrived.
The second fish saw what the first one did, learned, and when caught became still and escaped — it reacted correctly at the moment of crisis.
The third fish did nothing and got killed — it did not act at all.

Applied to data: are you going to handle the bad data after it enters, or prevent it before? Prevention is better than cure — apply it before data enters into a system. That is why pre-ingestion validation exists (the four checks at the door), and why the river-water analogy matters: test the water quality before the water enters the home, not after the family has drunk it.

Recap — the contact session in three lines. Attitude matters: apply the same careful, systematic process to data that the second boy applied to his seeds. Prevention beats cure: validate before the data enters (pre-ingestion), and verify after it lands (post-ingestion). And the whole chain — sources, stores, acquisition, ingestion, validation, quality, leakage, drift — exists so that the data entering your models is data you can trust.

DMML Lecture 8 notes · Data Ingestion, Validation, and Data Quality

Data Management for Machine Learning· postgraduate· 2026-08-07

Sections Breakdown

18.1 Midterm Examination Blueprint and Preparation

How the midterm is structured: case-study questions, syllabus scope (units 1–8 with weight on 6–8), how to present answers, and final advice before the exam.

28.2 Data Sources and Data Collection

Describing a data source (source, creator, origin, location, format), the categorization axes, malformed and unexpected data, and behavioral data feeding collaborative filtering.

38.3 Databases and Data Stores for Ingestion

Relational databases and ACID, NoSQL key-value and document stores, columnar databases, search databases for text, time-series and graph stores, and recommending a store for a scenario.

48.4 Data Acquisition: APIs, Data Sharing, and Third-Party Sources

Calling REST, GraphQL, and RPC APIs; role-based access control for data sharing; third-party and open data sources; message queues and event streams.

58.5 Data Ingestion: Batch, Streaming, and Ingestion Methods

Batch ingestion (time- and size-based, snapshot vs differential) and streaming, the State Street war story, TTL, push versus pull, and the ten ingestion channels.

68.6 ETL versus ELT and Ingestion Tools

Why classic ETL is not enough, the ELT model, hand-coded tools versus ETL suites and modern platforms, and source and destination types.

78.7 Data Contracts, Lineage, and Governance

What a data contract states, consent and contracts in everyday products, the five steps of data lineage, and governance under GDPR, HIPAA, PII, PCIDSS, DRP, and DPD.

88.8 Pre-Ingestion Validation

Validation at the entry point: the four core checks (presence, range, format, data type), the eight-question pre-ingestion checklist, and the river-water analogy.

98.9 Post-Ingestion Validation

Post-ingestion checks (consistency, business rules, statistical checks), the N records in / N minus X out investigation, the 10,000-record header story, leakage and model integrity, and skewness checks.

108.10 Worked Case Study: Retail Monthly Sales Data

A retail monthly sales case study applying all four pre-checks and the three post-checks, including the 12 x 560 = 6720 ledger consistency check.

118.11 Data Quality Dimensions

The core quality dimensions (accuracy, completeness, consistency, validity, timeliness) and the extended set, the three benefits of validation, and industry tools like Great Expectations.

128.12 Data Leakage

What data leakage is, its causes, detection and prevention, the GDP worked example, the idli-and-chutney analogy, and the student confusion over the term.

138.13 Skewness and Data Distribution Issues

Positive and negative skew, the skewness coefficient, detecting skew with tests, histograms and box plots, and fixing it with log or Box-Cox transforms.

148.14 Common Data Problems and Types of Bias

Costly, noisy, low-predictive, outdated data; outliers and the 1.5 x IQR fences; the nine-type bias catalogue; and handling missing features and unexpected values.

158.15 Model Drift and Training-Serving Skew

Why production models decay: data drift (schema skew and distribution skew), concept drift, and training-serving skew and its prevention.

168.16 Drift Detection: PSI, KL Divergence, and Entropy

Drift detection metrics: PSI with the threshold board and a worked example, KL divergence, entropy, information gain, and the Gini index.

178.17 Data Profiling

Data profiling as the CV of your data: summary statistics, data types, values, distributions, and key dependencies.

188.18 Teaching Stories: Attitude, Prevention, and Data Concepts

The guru and the sunflower seeds (attitude and ETL) and the three fishes (prevention is better than cure), tying the session together.

Postgraduate students in Data Management for Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Midterm Examination Blueprint and Preparation

Must-know: The midterm is case-study based: identify which data management concept (ingestion, validation, quality, ETL vs ELT, warehouse, data leakage, lifecycle) is hidden in the scenario and map it to the question. Syllabus is units 1-8, weight on units 6-8 and ingestion/validation/quality.

⚠️ Top pitfall: Reading the question once and answering in generic terms instead of mapping concepts to the case; using AI tools or plain typed paragraphs instead of handwritten, structured answers.

Self-check: A scenario describes sales data with missing months and negative values — which stages of the pipeline do you apply and why?

Connects to: Section 8.5, Section 8.6, Section 8.8, Section 8.9.

Data Sources and Data Collection

Must-know: A data source is described by creator, origin, location, and format. Data is categorized by form (analog/digital), boundary (internal/external), collection (primary/secondary), business role (core/business), and ownership (first/second/third-party). Highly heterogeneous high-speed data maps to streaming + NoSQL + time-series stores.

⚠️ Top pitfall: Validating only 'is this a string' instead of boundary cases like length limits, abbreviation tables, and ambiguous tokens; assuming one canonical format for names that arrive in every shape.

Self-check: A dataset is highly heterogeneous and arrives at high speed from many devices — what characteristics map to what data type and store?

Connects to: Section 8.3, Section 8.5, Section 8.8.

Databases and Data Stores for Ingestion

Must-know: Match store type to data characteristics: SQL/ACID for structured transactions; key-value for single-key scale; document (MongoDB) for heterogeneous JSON; columnar (Bigtable, Cassandra) for IoT/analytics; search (Elasticsearch) for full-text and log analysis; time-series for high-velocity temporal data; graph (Neo4j) for connected data.

⚠️ Top pitfall: SQL reflex — recommending a relational database for text search or log analysis; naming a tool without the reasoning from the question's keywords.

Self-check: A customer wants to find all issues for a product using 20 search keywords — what database and why?

Connects to: Section 8.2, Section 8.4, Section 8.5.

Data Acquisition: APIs, Data Sharing, and Third-Party Sources

Must-know: Data acquisition = client-server mechanism: call an API (REST, GraphQL for complex/graph systems, gRPC/RPC) and the server returns data. RBAC shares only the segment of data each pipeline needs at table and row level. Message queues use producer-subscriber delivery (Instagram model); event queues handle continuous streams.

⚠️ Top pitfall: Naming one API style for every situation; forgetting that RBAC controls access, not data quality; confusing message queues (discrete messages) with event queues (continuous streams).

Self-check: How do you pull data from a system that does not follow standard formats?

Connects to: Section 8.3, Section 8.5, Section 8.7.

Data Ingestion: Batch, Streaming, and Ingestion Methods

Must-know: Ingestion is the front door of the pipeline. Batch = time/size windows, snapshot vs differential, always a day behind (8-24 hours); streaming = continuous, with late arrival, duplicates, and out-of-order events handled via CDC, queues, and TTL balance. Push = producer decides (RabbitMQ push); pull = consumer fetches (Kafka, Kinesis pull). TTL: Google Cloud 7 days, Kinesis 365 days.

⚠️ Top pitfall: Forgetting batch freshness is inherent (batch is always a day behind); setting TTL too short (messages vanish) or too long (backlog, hacker exposure); assuming event order is preserved in async queues.

Self-check: Why do bank apps send the same confirmation message multiple times?

Connects to: Section 8.3, Section 8.4, Section 8.6, Section 8.8.

ETL versus ELT and Ingestion Tools

Must-know: ETL transforms before loading; ELT loads first and transforms in the target platform. Choose ELT for scale, complexity, and frequent changes because transformation runs inside the scalable warehouse and rework becomes a query change; choose ETL when the target cannot transform or compliance requires cleaning before entry.

⚠️ Top pitfall: Answering 'ETL vs ELT' with the acronym only; forgetting to say WHERE transformation happens (pipeline vs warehouse), which is the entire point.

Self-check: When do you use ELT instead of ETL?

Connects to: Section 8.5, Section 8.7.

Data Contracts, Lineage, and Governance

Must-know: Data contract: written agreement stating what data, by which method (full/incremental extract), how often, and who the contacts are; stored in a well-known location (GitHub repo or docs site); violating it is data leakage. Lineage: identify assets, track from ingestion, document sources, map the path, pinpoint serving. Governance: quality + security + availability under GDPR, HIPAA, PII, PCIDSS, DRP, DPD.

⚠️ Top pitfall: Confusing lineage (where data went and changed) with validation (whether it stayed correct); thinking consent boxes are irrelevant to data contracts rather than the same mechanism.

Self-check: What must a data contract state, and where should it be stored?

Connects to: Section 8.4, Section 8.12, Section 8.11.

Pre-Ingestion Validation

Must-know: Four pre-ingestion checks: presence (mandatory fields present), range (values within limits), format (dates etc. confirmed), data type (numeric fields contain numbers). Start date before end date = consistency check (relationship between two values), NOT range (which tests one value against limits, e.g. blood pressure -2).

⚠️ Top pitfall: Saying 'range' for the start-before-end-date quiz question; writing checks as prose instead of rules with trigger + action; validating only the easy things instead of boundary cases.

Self-check: Which check ensures a start date is before an end date, and why is it not a range check?

Connects to: Section 8.9, Section 8.10, Section 8.11, Section 8.18.

Post-Ingestion Validation

Must-know: Post-ingestion checks: consistency (relationships between values), business rule verification, statistical checks (skewed left/right, balanced/not). Pre checks (presence, range, format, data type) are NOT repeated. Records received = N - X, where N sent and X missing/rejected; always investigate the X. The 'SAS schema' wording is the star schema (star/snowflake/ER are the warehouse models).

\[ \text{records received} = N - X \]

⚠️ Top pitfall: Repeating the four pre-checks post-ingestion instead of doing consistency, business rules, and statistical checks; ignoring the X missing records instead of investigating their reasons.

Self-check: A file of 10,000 records loads but the audit table shows 9,990 — what do you do?

Connects to: Section 8.8, Section 8.10, Section 8.12.

Worked Case Study: Retail Monthly Sales Data

Must-know: Retail case answer: pre-ingestion presence (month mandatory, missing months skipped/flagged), range (block negative sales), format (YYYY-MM), data type (numeric). Post-ingestion: consistency (12 x 560 = 6720 vs ledger total), business rules, statistical anomaly detection. Answer structure: stage -> check -> what it catches -> action (reject/flag/block).

\[ 12 \times 560 = 6720 \]

⚠️ Top pitfall: Answering only one stage (pre or post); describing checks without stating what they catch in THIS data; missing the system action (skip/reject/block/flag) for each check.

Self-check: What do you do with a missing month record at pre-ingestion, and when would you impute a value instead?

Connects to: Section 8.8, Section 8.9, Section 8.11.

Data Quality Dimensions

Must-know: Core dimensions: accuracy (reflects reality), completeness (no missing mandatory fields), consistency (same across systems), validity (matches schema/rules), timeliness (up to date). Extended: uniqueness, integrity, traceability, accessibility, lineage, reliability, coverage. Benefits of validation: prevents pipeline failures + alert fatigue, reduces cleansing cost, builds trustworthy BI dashboards.

⚠️ Top pitfall: Reciting the five core dimensions and forgetting the extended set; confusing validity (schema/rules) with accuracy (reality); forgetting the feedback loop for consistent labels.

Self-check: Which dimension is violated by storing the same person as 'Raja Kumar' and 'R. Kumar'?

Connects to: Section 8.8, Section 8.9, Section 8.10.

Data Leakage

Must-know: Data leakage: information unavailable at prediction time (test set, future data, the target) reaches the model during training. Causes: target variable in feature engineering, future timestamps in training. Effect: over-optimistic expectations, production failure. Checks: strict train-test split, audit feature sources, cross-validation with leakage detection. Quiz scenario: 'leakage occurs when future or target information contaminates training data'.

\[ \text{GDP} = f(\text{area}, \text{population}, \text{geographic region}) \]

⚠️ Top pitfall: Running preprocessing (scaling, imputation) on the whole dataset before the train-test split — that is a leak; ignoring a suspiciously high validation score, which is itself a leakage symptom; mishearing 'leakage' as 'liquor' (it means data escaping its proper boundary).

Self-check: If test data gets introduced during the training phase, is that leakage?

Connects to: Section 8.7, Section 8.9, Section 8.13.

Skewness and Data Distribution Issues

Must-know: Skewness = asymmetry: positive skew = long right tail (mean > median), negative skew = long left tail (mean < median). Coefficient gamma_1 = (1/n) sum ((x_i - x_bar)/s)^3; zero = symmetric. Detect with statistical tests, histograms, box plots. Fix with log or Box-Cox transforms because skew biases mean-based models.

\[ \gamma_1 = \frac{1}{n}\sum_{i=1}^{n}\left(\frac{x_i - \bar{x}}{s}\right)^3 \]

⚠️ Top pitfall: Thinking 'range' or 'spread' when asked about skew — skew is about the tail direction and mean-vs-median imbalance; applying a transform to tree-based features where skew does not matter; confusing the direction (positive skew = long RIGHT tail).

Self-check: A dataset has mean 30.2 and median 14 — which way is it skewed, and which transform would you try?

Connects to: Section 8.9, Section 8.14, Section 8.16.

Common Data Problems and Types of Bias

Must-know: Bias list with an example each: omitted variable, sponsorship/funding, label (6 vs 9), selection (non-representative sample), implicit, stereotype ('big company data is correct'), experimental, reporting, collection (non-representative collection). AI needs fairness, accountability, trustworthiness. Outliers: outside the interquartile range (1.5 x IQR fences).

⚠️ Top pitfall: Auto-deleting outliers without investigating whether they are errors or real extremes; confusing selection bias (who is in the sample) with label bias (how items are labelled); forgetting to state the example when listing a bias type.

Self-check: A study includes only urban users in its training sample — which bias is this, and why is it a problem?

Connects to: Section 8.13, Section 8.15, Section 8.12.

Model Drift and Training-Serving Skew

Must-know: Model decay in production = drift. Data drift: serving data diverges from training data — schema skew (fields/types differ) or distribution skew (values moved, e.g. real estate prices). Concept drift: labels/interpretation change (concept hierarchy goes out of the box). Training-serving skew: training and serving pipelines handle data differently; reuse the same feature engineering code and feature vector. Remedies: retrain, retune, re-model.

⚠️ Top pitfall: Blaming the model code when nothing changed in it — the data drifted; conflating schema skew (structure) with distribution skew (values) with concept drift (meaning); hand-writing serving features a second time instead of reusing training code.

Self-check: A model's schema matches training, but the values have moved — which drift type is this?

Connects to: Section 8.16, Section 8.14, Section 8.12.

Drift Detection: PSI, KL Divergence, and Entropy

Must-know: PSI = sum_i (A_i - E_i) ln(A_i/E_i), A_i actual share in bin i (new data), E_i expected share (training). Thresholds: > 0.25 significant, 0.1 < PSI <= 0.25 moderate, <= 0.1 no significant drift. KL divergence D_KL(P||Q) = sum_x P(x) log(P(x)/Q(x)) is relative entropy, asymmetric. Entropy H(X) = -sum p(x) log2 p(x); Gini G = 1 - sum p_k^2. PSI is the answer for credit scoring/customer segmentation drift scenarios.

\[ \text{PSI} = \sum_{i=1}^{n} (A_i - E_i) \times \ln\left(\frac{A_i}{E_i}\right) \]

⚠️ Top pitfall: Forgetting PSI is over histogram bins (shares, not counts); answering 'model metric' instead of PSI for the population-drift scenario; treating KL as symmetric when its asymmetry is exactly its strength; saying high entropy = pure data (it is the opposite).

Self-check: A credit model's PSI is 0.31 — what does the threshold board say, and what do you do?

Connects to: Section 8.15, Section 8.13.

Data Profiling

Must-know: Data profiling = understanding the data before modeling: metadata, summary statistics, data types, values, distributions, and key dependencies (multiple fields dependent on each other). It is the 'CV of the data'. Covered up to this point is enough for the midterm; expansion comes later.

⚠️ Top pitfall: Skipping profiling and modeling blind data; confusing profiling (understanding) with validation (checking rules) — profiling is the discovery step that tells you which checks to write.

Self-check: What five things does a data profile capture?

Connects to: Section 8.13, Section 8.9, Section 8.7.

Teaching Stories: Attitude, Prevention, and Data Concepts

Must-know: Attitude always matters — the systematic, careful process (check before, understand input quality) is both good data engineering and a good life attitude. The sunflower story maps to ETL: extraction (collecting seeds), transformation (segregating good/boiled seeds, right soil = clustering/classification, cleansing), loading (planting). Three fishes: prevention is better than cure — validate before data enters the system.

⚠️ Top pitfall: Treating the stories as decoration — the sunflower story maps 1:1 to ETL and data cleansing, and the three fishes justify pre-ingestion validation, so both can appear in exam answers as the 'why'.

Self-check: How does the sunflower story map to ETL, and what do the three fishes teach about validation timing?

Connects to: Section 8.8, Section 8.6, Section 8.5.

Previous LectureLecture 07 · The Machine Learning Lifecycle: From Business Understanding to Model Serving

Next LectureLecture 09 · Data Integration and Data Transformation

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Key

Select Provider & API Key

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

8.1 Midterm Examination Blueprint and Preparation

8.1.1 Question Format: Case Studies and Scenarios

8.1.2 Syllabus Scope for the Midterm

8.1.3 How to Present Answers

8.1.4 Final Advice Before the Exam

8.2 Data Sources and Data Collection

8.2.1 What a Data Source Is

8.2.2 How Data Is Categorized

8.2.3 Format Problems: Malformed and Unexpected Data

8.2.4 Behavioral Data and Collaborative Filtering

8.3 Databases and Data Stores for Ingestion

8.3.1 Relational Databases and ACID

8.3.2 NoSQL, Document, and Columnar Stores

8.3.3 Search Databases for Text

8.3.4 Time-Series, Graph, and Other Stores

8.3.5 Recommending a Database for a Scenario

8.4 Data Acquisition: APIs, Data Sharing, and Third-Party Sources

8.4.1 Calling APIs: REST, RPC, and GraphQL

8.4.2 Data Sharing and Role-Based Access Control

8.4.3 Third-Party and Open Data Sources

8.4.4 Message Queues and Event Streams

8.5 Data Ingestion: Batch, Streaming, and Ingestion Methods

8.5.1 What Data Ingestion Is

8.5.2 Batch Ingestion: Time-Based and Size-Based

8.5.3 The Batch War Story

8.5.4 Streaming, Replication, and Late Arrival Data

8.5.5 Time to Live (TTL)

8.5.6 Push versus Pull Ingestion

8.5.7 Ways to Ingest Data

8.6 ETL versus ELT and Ingestion Tools

8.6.1 Why ETL Is Not Enough

8.6.2 The ELT Model

8.6.3 Tools: Hand-Coded, ETL Suites, and Modern Platforms

8.6.4 Source and Destination Types

8.7 Data Contracts, Lineage, and Governance

8.7.1 What a Data Contract Is

8.7.2 Consent and Contracts in Everyday Products

8.7.3 Data Lineage

8.7.4 Data Governance and Regulatory Standards

8.8 Pre-Ingestion Validation

8.8.1 What Pre-Validation Does

8.8.2 The Four Core Checks

8.8.3 The Pre-Ingestion Checklist: Freshness, Risk, and More

8.8.4 The Water Analogy and Prevention

8.9 Post-Ingestion Validation

8.9.1 What Post-Validation Does

8.9.2 Worked Example: N Records In, N minus X Out

8.9.3 Worked Example: The 10,000-Record File with Header

8.9.4 Leakage Checks, Model Integrity, and Skewness Checks

8.9.5 Post-Ingestion Checks in Summary

8.10 Worked Case Study: Retail Monthly Sales Data

8.10.1 The Scenario

8.10.2 Pre-Ingestion Handling

8.10.3 Post-Ingestion Handling

8.10.4 How to Write This Answer in an Exam

8.11 Data Quality Dimensions

8.11.1 The Core Dimensions

8.11.2 The Extended Dimensions

8.11.3 Benefits of Pre- and Post-Ingestion Validation

8.11.4 Validation Tools in the Industry

8.12 Data Leakage

8.12.1 Definition and Causes

8.12.2 Detection and Prevention

8.12.3 Worked Example: Predicting GDP

8.12.4 The Paratha and Chutney Analogy

8.12.5 Student Confusion and Clarification

8.13 Skewness and Data Distribution Issues

8.13.1 Positive and Negative Skew

8.13.2 Detecting Skewness

8.13.3 Impact and Fixes

8.14 Common Data Problems and Types of Bias

8.14.1 Costly, Noisy, Low-Predictive Data

8.14.2 Outliers

8.14.3 The Many Kinds of Bias

8.14.4 Missing Features and Unexpected Values

8.15 Model Drift and Training-Serving Skew

8.15.1 Why Production Models Decay

8.15.2 Data Drift and Schema Skew