Data Management for Machine Learning: Foundations and Data Representation

1.1 Course Overview and Expectations

1.1.1 What this course is about

This course is called data management for machine learning — the discipline of keeping the data that feeds ML systems in good shape, from the moment it arrives until the moment a model is trained on it. The opening session sets expectations, builds shared vocabulary, and then moves straight into the day's real topic: data representation — the formats data comes in, the data models that organize it, data storage layout, data serialization, and data processing types.

The promise of the course is a comprehensive knowledge: not everything about data, but a solid, connected foundation. The motivation is a real gap the professor keeps seeing: some students submit large language model (LLM) projects with big, impressive pipelines but no basics of the foundation, because tools like ChatGPT hand them the whole stack. This course exists so that nobody finishes with only surface skills — the goal is foundation terms and their interrelations. The teaching style is participative, collaborative, explorative, and experiential: "there is no one answer that is correct," questions and shared experience are welcome, and nobody is superior or inferior.

The course spans 16 contact sessions plus a lab component. There is no dedicated lab machine: every exercise runs on your own system using Google Colab or any other cloud tool. There will be a definite assignment, and there will be a teaching assistant (TA) who handles some interactions and conducts webinars.

1.1.2 The field around data management

Data management for machine learning is an interdisciplinary, ensemble, hybrid subject — it stands where many older subjects meet. In the old days there was a subject called database management systems; later came distributed systems; now this course merges the relevant parts. Related subjects include:

• Database management systems (DBMS)
• Distributed systems — the home of the Hadoop ecosystem, AWS, big data systems, clusters
• Basics of operating systems and of networks
• Data engineering, data science, basic machine learning, basic deep learning
• Software engineering — a discipline built on process cycles — and the move from DevOps to MLOps
• Cloud computing concepts
• Basics of probability, statistics, linear algebra, and optimization
• Big data analytics
• Data governance, data ethics, security, and privacy
• Explainable AI and fairness — summarized as FACT: fairness, accountability, trust, with fairness and bias at the heart

A recurring emphasis: data management is not just storage. It spans the entire lifecycle of the data — we travel with the data, work with the data. The relationship between data and ML is two-way: data is needed for ML, and ML is needed for data. The whole point is to synergize that communication — how data management systems properly work with machine learning.

The course description covers data models and query languages (declarative and imperative), MapReduce (with actual programming), data encoding, the machine learning workflow, pipelines, the challenges of data management, data pipeline patterns, modern and diverse data, cloud warehouses, the data lake, and how data privacy and anonymity are maintained. Module 1 is data representation — today's session — followed by data management, fundamental data architectures, data pipelines, data ops, ML workflows, data ingestion, feature preparation and engineering, distributed and scalable data, a data pipeline for LLMs (one that was recently built), and responsible AI and responsible ML.

Real-world: the class itself is a snapshot of the industry — 117 working professionals from companies including Comcast, Optum, American Express, EY, Airtel, Cognizant, Ecolab, S&P Global, NSF International, EXL, and a fintech background at Paytm. One student is an applied research scientist in computer vision working with OpenCV. One data engineer at Ecolab described terabytes of data that need to be analyzed and modeled across environmental domains; several healthcare data engineers from Optum and EXL work daily with diagnosis and claims data. The point: everyone here already touches the data problems this course formalizes.

1.1.3 Evaluation and examinations

The evaluation is a useful lesson in itself: marks are split across many small checkpoints — a quiz, an assignment, a midterm, a comprehensive exam, webinars — rather than resting on one big test. That mirrors how data work is judged in industry: not by one deliverable, but by how the whole pipeline holds together over time.

1.1.4 Prerequisites and ground rules

The professor explicitly sets the ground rules because students asked about prerequisites: it helps if you know some basics of databases, some SQL, how to work with Excel, basic computer science terms, and some data structures — but these are good to have, not required. The course is designed to carry everyone from basics upward.

Because the class mixes people with 4.5 years of data engineering experience and people with no data background, the professor paces using a box plot picture: students fall into quartiles Q1, Q2, Q3 (sometimes Q4), and the goal is to cover Q1 to Q3 as much as possible — sometimes going basic, sometimes medium, sometimes high. High-level material is aimed at the aspirational quartile. The advice attached to this: have patience, don't feel it's boring, and practice "unlearn to learn" — sit quietly and listen even when a topic feels familiar, because curiosity and asking questions are encouraged.

Every class opens with a positive point and a story — 16 sessions planned, 16 stories, on themes like patience, leadership, and anger. Stories are a deliberate teaching device: "stories are one of the ways I used to get attention and focus towards a class." Education should be fun, not stress; nobody is perfect, nobody is inferior.

1.1.5 How the classes run

The rhythm: the professor poses questions and expects answers ("whenever I ask questions, you should be able to answer"), and students who already apply a topic at work are invited to share their experience. Students type or unmute freely; the opening sessions included a WhatsApp group for shared materials — notes, quotes, tips and tricks useful for the midterm and comprehensive exam, and code for data engineering and EDA. A student coordinator takes the lead role for the group, which also avoids duplicate data: "we just send one" message — one of the first principles of data engineering, a read-once-put-in-cache idea.

Visual intuition — the box plot of the class: picture a number line with students spread along it; a box spans the middle half (Q1 to Q3), with whiskers reaching toward the extremes. The professor aims the lectures at the box, not the whiskers: the bulk of students are covered, the low quartile is lifted up, and the top quartile gets aspirational material to reach for. One-sentence takeaway: pacing follows the crowd, patience keeps everyone in tune.

Real-world & domain connection: this opening session is itself a lesson in data management culture. The class — 117 working professionals shipping real systems — shows the industry reality that data work is a team sport: shared vocabulary, shared channels, one canonical copy of each message, and a coordinator who prevents duplicates. Those habits are the same habits that keep production data pipelines clean.

1.2 Data in Many Forms: The TAVI Riddle

1.2.1 The riddle

"Computer science is common sense — just connect the words." That hint is the whole game: TAVI is not a technical term hidden in some syllabus; it is built from four everyday words about how data arrives in the world. The professor asked the class to guess, and the class could not — which was exactly the point. A room full of working data professionals failing an easy word puzzle shows how easily assumptions block simple answers.

1.2.2 The answer: text, audio, video, image

TAVI is simple once you connect the dots: T is text, A is audio, V is video, I is image. The point of the riddle: in this world, data is not just numbers and numerals. Data arrives in text form, audio form, video form, and image form — and data processing has to handle all of them. Modern workloads routinely need audio processing, video processing, and image processing, not just number crunching.

Visual intuition — the four streams into one pipeline: picture four pipes — labeled text, audio, video, and image — pouring into a single processing tank. In the old world one pipe (numbers) fed everything; today all four flow at once, and the tank must be built for all four. One-sentence takeaway: the shape of the data arriving decides the shape of the pipeline needed.

Real-world: the same idea shows up in the formats the course will touch later — sensor data, social media data from Instagram, Facebook, and Twitter, video, and audio all coexist in the same pipeline.

1.2.3 From EDP to modern data processing

A quick history explains why the riddle matters. About 25–30 years ago, industries like Daimler and Toyota ran teams called EDP — electronic data processing — teams that dealt only with data. But their data was pretty much numbers; anything beyond a number they could only label, not process. A story from 1991 illustrates how far things have come: a microprocessor was wired to a speaker, a song was recorded through it, and a Turbo C program (with menus — "one, two, three": listen to the song, hear the story) played it back. That took about 20 days of coding for the group. Today, audio, video, and image processing are ordinary engineering tasks.

The deeper lesson: data is not just one form. A data pipeline built only for numbers would fail in the modern world, where text, audio, video, and images dominate.

1.2.4 Student questions and answers

Real-world & domain connection: the TAVI idea is not a curiosity — it is the daily reality of modern data teams. A healthcare analytics group ingests diagnosis images, clinical text, and voice notes; a social media analytics team ingests posts, videos, and photos; an IoT platform ingests sensor streams. Every one of these pipelines must be built to carry at least one TAVI form alongside numbers, and often all four at once.

1.3 The Data–Information–Knowledge–Wisdom Hierarchy

1.3.1 The pyramid

The session's centerpiece is the DIKW hierarchy: data → information → knowledge → wisdom, drawn as a pyramid. As you move up the levels, the material becomes more refined and more abstract, and you get a shorter and crisper version — and correctness increases at each level. At the bottom, data is raw, unrefined, and huge; at the top, wisdom is compact and distilled. This is an information hierarchy: data is the raw material of information; knowledge is a true belief we infer and can act upon; wisdom is distilled, integrated knowledge of a higher level of understanding.

1.3.2 Data: the raw material

Data is the raw material of information — numbers without meaning. "99, 90, 80" could be anything: body temperature, sensor temperature, marks, or prices. "1 liter," "10 liters," "50 kilometers," "200 rupees" — these are just values until something gives them a frame. Data on its own is not refined; it has no context, no units that matter, no agreed scale. The same number can be a temperature reading or a score depending on the story around it.

1.3.3 Information: data with context

Information is data organized in a particular manner — data placed into a context. The same numbers become something when you say "these are kilometers" and "this is liters of petrol": now you have the ingredients for mileage. Information is the first level where the raw numbers acquire meaning, but it is still just organized facts.

1.3.4 Knowledge: a true belief we can act on

Knowledge is basically a true belief — an inference that can be acted upon. The example used: if your temperature is above normal (a Fahrenheit reading around 99.6), you most likely have a fever. The number is data; "above normal means fever" is knowledge; you can act on it.

Take the number 1995. As a bare set of digits it is data. Add the context "this is my birth year" and it becomes information. From that information you can infer "I am 30 years old" — an inference you believe and can act on (for example, when filling an age field or checking eligibility). That is knowledge.

Similarly, once many people contribute bike-performance data — "50 kilometers on 2 liters," "50 kilometers on 1 liter," "50 kilometers for 200 rupees" — you can derive knowledge about mileage and compare bikes like the Honda, the TVS Apache, or a Royal Enfield. That comparison then supports a decision: if you want high mileage, buy this bike; if you want low cost, buy that one.

1.3.5 Wisdom: distilled, integrated knowledge

Wisdom is distilled, integrated knowledge — a higher-level understanding that comes from experience. The simplest wisdom example: if you don't feel well, see a doctor. That sentence compresses years of accumulated knowledge into one rule of thumb. The mechanism behind it: experts build wisdom by processing large volumes — studying 10,000 or 20,000 cases, working through permutations and combinations until patterns emerge. The example given was a person who can estimate a sum of money by its weight, or people who read patterns from birth months — not because of magic, but because enough data and experience was fed in. The wise person can "make a decision quickly" where a beginner still needs to collect facts.

1.3.6 Worked examples

1.3.7 Student questions and answers

1.3.8 Explicit and implicit knowledge

The hierarchy connects to a classic distinction in knowledge management (the professor's PhD area). There are two types of knowledge:

• Explicit knowledge — whatever is codified, inferred, already defined, or found in documents and systems. Anything that is written down and shared is explicit.
• Implicit knowledge — knowledge that can only be tapped through deeper processes, over time and through many interactions. One class cannot reveal a person's implicit knowledge; judging someone after a single conversation risks judging wrongly.

Because of this, knowledge is very subjective — like value — and cannot be fully measured person to person. There is always something not yet captured. This is why finding patterns in data matters: patterns let us surface knowledge that no single document contains.

1.3.9 What this means for data management

The practical attitude the hierarchy supports: appreciate the data. Learn to love the data, encourage more data, keep a curious mind about it, involve yourself with it, evolve with it. People who provide data should be appreciated — "just give some thumbs up" — because more data, well managed, is what feeds the climb from raw numbers to wisdom. As a student of data management, data is your material, and the pyramid shows where well-managed data ends up: decisions and understanding.

Real-world & domain connection: every data-driven industry runs on this ladder without always naming it. A hospital's monitoring system gathers raw vitals (data), a dashboard frames them with patient context (information), a clinician infers a diagnosis (knowledge), and the hospital's protocols — distilled from years of outcomes — decide the treatment pathway (wisdom). The same ladder explains why data quality and volume matter so much to machine learning: a model can only distill the wisdom the data's upper levels contain.

1.4 The Value of Data

1.4.1 The story of the three stones

Every class opens with a story; the first class tells the value of data through a parable. A father, unable to convince his son of his own worth, asks a Guruji for help. The Guruji gives him three stones and a rule: "Ask your son to evaluate the stones. He should not sell them. Give him one day; come home at the end of the day and we will discuss." The son visits four places:

1.4.2 What the story teaches

The class offered interpretations, all accepted: value changes with place; what you value yourself matters; value is very subjective; you need patience; knowledge changes value; the expert knows the value of items. The professor's own addition: your expertise and capability may not be useful to every company or person, so their valuation of you changes with their perspective — and this is exactly how the value of data behaves. Data that is worthless in one context is priceless in another; value is perspective-dependent, and the right expert recognizes it.

A personal experience anchored the point: in a two-page profile, an interviewer (Peter Morgan, referenced on LinkedIn) found only two words that mattered — "standardization" — and hired him on that signal. That began seven years building a data warehouse and a mutual fund business engine in the US, with a green card and a sponsored master's degree. The lesson: don't complain or worry when someone doesn't value you — there will be a person who knows your potential and your value. Patience is part of the story.

1.4.3 Data as the new wealth

The first-class mantra: data is the new wealth; data is the new understanding; data is like a good skill. Students echoed it — "data is indeed the fuel" (from a healthcare data engineer at Optum) and "without data you don't have anything" (from a data engineer at S&P Global). "Data is a good fuel, new fuel." If you do not manage your data properly, you lose it and cannot extract patterns from it; managed correctly, data enables anything you want to build with your application, with a good user experience.

Real-world & domain connection: the stones story is a management lens for data teams. A retailer may sit on years of transaction logs it never queries, while a competitor pays a fortune for similar data to tune pricing; a hospital's diagnostic images are "worthless" until a model or a specialist reads them. Every return-on-investment discussion in the course — what data can do for the business — is the four-shop story in company form.

1.5 The Shared Vocabulary: Abbreviations and the One Rule

1.5.1 Why a shared vocabulary matters

The course is full of abbreviations — the professor lists them early and adds more throughout, because professionals in the room use these terms daily and the class must speak one language. Understanding the terms is part of "connecting the dots" the right way.

1.5.2 The glossary

The field also includes graph databases, network databases, and data warehouses. Expect more abbreviations in later classes.

Comparison — ETL vs ELT. The two three-letter sequences look alike but differ in the order of work:

ETL cleans the data before it enters storage — good when storage is precious or the target cannot transform. ELT stores first and transforms later — good when storage is cheap and transformation power lives inside the warehouse. When to pick which: pick ELT for big-data scale in a modern warehouse; pick ETL when the downstream system must never see raw data.

1.5.3 Draw page in, draw page out

The one rule repeated in every class: draw page in, draw page out — the plain-language version of garbage in, garbage out. If you carry all the garbage into your pipeline, the garbage comes out; your machine learning models — any model, mathematical or ML — will run very slowly and badly on messy data.

The teaching picture: imagine an expert swimmer trying to swim fast through very hard, dusty water with tubes and debris. However good the strokes and the pattern, the swimmer will not be fast — the water itself is the problem. Same with ML: irrespective of your machine learning algorithm or deep learning model, if the data is not properly handled, the model cannot perform better. That is why pipelines exist: data must be filtered, organized, and categorized before a model ever sees it.

Real-world: the next session brings a Python exercise — a full EDA and feature engineering walkthrough on car data in Google Colab, shared with the class, showing the complete process of taking raw data (garbage, if you like) to nicely organized data. "You can take the garbage, but if you can take care of the garbage into something useful, then it's okay."

1.6 What Is Data Management?

1.6.1 The working definition

The class definition, read aloud, is:

Data management is the practice of ingesting, processing, securing and storing an organization's data, where it is then utilized for strategic decision-making to improve business outcomes.

Each phrase deserves unpacking, and the class unpacks them one by one: ingestion vs. processing, securing, storing, and — at the end — strategic decision making to improve business outcomes. "End of the day, we are talking about ROI, return on investment. End of the day, we are talking about benefits. What the data can do for me?"

1.6.2 Ingestion: the first step

Data ingestion is the practice of importing, transferring, and loading data — "getting the data into your system, or integrating any other platform to get the data into your application." The older term was data acquisition: the process of gathering, managing, and utilizing data efficiently. Ingestion is a foundational step in the data processing pipeline: first you feed the data in, only then does processing start.

The full picture of ingestion: seamlessly importing, transferring, and loading raw data from diverse external sources — heterogeneous sources, not homogeneous — into a centralized system or storage structure: a data warehouse, a big data platform, or a data lake, where it awaits further processing and analysis. In the pipeline, ingestion is very, very important — if you get this wrong, everything downstream suffers.

Comparison — batch vs. streaming ingestion. Ingestion is not one fixed style. Batch ingestion moves data in scheduled chunks (the nightly CSV copy); streaming ingestion moves data continuously as events happen (a sensor feed, a click stream); micro-batch sits between them (a batch every few minutes). Pick batch when daily freshness is enough; pick streaming when the business needs to react within seconds — the same central destination, different feeding rhythm.

1.6.3 Processing, securing, storing

The definition lists three more duties after ingestion: processing (transforming the loaded data into usable form), securing (protecting it), and storing (persisting it). What we store is organizational data — though "organizational" is not a hard boundary: an organization's own data is its primary data, and there are other types (internal vs. external data, text vs. numeric data).

Data management spans the whole data lifecycle, not just storage: the moment data arrives is part of management, the years it sits in storage are part of management, and its retirement is part of management too.

1.6.4 Strategic decisions and ROI

The purpose clause is the payoff: data is managed so it can be used for strategic decision-making to improve business outcomes. The economic framing: end of the day it is ROI — return on investment — and benefits. Real-world: how much can social media data — Instagram data, tweets — influence my customer? How can my products grow? Data management exists to answer questions like these, not merely to store bytes.

1.6.5 Internal and external data

Data can be internal (produced inside the organization) or external (purchased, scraped, or gathered from outside); it can be text, numeric, or any of the TAVI forms. These categories cross-cut each other: internal data can be text, external data can be numeric, and so on. No single classification is exhaustive — another reminder that flexible thinking, not rigid rules, is the right mindset.

1.6.6 Student questions and answers

A second student checked whether the term really covers everyday wording:

Real-world & domain connection: ingestion is where data engineering careers start and where data pipelines most often break. In production, the ingestion layer handles the daily chaos of source systems — a partner's schema changes, a feed is late, a file is corrupt — and the rest of the organization builds on whatever it delivers. That is why the course treats ingestion as foundational: if you get this wrong, everything downstream suffers.

1.7 Why Data Management?

1.7.1 Why we need managers

The joke that makes the point: why do we need managers at all? Because management — whether in a family or a company — guides, supports, and encourages people, takes risks, and keeps everything in control and seamless; without it "we will dance, we go out of tune." (The humorous illustration: before marriage your mother is the big boss, after marriage your wife is the big boss — "otherwise we are royally screwed.") The same rule applies to data: data without management goes out of tune. Data needs a manager in the same sense a team does.

1.7.2 The growth of big data

The pressure behind data management is the growth of big data. Today every device is connected: your mobile, your TV, your electrical meter, traffic signals. Sensors and small devices compute at the edge — edge computing at the node level — and the data flows constantly: a bus transfers data, a car transfers data, a light transfers data. Big data grows, priorities change, and data can age — the value of data changes over time, exactly like the stone in the story. Because the one who knows the data conquers the business, data management as an organizational discipline becomes an increasing priority.

1.7.3 Data silos

Data silos are pockets where data sits isolated. When teams are siloed, they are isolated and non-collaborative, and their data is too. The symptoms: the same data exists in many places, some of it duplicated, which causes bottleneck problems and blocks analysis. Handling silos means cleaning and integrating data so the organization can improve business outcomes and increase revenue.

1.7.4 Student questions and answers

The follow-up question moved from people to the data itself:

Real-world & domain connection: the connected-everything picture is not hypothetical. Electricity meters report usage continuously, city traffic signals stream flow data, buses and cars transmit location — all computed at the edge and all flowing into central systems. Each of those streams is data that needs a manager; each one left unmanaged is a future silo. This is why the course treats data management as an organizational discipline, not an optional extra: the one who knows the data conquers the business.

1.8 Why Data Management for Machine Learning?

1.8.1 How databases worked before machine learning

The funny but sharp answer starts with history. When the professor studied engineering (1988–1992), the subjects were database management systems and something called artificial intelligence — no machine learning. In those days, humans processed the databases: bank data, insurance data. Teams had ETL developers and ETL programmers, people in the data stage using tools like Ab Initio, Crystal Reports reporting people, Oracle, Business Objects. Extraction, transformation, and loading were done manually. The machine did not learn: the machine performed the logic. Everything was codified in the DBMS — the machine just ran the SQL on an Oracle DBMS or MySQL server, and the human looked at the output and made the decision.

1.8.2 Now machines learn

Is that the case now? No — machines are learning. We give instructions to the machine and the machine performs those actions. That shift is the whole reason this course exists: machine learning is a different way of doing things, deep learning is a different way of doing things, machines evolve and do things. The main ingredient is data — so the data has to be properly managed so that machine learning can properly deal with it, and so data management systems can properly work with machine learning. That is what the subject is about. Data gives you insight; based on that insight you can create a proper machine learning pipeline — the flow that takes the data into the model.

1.8.3 An ML system is a data processing pipeline

A machine learning system is best seen as a data processing pipeline. You build the pipeline; the pipeline extracts usable data and repeatable insights from the data, and you build things on top of it. Compare with traditional transaction processing or log processing: take the data, read it, make a decision. ML systems are different in three ways:

1.8.4 Structure, performance, and accuracy

The three quality lenses the course applies to managed data: structure (is the data organized the right way?), performance (does the pipeline run well?), and accuracy (is the data right?). A related warning from machine learning basics: when you fit a model to data, you can go wrong by overfitting or underfitting; if you don't have the right, relevant data, you may not build a proper model and the whole processing pipeline will not be correct. The model is only as good as the data that feeds it — the pyramid and the garbage rule both say the same thing.

1.8.5 Student questions and answers

Real-world & domain connection: this is the lens for everything from recommendation engines to LLM pipelines. A chatbot's quality depends less on the model architecture and more on the corpus it was trained on; a fraud detector's accuracy depends on the transactions fed to it; a healthcare model's safety depends on the diagnosis data it learned from. In every case, the pipeline — data in, insight out — is only as strong as the managed data at its front end.

1.9 Data Quality

1.9.1 What makes data good

The third question of the day: what do you mean by data quality — what are the key parameters that make data good? The frame: you got customer data, churn data, employee attrition data, healthcare diagnosis data. When do you say such data is high quality?

1.9.2 Common quality problems

Concretely, the issues to look for: interpretability problems, missing values, ambiguity, incompleteness, and inconsistent standards — "somewhere kilometers, somewhere miles," some fields in centimeters, some in millimeters, different formats for the same thing. A data engineer or data scientist should be able to look at a dataset and quickly spot these problems — that is the core skill.

1.9.3 The data doctor

The intuition offered: like a doctor who understands a lot about a person by touching the pulse, looking at the eyes, or checking the hair, a data doctor — the analyst — should glance at the data and immediately understand its quality issues. The moment you see the data, you should be able to read the problems in it. That trained eye is part of what this course builds.

1.9.4 Fixing data: imputation

There are many techniques for fixing data issues; the first one shown is data imputation. Imputation means changing incomplete or missing data by filling it with other data: if age is missing, derive it from the birth date; if null values exist, fill the null with a constant value. More techniques follow in later classes.

Visual intuition — the health check: picture a dataset as a patient chart with five vital signs — completeness, consistency, accuracy, interpretability, standardization — each drawn as a needle. A dataset with a healthy chart feeds models that behave; one with missing values and mixed units shows several needles in the red zone. One-sentence takeaway: quality problems are visible before modeling, and the trained eye is the cheapest quality tool.

1.9.5 Student questions and answers

Real-world: churn data, employee attrition data, and healthcare diagnosis data are exactly the datasets where quality parameters matter most — a healthcare model trained on inconsistent units is dangerous. The same checklist runs in every industry: a bank scoring credit risk, a telecom flagging churn, a hospital reading diagnosis codes — all of them start by reading the pulse of the data.

1.10 Data Formats: Structured, Unstructured, Semi-Structured

1.10.1 The three classic categories

Data can be qualitative or quantitative, and it arrives in many formats: sensor data, social media data from Instagram, Facebook, and Twitter, video, audio — the TAVI forms again. The classic classification — "as per the old book, the old testament" — has three major types: structured, unstructured, and semi-structured, plus the mixed or hybrid case that combines them.

1.10.2 Structured data and schemas

Structured data is like Excel: rows and columns, a label, a value — "you never go wrong." The defining feature is the predefined schema.

1.10.3 Unstructured data

Unstructured data does not adhere to any schema: news, announcements, video, text, and images — social media data, documents, audio, video, text files. Because there is no fixed shape, unstructured data is sometimes very difficult to organize, and you have to do work to "wrap" it before you can retrieve information from it.

1.10.4 Semi-structured data

Semi-structured data is a mix of structure and no structure. The email is the everyday example: it has a header — subject, from, to — which is structured, plus an unstructured body. Semi-structured data has an evolving schema (unlike structured data, where the schema is fixed) and is highly interoperable.

1.10.5 Quasi-structured and hybrid data

The term quasi-structured appeared as a class question: quasi means "apparently but not really." So quasi-structured data is loosely organized, often inconsistent, but still has a pattern that can be parsed — it is not purely structured, nor fully unstructured. The class correctly identified this as the same idea as semi-structured — the hybrid. Real-world examples: email data, log data, server outputs, and combinations of all of them. The professor also flagged something coming later in the distributed-systems part of the course: the CAP theorem.

Comparison — the format spectrum. The three categories are not a strict ladder but a useful spectrum:

When to pick which: structured when rules and joins matter; unstructured when flexibility matters and you can afford the wrapping work; semi-structured when you want both — structure where it helps, freedom where it does not.

1.10.6 Trade-offs between the formats

Structured data has advantages: no further manipulation needed — data pull and data push-back are easy. Its disadvantage is the strict format: an employee can be in a department or a manager — nothing beyond that.

1.10.7 A preview of normalization

Because the golden days had nothing except tables and now everything is mixed, the session previews normalization: a process used to reduce data redundancy and improve data integrity. Normalization creates multiple forms — first normal form, second normal form, third normal form — and the next class works through an example of it, using healthcare-style data. The preview example: student ID, student name, course ID, course data — mapped correctly via a mapping table that says which course which student has taken.

1.10.8 Student questions and answers

The second question triggered a vocabulary correction that the class needed:

Real-world & domain connection: format decisions decide tooling in every data stack. Relational warehouses are built for structured data; document stores and JSON/XML pipelines carry semi-structured data; image, video, and audio systems (healthcare imaging, video analytics, voice assistants) handle unstructured data with models — often all three formats in one pipeline, each with its own validation rules.

1.11 Data Models

1.11.1 What is a model?

Before data models, the question: what is a model? Suggestions from the class: a blueprint, something that stores specifically a data type, a relation. The working answer (the professor goes with the Wikipedia-style definition): a model is a representation of anything — it provides a kind of skeleton, an abstract representation. Machine learning models and deep learning models are representations of patterns in data.

1.11.2 The relational data model

The relational model is "one of the very, very old and gold data models." Data is organized into relations — rows and columns. Each relation is a set of tuples; a tuple is nothing but a row. A table is the visual representation of a relation. Relations are unordered — you can shuffle the order, which is why you have ascending and descending order, GROUP BY on columns, ORDER BY on columns. Columns have many names: object, attribute, subfield, feature — in ML speak, a feature.

1.11.3 The hierarchical data model

The hierarchical model follows a proper hierarchy: parent and child, a tree structure. It suits very rigid structures where strict rules are followed; clustering and non-clustering indexes operate in this space. It is mostly used with a hierarchical tree structure and XML, and registry editors are an example of a hierarchical model.

mammal
 └── human
      ├── man
      │    └── boy
      └── woman
           └── girl

1.11.4 The graph data model

In the graph model, every piece of data is a node and relationships are edges. Nodes and edges let you create many complex relations.

Customer ──places──▶ Order
   ▲                   │
   │                   │ has
   └──────selects──────┘
             │
             ▼
         Product

Real-world: Neo4j (the professor is expert in it and will show graph models before the midterm), Giphy, Neptune — and the social graphs of LinkedIn and Facebook: "this person is a friend of that person; this friend likes this."

1.11.5 The document data model

The document model stores every record as a document — for example, a bio-data — in a collection. Everything is stored as a PDF, XML, or JSON file. Each document is unique in the collection. The analogy: a collection of documents is like a table; a document is like a row.

{
  "title": "Fundamentals of Data Engineering",
  "author": "Reis & Housley",
  "publisher": "O'Reilly"
}

Real-world: MongoDB is one of the popular databases and is both a document database model and a key-value model; Amazon DocumentDB, CouchDB, Firestore, and Firebase are other document databases. It is used for both structured and unstructured data.

1.11.6 The key-value data model

In the key-value model, every key has a value, and the pair is represented in JSON format: item, braces, key, value. JSON's structure makes it the natural carrier for key-value and document data alike.

{ "id": "C1001", "name": "Priya", "bank_card": "1234-5678-9012" }

1.11.7 Other models and the NoSQL choice

Beyond the five: object-oriented data models exist (missing from the slide but real), and object-relational models mix both. There are also network databases. The practical takeaway for choosing: most of the time you either go with the relational model or with the other models — which are called NoSQL, "no structured query language" — so the choice is really relational vs. NoSQL, and the NoSQL family covers graph, document, and key-value data models.

Comparison — the five models at a glance:

When to pick which: start from the questions the data must answer — relationships point to graph, flexible records point to document, raw speed of lookup points to key-value, and enterprise reporting still points to relational.

1.11.8 Student questions and answers

The class was then asked to examine the sample table itself:

And before the models were introduced, the definition was settled by the class:

1.11.9 Exam notes

Real-world & domain connection: the data model choice is a business decision with real systems behind it. Banks run on relational engines (Oracle, MySQL) where transaction integrity is non-negotiable; social platforms model friendship on graph engines; e-commerce catalogs and content systems use document stores; caches and session stores run on key-value. Choosing a model means betting on the questions your data will face.

1.12 Working with Data Without Being a Programmer

1.12.1 The second riddle

The question sounds like a riddle: if data work belongs to programmers, how do thousands of analysts, managers, and domain specialists earn a living from data without writing a line of code? The answer is that the tooling layer has grown so strong that coding is no longer the only door into data work.

1.12.2 The answer: no-code and low-code tools

Multiple tools exist for no-code and low-code solutions. The map given: data analysis using Excel; BI (business intelligence) using Power BI or any other BI tool; data preparation using Alteryx; data integration using Talend-like tools. And if you know a little low-code, Databricks workflows and similar platforms let you build pipelines with minimal programming. Visualization tools like Tableau round out the picture.

1.12.3 The catch: domain knowledge and analytical skills

There is a catch. Even without deep coding skill, you need domain knowledge — you must understand the field the data comes from — and you must be good at analytical skills: the ways and means to deal with the tools. "As long as you have at least some domain knowledge, and you are good at analytical skills, even if you don't know much about coding you can deal with data." Tools amplify people who understand the problem; they do not replace that understanding — the same lesson as the pyramid and the garbage rule.

1.12.4 Student questions and answers

Real-world & domain connection: this is how data teams actually staff themselves. Business analysts run dashboards in Power BI and Tableau, supply-chain specialists shape data in Excel and Alteryx, and integration specialists wire sources with Talend — while engineers build the platforms underneath. The same mix appears in the class itself, where working professionals from healthcare, finance, and environmental domains bring exactly the domain knowledge the tools cannot supply.

Exam Guidance Summary

The evaluation structure for the course, consolidated:

• EC1 (internal component): 30 marks total. Quiz 1 is worth 10 marks; the assignment is worth 20 marks.
• Quiz 1: opens 24 May, closes 28 May, on the Taxila portal. Dates are also shared in the WhatsApp group.
• Quiz 2: comes after the midterm. Pattern: one quiz before midterm, one after.
• Assignment: opens 5 July, closes 19 July.
• Exams: a midterm exam and a comprehensive exam at the end of the course.
• Webinars: four webinars are part of the course, conducted with the teaching assistant.

Content-focused guidance from the session:

• Exam note: data models (relational, hierarchical, graph, document, key-value; relational vs. NoSQL) may appear as a midterm examination question — the professor explicitly flagged the summary slide of data models for the group.
• Exam note: the DIKW hierarchy — definitions of data, information, knowledge, and wisdom, with the worked examples (birth year, mileage, fever, meeting-audio-to-knowledge-graph) — is foundational material; the definitions and their progression are exam-relevant.
• Exam note: the distinction among structured, unstructured, semi-structured (quasi-structured) data, and the schema definition, is core vocabulary for the midterm.
• Exam note: be ready for the abbreviation set — EDA, ETL, ELT, LLM, NLP, TAVI, RDBMS, SQL, CSV, JSONB, BSON, XML, OLAP, OLTP, pickle, IPYNB, UML — and for concepts such as ingestion, data quality parameters, and imputation.
• Exam note: the definition of data management itself — ingesting, processing, securing, and storing data for strategic decision-making and ROI — is a core definitional concept; be ready to unpack each phrase.

The WhatsApp group carries shared materials: notes from every contact session, quotes, tips and tricks, and code for data engineering and EDA — the instructor calls these useful for the midterm and the comprehensive exam.

Recommended book (PDF shared for the group): Fundamentals of Data Engineering.

Next class: a worked normalization example (first, second, and third normal form) and a Python EDA plus feature engineering exercise on car data in Google Colab.

Study attitude: classes move basic → medium → high, covering quartiles Q1–Q3 and sometimes Q4; have patience, and "unlearn to learn" — the course builds from the foundation upward, so earlier material may feel simple for experienced students but is the base for everything after.

Key Industry Applications

The session's real-world connections, consolidated:

• Data platforms and warehouses: data warehouses, data lakes, big data platforms, cloud warehouses — the storage destinations of data ingestion; the professor's own career included a data warehouse and mutual fund business engine built in the US.
• The EDP heritage: Daimler and Toyota ran electronic data processing (EDP) teams 25–30 years ago, handling numbers only; modern pipelines must handle text, audio, video, and images.
• ETL and reporting tools: ETL developers and programmers, data stage tools like Ab Initio, Crystal Reports, Oracle, Business Objects, MySQL — the old-world stack where humans ran the databases before ML.
• NoSQL databases: MongoDB (document + key-value), Amazon DocumentDB, CouchDB, Firestore, Firebase; graph databases Neo4j, Giphy, AWS Neptune; the social graphs of LinkedIn and Facebook.
• BI and analytics tooling: Excel, Power BI, Tableau, Alteryx, Talend, Databricks workflows — the no-code/low-code path to working with data.
• Cloud and big data: Google Colab for lab work, the Hadoop ecosystem, AWS, clusters — the distributed-systems side of the field.
• ML and LLM pipelines: machine learning systems as data processing pipelines; large language model pipelines (including one recently built); responsible AI and responsible ML; edge computing with connected devices — mobiles, TVs, electrical meters, traffic signals — feeding big data.
• Domain data at work: healthcare data (diagnosis, churn, attrition datasets from Optum, EXL, and healthcare analytics), terabytes of environmental data at Ecolab, instant-delivery price comparison apps, and social media analytics (Instagram, tweets) for customer influence.
• Data quality in practice: imputation for missing values (age from birth date, nulls filled with constants), unit standardization across datasets.
• Professional framing: data as the new wealth and fuel; ROI as the business end of data management; value of data is subjective and context-dependent, like the three stones.

This session's foundation: data representation — formats (structured, unstructured, semi-structured), data models (relational, hierarchical, graph, document, key-value), the DIKW hierarchy, data management definition, ingestion, data quality, and the case for managing data for machine learning.