The Machine Learning Lifecycle: From Business Understanding to Model Serving

7.1 The CRISP-DM Lifecycle

CRISP-DM — the Cross-Industry Standard Process for Data Mining — is the de facto standard lifecycle for machine learning, data mining, and data management projects, and it is still used in modern projects. Every project of any of these kinds starts the same way: with business understanding. The traditional ML lifecycle you already know — business goal, problem framing, data processing, developing the model, evaluation, deployment, monitoring, and then the cycle continues — maps directly onto CRISP-DM's stages. The rule to keep in mind through all of it: a model is only effective if you have a good understanding of the business, a good understanding of the data, and good data preparation. Garbage in, garbage out.

7.1.1 Business Understanding

Before anything else, the business must be understood. Determine the objectives of the project, and assess the situation you are trying to evaluate: what is feasible, what are the risks, what contingencies exist, and what goal are you trying to achieve? Then separate two related goals — the data mining goal and the machine learning goal — and come up with a project plan for the project, following whatever method you choose, for example an agile approach.

To see why the two goals must be kept separate, take a churn-prediction project for a subscription service:

• Business goal (what the company wants to happen): reduce customer loss by 15% within two quarters, measured by the subscription renewal rate.
• Data mining goal (the measurable technical target that serves the business goal): build a model that predicts, for each customer, the probability of leaving next month, with at least 85% accuracy on a held-out test set.

The business goal is the why; the data mining goal is the how we will measure progress. A project whose only goal is "build a good model" has no success criterion — you cannot tell whether you succeeded until you know which business number the model is supposed to move.

7.1.2 Data Understanding

If you do not understand the data, you can do nothing with it. Data understanding has four activities, each with its own report:

• Collect initial data — produces the initial data collection report.
• Describe data — produces the data description report.
• Explore data — produces the data exploration report.
• Verify data quality — produces the data quality report.

Before going anywhere, you need to know how your data is dispersed: what is the lowest value, what is the middle one, what is the higher one, what is the mean value, what is the median value, what is the range of the data, what data types it has, and what the quality of the data is. Understanding data quality is the first and foremost step in any data management project.

These four reports are the deliverables of the stage, and each answers a different question. The initial data collection report answers "where did the data come from and what did we actually get?" — how many rows arrived, from which source systems, over which period. The data description report answers "what do the columns look like?" — data types, means, medians, minimums, maximums, and ranges, which is exactly the dispersion check listed above. The data exploration report answers "what patterns are hiding in the data?" — distributions, relationships between columns, and anything surprising. The data quality report answers "can we trust this data?" — missing values, duplicates, outliers, and inconsistencies. In short, data understanding is the look before you leap step: you cannot clean data well, or later build a model on it, if you never found out what it contains.

7.1.3 Data Preparation

Data preparation continues from data understanding and covers these activities:

• Data set description.
• Select data — with criteria for inclusion and exclusion.
• Clean data — produces the data cleaning report.
• Construct data — derive attributes, generate reports.
• Integrate data — merge data.
• Format data — produce reformatted data.

Four things are addressed here: data transformation, data mapping, data integration, and data construction with derived attributes. In industry, data understanding and data preparation are sometimes lumped together and called data cleaning or data pre-processing — different people call it different ways. Whatever the name, by the end of this stage the data must be completely ready before it is handed to modeling.

7.1.4 Modeling

Modeling has six activities:

• Select modeling technique.
• Modeling assumptions.
• Generic test design — the test design.
• Build model — with parameter settings and a model description.
• Assess model — the model assessment.
• Revise parameter settings.

This is the main thing for the machine learning and data mining part of the project. The selection step is where you decide: for my problem, should I use linear regression, logistic regression, a decision tree, a random forest? What method, what algorithm, what technique will be useful for me? Then you state your assumptions, generate training and test data, and decide the parameters — for a neural network these include batch size settings and the learning rate. Normally in machine learning we talk about precision, accuracy, bias, and many other things we define. Then you assess the model and revise the settings. This core stage works only when the earlier stages worked — the model will be effective only with a good understanding of business, a good understanding of data, and good data preparation.

Notice the discipline built into the stage: you do not just "run an algorithm." You pick the technique with a reason (the data is tabular and small, so a gradient-boosted tree beats a deep network), you write down the assumptions you are making, you design the test before building the model (so you cannot tune toward the test), and you treat the parameter settings as something to revise. The assess — revise — rebuild loop is the modeler's real work: build, measure, adjust the batch size or learning rate, build again.

7.1.5 Evaluation

Evaluation results cover two things. First, assess the data mining results with respect to the business success criteria, and choose the approved models. Second, review the process that was done and determine the next step — a list of possible actions and decisions. In practice you may use a couple of techniques, and based on the results you select the model: this one is performing better, this is my approved model, I want to go with it. Then you review the process: anything else missed out? Could we freeze it? And what goes with the next stage?

This is the stage where the two goals from 7.1.1 finally meet. The data mining goal says the model reached 87% accuracy — good, but the business goal says customer loss must drop 15%, so the team must check whether that accuracy actually produces the promised business effect, and must review whether the whole project was run honestly: was the test data ever touched during training? Were the assumptions recorded in modeling actually true? Evaluation answers two different questions — "is the model good enough to approve?" and "did we do the project the right way?" — and both must be answered before the model is allowed to move on.

7.1.6 Deployment

Deployment is the final stage of the classic cycle:

• Plan deployment — the deployment plan.
• Plan monitoring and maintenance — the monitoring and maintenance plan.
• Produce the final report — the final report and final presentation.
• Review project experience documentation.

Once the code is done you deploy it — on AWS, on Google Cloud, on Azure, on a website, or on a mobile app, whatever you have built — and then you monitor and maintain it, produce the final report, and review the project experience. That final review is called the post-mortem analysis: how it all happened, what went well, what went wrong. Deployment is not the end; the machine learning cycle continues from there.

Real-world connection: every major cloud platform ships tooling that mirrors these stages — AWS SageMaker, Google Vertex AI, and Azure Machine Learning each provide labeled phases for data preparation, training, evaluation, and deployment, and modern ML platforms such as MLflow and Kubeflow track the artifacts and parameters produced at each stage. Data engineers and ML engineers in industry effectively implement CRISP-DM whether or not they use the name; the lifecycle survives because the failure modes it prevents — starting with the wrong question, or shipping an unmonitored model — are exactly the failures that still sink projects today.

7.2 Post-Mortem Analysis and Unbiased Analytics

The deployment stage's post-mortem analysis is worth a story, because it shows what a post-mortem is for and how hard it can be to stay objective while writing one. After the Vietnam war — a small country that beat the American army through guerrilla tactics, then suffered a brutal return attack with massive civilian damage — the army chief assigned two experienced officers to fly out separately and write complete post-mortem reports of what had happened in the war. Both were skilled, experienced people. The same incident, the same war, the same caliber of person — and two opposite outcomes.

7.2.1 The Two Post-Mortem Reports

The first officer started walking the ground and looking at the people. He saw children crying without a father or mother, mothers who lost their children, people who lost eyes, ears, hands. He could not control his emotion; he felt so much empathy and pity that he could not digest it. In his letter he realized that when people speak in anger they do not know what they say — during the war nobody cared about anything else. He wrote that he could not tolerate what he had been part of, called it a sin, and committed suicide on the spot, sending the letter back with the pilot.

The second officer also felt the pity, but he moved in the opposite direction. He walked among the people and talked to them, pacified them, gave them what he had — milk and biscuits for a hungry child — and arranged help through the Red Cross. And he documented everything: he took pictures and videos, recorded audio, captured every scene — a mother who lost a son, a son who lost a mother, people missing limbs — and he measured the damages. Back home he submitted a two-page report of the damages caused and how many people were killed and wounded. Right after that he handed in a second letter: he was resigning his position, no reason given. On his way home he was walking, and his son asked why there was no jeep. He answered: see this person, he lost his father; see this boy, he does not even have a leg; you have two legs — today we walk to the park. He started counseling his own family, then his neighbors, reframing every small complaint — a wife complaining her husband snores, people unhappy with their noses or eyes — against the suffering he had documented. He became a well-known psychology consultant.

The two reports differ in exactly the way that matters for analytics. The first officer's report was a record of his emotion — he could not separate what he saw from how it made him feel, and the report became a confession rather than an account. The second officer's report was a record of evidence — photographs, audio, measurements, counts of people killed and wounded. Both men felt the same pity; only one of them kept the pity out of the report. That is the entire difference between a post-mortem that informs and a post-mortem that only expresses.

7.2.2 What the Story Teaches About Analytics

Two lessons come out of the story. First, the coin always has two sides — in life you always have two options, and in a tough situation you should not get disappointed; always look for alternatives. Second, the evidence the second officer brought back — the videos, the photos, the measurements — is what let people finally understand how brutal the war had been. That is what good analytics does: sometimes you find findings you did not expect, and the evidence carries the truth. But for that to work, you have to be unbiased and agnostic — tool agnostic and model agnostic. A good analyst stays neutral, like a fair judge, and lets the data speak even when the result is inconvenient. If you come into the analysis already decided, the post-mortem is worthless.

Real-world connection: the same discipline is institutionalized in modern engineering practice as the blameless post-mortem — Google's Site Reliability Engineering approach and similar incident-review culture at cloud providers require the review to focus on what the system did and what evidence exists, never on blaming the person who was on call. Modern ML platforms carry the idea further with automated lineage tracking: the model registry records exactly which data, code, and parameters produced a model, so a later "post-mortem" of a bad model in production can be conducted on reproducible evidence instead of memory and emotion.

7.3 Problem Framing, Confusion Matrix, and Error Costs

Goals have to be identified before a machine learning project starts. Part of that is evaluating the cost of data acquisition — how much it costs to get the data — plus the cost of training, the cost of inference, and the answer to this question: what happens if I make a wrong prediction?

7.3.1 Goals, Costs, and Problem Framing

In problem framing you need to find the criteria: what is the criterion that makes a successful project? What is observable? What are all the quantities you can measure? And you must check with the business stakeholders: what is the input, what is the output we are getting, and are we getting the right performance?

A wrong prediction is not a free event. If a machine learning code predicts the gold price wrong, or predicts a wrong material composition and the production line produces something wrong, people who invested in tools and equipment lose a lot of money — budgets and sizing issues cause serious problems and headaches. So part of framing the problem is deciding how you will handle the errors.

Problem framing has four concrete questions that must be answered in writing before any modeling starts:

1. What is the success criterion? Which number, measured how, tells us the project worked? Without a criterion, the project can never "succeed" — it can only be declared finished.
2. What is observable? Which quantities can we actually measure in the business? The churn model can observe past subscriptions and past exits; it cannot observe customer intent directly.
3. What are the costs? Data acquisition (buying data or building pipelines to collect it), training (compute, people, time), inference (running the model on every prediction — especially when predictions happen thousands of times per second), and error (what a wrong prediction costs).
4. What is the input and output contract? Confirm with the business stakeholders exactly what goes into the model and what comes out, and how performance will be judged — "are we getting the right performance?" asked before, not after, the project runs.

The gold-price example shows why error cost belongs in this list: a model that predicts tomorrow's gold price and is wrong by a few dollars can wipe out positions held by people who trusted it, and a manufacturing model that predicts a wrong material composition sends a production line making parts that must be scrapped. Budgets and sizing issues cause serious problems and headaches — the framing stage is where those headaches are either planned for or ignored.

7.3.2 Confusion Matrix and Error Types

You have all studied the confusion matrix, so this is a review, and it is the vocabulary for error costs. The professor's example: he plays a doctor, and predicts that a person does not have Corona, while the machine learning program predicts that he does have Corona. The actual result is negative, the predicted result is positive — the machine got it wrong. Four outcomes are possible when we line up actual versus predicted:

When actual positive and predicted positive agree, that is a true positive — no problem. When actual negative and predicted negative agree, that is a true negative — fine. When you predicted positive but the actual is negative, that is a false positive. When you predicted negative but the actual is positive, that is a false negative. The professor labels false positives as type 1 errors and false negatives as type 2 errors, and the whole point of evaluation is to reduce the false predictions — both the type 1 error and the type 2 error.

7.3.3 Qualitative vs Quantitative Metrics

There are always two kinds of parameters to keep in mind: qualitative parameters and quantitative parameters. Saying a teacher is nice, jovial, and good at teaching is qualitative — it does not substantiate anything. Saying the teacher was on time for seven out of seven sessions, that the class of thirty students gets good results, that the CGPA numbers and pass counts are good — those are all quantitative metrics. When you frame a machine learning problem, you need observable, quantifiable performance metrics — accuracy, how accurately we are predicting — not impressions. That is why the success criteria for the project have to be stated in measurable terms.

The lesson for framing: a project's success criteria must be quantitative. "The model should work well" is qualitative and unverifiable. "The model should reach at least 85% accuracy on the held-out test set, and reduce false positives by half" is quantitative — it is observable, measurable, and checkable by an independent reviewer, which is exactly the fair-judge discipline from the post-mortem section.

Real-world connection: error costs are why medical screening (Corona tests, cancer scans) weights the four cells by lives, while fraud detection weights them by money — a bank losing a genuine transaction (false negative) costs the transaction's value, while a false positive costs a customer relationship. Credit scoring, insurance pricing, and algorithmic trading all carry explicit cost matrices over the confusion matrix cells; modern ML platforms expose this as "business metrics" that convert raw counts into the dollars the business actually cares about.

7.4 Feature Stores, Model Registry, and the Drift Feedback Loop

Once the data is prepared, the platform work begins. Everything that happens to the data — data processing, data pre-processing, data engineering, data acquisition, data collection, data staging — whatever you want to call it, falls under one category: process data. Then, with the data ready, the features are stored, the model is trained and tuned and evaluated, and the deployed model is monitored and kept healthy.

7.4.1 From Data to Features: Online and Offline Feature Stores

After the data is pre-processed and engineered, the right next step is to store the features — and sometimes you do it both online and offline.

The online feature store offers low-latency retrieval, which is ideal for real-time inference. The offline store is used for duplication and deduplication purposes. When the model is deployed and an application starts giving data, you monitor it, check it, and store the artifacts — data, code, and model. The model is registered, the data is registered, and catalogs are maintained.

The check-before-train step deserves emphasis: after storing the features, you examine the correlation between features before building the model. Two highly correlated features (for example, years as customer and total months billed) carry largely the same information; keeping both inflates the model without adding insight. The correlation check is the practical version of the feature-selection exercise in section 7.6.

7.4.2 Saving and Versioning Models: The Model Repository

A trained model itself can be saved to disk and reused later, and there are two standard ways to do it.

So the model repository combines serialization with version control. The professor's tip was a versioning scheme: name the first registered model 1.0; when someone else checks out, updates the weights for the new data that comes from different sources, and checks back in, that becomes 1.2.

7.4.3 Model Registry and Lineage

The model registry is a repository for storing ML model artifacts, including the trained model and related metadata such as data, code, and model. It enables the tracking of the lineage of ML models, because it can act as a version control system. Most people in the room already know version control: the history runs from SCCS, VCS, RCS, through ClearCase, to GitHub. We check in, we check out, check in, check out, and we keep a proper code base for normal non-ML code. When it comes to ML, we can use a version control system for the models too.

The lineage tracker goes further: it enables a recreation of the ML environment at a specific point in time. That is like point-in-time recovery in databases — in Oracle there is point-in-time recovery and rollback, with RPO-type recovery, so you can roll back to the version you want. For change auditing we use this: you can re-create what environment and what resources existed at the time you want to go back to.

A model registry, then, answers four questions about any model in production:

1. What is this model? — the artifact itself, the algorithm, the version number.
2. What produced it? — which training run, which data snapshot, which code commit.
3. How good was it? — the recorded accuracy and statistics attached at registration time.
4. What did it replace? — the version history, check in / check out, 1.0 to 1.2 to 2.0.

The version-control lineage of the field runs from SCCS (the earliest source control system) and RCS through ClearCase to GitHub — the same check-in/check-out discipline applied to ordinary code is applied to models. The lineage tracker is the stronger cousin: point-in-time recovery in Oracle databases can restore the database to any chosen moment; the ML lineage tracker can re-create the whole environment — data, code, model version, resources — as it existed at any recorded point, which is what makes change auditing and rollback possible for ML systems.

7.4.4 The Model Drift Feedback Loop

The model drift feedback loop informs the iterative data preparation phase, based on the evaluation of the model during the production deployment phase. What happens is that the model drifts: we keep iterating — this is not good, let me prepare the data again, let me do it again — and we do not allow the model to be considered ready until active learning has done its work. We have an alarm manager to monitor the models; any model that is not giving the proper targets can be flagged, and retraining can happen. Retraining can be scheduled — the model runs and retrains at a certain time. And because data changes, the model keeps learning and relearning: learn, relearn, learn and relearn.

Real-world connection: feature stores such as Feast, Tecton, and AWS SageMaker Feature Store, and model registries such as MLflow Model Registry and Vertex AI Model Registry, are the production products of exactly these concepts — they store features online and offline, version models, and record lineage so an audit can reproduce which data and code produced which model. The drift feedback loop is institutionalized in continuous training pipelines (for example, scheduled Airflow or Vertex Pipelines jobs that retrain nightly and gate deployment on evaluation), and drift monitoring tools quantify distribution change with metrics such as PSI and KL divergence, exactly the "alarm manager" role from the lecture.

7.5 Data Pre-processing and Sampling

This section reviews the data pre-processing toolkit. In the data pre-processing stage we clean the data, remove the outliers, partition the data into training and test data, scale the data, normalize it, standardize it, balance it so it is unbiased, and augment it — whenever you want to expand something, you can expand it. We have already studied feature engineering: feature selection, feature transformation, encoding, binning.

7.5.1 Cleaning, Imputation, and the Pre-processing Toolkit

Imputation is the standard answer to missing values: fill them with the mean or another chosen value. Alongside imputation, the cleaning toolkit removes outliers and removes duplicates, and then you partition the data. Data collection matters too: data can be time series data, it can come from sensors, from IoT, from social media, from multiple sources. Data ingestion means you capture the data and store it on storage media — for example, a smart refrigerator or a smart device connected by Bluetooth or Wi-Fi collects readings and pushes them to a disk, or the data is pushed directly through a streaming method. Then come the data technologies already seen: ETL and ELT pipelines that extract from Oracle, SQL Server, MongoDB, or Excel, then the data catalog, then the data lake, then EDA.

The new mantra in industry is no-code and low-code platforms: within fifteen minutes you can build an ML pipeline — for example, by giving a proper prompt to an AI assistant you can build a nice data pipeline. These automation platforms offer visual capabilities that improve productivity and reduce cost, and they are used together with generative AI code tools.

The toolkit, in the order a real project uses it:

1. Clean — fix or drop corrupt rows, correct inconsistent formats.
2. Impute — replace missing values with the mean, the median, a constant, or a model-based estimate. A column of monthly charges with 5% of rows missing gets its missing cells filled before training.
3. Remove outliers — drop or cap extreme values that would distort the model.
4. Remove duplicates — delete repeated rows; duplicates are the classic channel for data leakage (see 7.5.3).
5. Partition — split into train, validate, and test.
6. Scale and normalize — z-score or min-max (see 7.5.4).
7. Balance and augment — fix skewed classes and expand the data.

An imputation example with real numbers: a column holds monthly charges in rupees, and the column mean is 1200. The missing 5% of cells are filled with 1200. The choice of filler matters — filling with the mean preserves the column's average but shrinks its variance, so it is a trade-off, not a free fix. That is why the professor's answer says "the mean or some value": the right filler depends on the column's distribution and the model that will consume it.

7.5.2 Sampling: The Shirt Story

Sampling was discussed in the previous session with a shirt story. At the age of five, the professor's son could wear his father's shirt — it did not fit, it was overfitting. Now the son and father are almost the same size, so the shirt fits well — best fit. And if the father tried to wear his son's shirt, that would be underfitting. The point: you need to choose the sample correctly, and the fit of your sample to the population decides whether your model overfits or underfits. Data sampling is the key. There are different types of sampling techniques, and choosing the right strategy for your machine learning work is a question worth your time — you can use an AI assistant for ideas, but you should be able to argue for the strategy yourself.

The practical message: the fit between your sample and the population decides overfit or underfit. A sample that does not represent the population — only churned customers, say, when the population is mostly loyal ones — guarantees a model that memorizes or misses instead of generalizing.

Two sampling strategies, and when each is the right tool:

• Random sampling (the default). Every row has an equal chance of entering the sample. Simple, unbiased in expectation, and fine when the data is large and fairly balanced.
• Stratified sampling (the careful choice). You split the population into strata (groups) — for churn, the strata might be "will churn" versus "will stay" — and sample from each stratum so that the sample mirrors the population's proportions. If 20% of customers churn, a stratified sample keeps roughly 20% churned rows in the training set; a random sample of a small dataset could accidentally draw 5% or 40%.

For churn prediction specifically, stratified sampling matters because churn is usually rare: a random sample can end up with so few churned examples that the model never learns to see churners. Stratified sampling guarantees the rare class appears in proportion. The professor's final note is a fair-judge reminder: you can ask an AI assistant for ideas, but you must be able to argue for your strategy yourself.

7.5.3 Train, Validate, Test Partitioning and Data Leakage

Why partition at all?

Common split ratios are 60-40-20, 60-20-20, 80-20, 80-10-10, or 90-10 — whatever you choose, before you split you must make sure the data conforms and there are no duplicates. That is very, very important: duplicates that appear in both the training side and the holdout side are the classic channel for leakage.

The three sets each have a different job: the train set teaches the model; the validate set is the tuning judge (it is used repeatedly while choosing hyperparameters, so it also gradually "leaks" — which is why it must never be used for the final score); the test set is the final, untouched judge that gives the honest accuracy number. A split ratio with only two numbers, like 80-20 or 90-10, is a train/test split that uses cross-validation (section 7.7.4) for the validation role instead.

7.5.4 Scaling and Normalization

To scale the data we use z-score normalization, min-max normalization, and other encoding techniques. Both formulas below are the standard forms used across ML practice: z-score normalization maps each value to how many standard deviations it sits from the mean, and min-max normalization squeezes the column into a fixed range.

\[ z = \frac{x - \mu}{\sigma} \]

Here \(x\) is an original value, \(\mu\) is the mean of the column, \(\sigma\) is the standard deviation, and \(z\) is the scaled value — it tells you how many standard deviations the value sits from the mean.

\[ x' = \frac{x - \min(x)}{\max(x) - \min(x)} \]

Here \(\min(x)\) and \(\max(x)\) are the smallest and largest values in the column, and \(x'\) is the rescaled value, squeezed into the unit interval \([0, 1]\).

On the imbalance side, when classes are skewed you can add bias to the algorithm, augment the data, or synthesize more extra samples to get multiple insights; you can also use the standard techniques that regularize and reduce overfitting. Three remedies for a skewed target class: add class weights (bias the algorithm so rare classes are punished more when missed), augment the data (create realistic new examples of the rare class), and synthesize samples (for example SMOTE-style interpolation between rare-class neighbors). These run alongside the standard regularization techniques — L1/L2 penalties, dropout, early stopping — that reduce overfitting when the model has too much capacity for the data.

7.5.5 Feature Engineering and Dimensionality Reduction

Feature engineering is all about selecting the feature, transforming the feature, creating new features, doing the encoding, doing the binning, and feature extraction. In deep learning, feature extraction becomes automated. Everything is a feature: customer location is a feature, age is a feature, income level is a feature — and every variable needs a proper way to be selected and transformed when you apply the predictive model. The toolset includes feature transformation, feature imputation, and combining features.

For feature reduction and feature selection people use PCA — principal component analysis — plus linear discriminant analysis and independent component analysis. The goal is to have the right amount of data by reducing the dimensions: from 10,000 features down to 1,000, or from 50 columns down to 5 columns. You can reduce the columns, reduce the features.

The feature toolset in one place:

• Feature selection — pick the useful columns and drop the rest (variance and correlation are the guides; see the churn exercise in 7.6).
• Feature transformation — scale, normalize, take logs, bin continuous values into categories.
• Feature creation / derived attributes — build new columns from existing ones (the car-age example in 7.1.3).
• Feature extraction / dimensionality reduction — PCA, linear discriminant analysis (LDA), and independent component analysis (ICA) compress many raw columns into fewer engineered ones.

The three reduction methods differ in what they optimize: PCA finds the directions of greatest variance in the data and projects onto them; LDA finds the directions that best separate the classes; ICA separates a signal into statistically independent components. The lecture's scale of ambition: 10,000 features down to 1,000, or 50 columns down to 5 — the churn exercise in the next section actually reduces its feature set to five.

Real-world connection: every serious ML platform exposes these operations as built-in components — Scikit-learn's StandardScaler, MinMaxScaler, SimpleImputer, and train_test_split; Pandas for cleaning; feature stores that apply the same normalization at training and serving so the transform never drifts between the two. In regulated industries, the stored \(\mu, \sigma\) and \(\min, \max\) values are themselves versioned artifacts: an audit must be able to reproduce exactly how every input was scaled.

7.6 Feature Selection Exercise: Customer Churn Prediction

Here is a live exercise: a customer churn prediction dataset. The question to answer: which features are redundant — which are likely to be removed due to low variance or irrelevance?

7.6.1 The Churn Prediction Problem

The business problem: a service — a ticketing tool, a mobile application, a subscription model — gives monthly charges for the service, there is a tenure (how long each customer has used it), and customers report a satisfaction score. We have about five years worth of customer data. The question: in July — the next month after June — what will be the percentage of satisfied customers? Dissatisfied customers will leave the product, the tool, or the support service. So we want to predict the expected percentage of satisfied customers, and from that the percentage of dissatisfied customers who will churn next month.

7.6.2 Low Variance and Redundancy

Variance is the measure of how far a feature's values spread around their own average:

\[ \operatorname{Var}(X) = \frac{1}{n}\sum_{i=1}^{n}(x_i - \bar{x})^2 \]

Here \(X\) is the feature, \(x_i\) is the \(i\)-th value of the feature, \(\bar{x}\) is the mean of those values, and \(n\) is the number of rows. This is the standard form for the population variance; some texts write the sample version with \(n - 1\) in the denominator for the same reason they do for any sample statistic. A feature that barely varies carries little information, so low-variance features are the first candidates for removal. But "irrelevance" is a business question, not just a statistics question — a feature with low variance can still matter if it explains churn.

7.6.3 Student Q&A: Which Features Can We Remove?

The key correction: a feature is not redundant just because it looks irrelevant at first glance. If you want root-cause and prescriptive insight, age, gender, and even payment method can be the features that reveal why customers leave.

Why the students' answer seemed right, and why it was wrong:

• Why it seemed plausible. "Payment method, age, and gender" look like demographics, not like churn drivers. Dropping them shrinks the feature set, which usually sounds like simplification.
• Why it was rejected. Three separate reasons:

1. Satisfaction depends on them. If satisfaction scores differ by age group or gender, those features carry signal the model needs.
2. Root-cause and prescriptive analytics. Age and gender let the business answer why churn happens and prescribe action — "customers under 25 on yearly plans churn at twice the rate." A model without those features cannot be used to explain churn, only to predict it.
3. Payment method is a business fact, not noise. In this app's data, SBI UPI payments are rejected about 70% of the time — UPI sometimes slows down and does not work — so a customer paying by SBI UPI is measurably more likely to be frustrated and leave. A "redundant" feature that marks the frustrated cohort is the opposite of redundant.

• The replacement mental model. The fair judge, again: even a hypothesis you already believe has to prove itself in the data. The students "successfully biased our thinking" — the professor's point is that every proposed removal is a hypothesis about the business, and hypotheses get tested, not assumed.

7.6.4 The Modeling Code and PCA

The code for this exercise builds the feature set for customer churn prediction next month. Gender, contract type, payment methods, and region are treated as categorical columns. There is a churn threshold for the target "churn next month", and PCA is used to reduce the feature set — the exercise reduces the data to five features. Then the model is developed and trained.

The pipeline the code follows:

1. Load and inspect the five years of customer data; confirm the target column (churn next month) and its threshold — a customer is marked as churned when the churn probability crosses the chosen cutoff.
2. Encode categorical columns — gender, contract type, payment methods, and region are converted from text to numbers (one-hot or label encoding), because models need numbers.
3. Scale the numeric features with the techniques from 7.5.4, since PCA is sensitive to units.
4. Reduce with PCA — principal component analysis compresses the full feature set down to the five most informative components for this exercise (recall 7.5.5: from 50 columns to 5).
5. Train the model on the reduced features and evaluate it.

The important tension: PCA gives you five engineered components (linear combinations of the originals), while the feature-selection question from 7.6.3 was about keeping the original features for explainability. For a pure prediction score, the PCA-reduced set is compact and fast; for root-cause and prescriptive analysis, the original age, gender, and payment-method columns must survive — which is exactly why the professor kept them in the earlier exchange.

Real-world connection: the SBI UPI example is a real, named business case — payment-failure frustration driving churn — and it is why banks and payment providers monitor their UPI rejection rates as a churn-leading indicator. The same "explainability versus compactness" trade-off is what separates prediction platforms (which happily ship five PCA components) from regulated analytics teams (which must keep interpretable columns for audit and root-cause reporting).

7.7 Model Development, Algorithm Selection, and Hyperparameters

After the exercise, model development: we build the model and train it. This is where algorithm selection happens — which model are you going to choose, and are you going to do model parallel or data parallel?

7.7.1 Choosing the Algorithm: What to Consider

When many projects are reviewed — the professor has been a reviewer for many data science and AI projects — the standard advice is always the same: do the literature review. Try, try, try multiple algorithms, and see which one works for you. You have to select the right algorithm and find the factors to consider: what metrics you are going to use, whether the model is explainable, and what type of computations it is going to take. All of these matter.

On parallelism: data parallel means the data is split across multiple instances or nodes, each running the model on its share; model parallel means the model itself is split across nodes. Bagging and boosting are the related ensemble ideas. Then come debugging and profiling: debug the code, find the errors, and profile the entire system.

The selection factors, as a checklist:

• Literature review first. Look at what others solved with this type of problem and this type of data. The review is both a time-saver (avoid re-discovering that trees beat raw deep networks on small tabular data) and an exam discipline (the professor's review advice, repeated in the Exam Guidance Summary).
• Metrics. What will you measure? Accuracy, RMSE, precision and recall — the metric must match the business criterion framed in 7.3.
• Explainability. Can you say why the model made a decision? A credit-rejection model must be explainable; a recommender's ranking score may not need to be.
• Computational cost. What type of computations does the algorithm need — how much memory, how long per training run, and what will inference cost per prediction?
• Parallelism when the data or model is huge. Data parallel splits the data across multiple instances or nodes — each node runs the whole model on its own share of the data, and the results are combined (the standard choice for deep learning on large datasets). Model parallel splits the model itself across nodes — needed when a single model does not fit in one machine's memory. Related ensemble ideas are bagging (many models trained on different samples, results averaged — the "bootstrap aggregating" behind random forests) and boosting (models trained in sequence, each correcting the previous one's mistakes).
• Debugging and profiling. Debug the code and find the errors, then profile the whole system — measure where time and memory actually go before assuming the algorithm is the bottleneck.

7.7.2 Validation Metrics: RMSE

The validation metrics tell you whether the model is actually good. The very important one used here is the root mean square error — RMSE — or the mean square error, MSE. RMSE compares predicted versus actual: the professor's example is a prediction of 90 where the actual value is 80.

\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2} \]

Here \(y_i\) is the actual value for sample \(i\), \(\hat{y}_i\) is the predicted value, and \(n\) is the number of samples. This is the standard form of RMSE: the square root of the average of the squared prediction errors. We do this for multiple samples to find out what the error is, and RMSE is one of the methods used to validate the system. The other pillar is hyperparameter tuning, which gets its own session later.

7.7.3 Hyperparameters: How the Brain Works

Hyperparameters were introduced with a live brain exercise.

The brain exercise maps directly onto machine learning vocabulary:

• Known parameters — the things you can observe directly: ask a question, get a yes or no, correct or not correct. For a model, these are the weights and biases the training algorithm learns and writes down; you can read them out of the model at any time.
• Hyperparameters — the hidden settings you cannot observe directly, which control how the system learns: step size (how big a jump each learning step takes), epochs (how many passes over the training data), and error rates or tolerance thresholds. They are not learned from data; they are chosen before training and tuned by training — which is why the tuning is called hyperparameter tuning and gets its own session later.
• The 1781 lesson. Reversing 1781 step by step gives 1871, and it takes a little time. People who take faster steps make mistakes; people who take smaller, careful steps may do it better. Step size is the model's equivalent: a step size too large overshoots the optimum and the loss jumps around; a step size too small converges painfully slowly. The right step size is a hyperparameter that must be tuned.

The model artifacts are the outputs that result from a trained model. Before production there is a pre-production pipeline: we prepare the system, we get the pipeline — online features, offline features — and finally the CI/CD/CT pipeline: continuous integration, continuous deployment, and continuous testing. That is the future now. Once the pipeline is built, we evaluate; the train pipeline keeps training, confirming that performance matches expectations — and if not, you retrain, retrain.

7.7.4 Training Many Models with Cross-Validation

Model training applies a machine learning algorithm on training data to train the model, and the key practice is: train many ML models from different categories — linear regression, logistic regression, K-means, naive Bayes, SVM, random forest — using standard parameters. That is what ensemble technique is about: apply multiple models with standard parameters, measure and compare their performance. For each model use n-fold cross-validation — the method is available as a package in Scikit-learn, so you just apply it and use the cross-validation score.

\[ \mu_{\text{CV}} = \frac{1}{k}\sum_{i=1}^{k} s_i \]

Here \(k\) is the number of folds and \(s_i\) is the score of fold \(i\). The cross-validation score is the average of the per-fold scores — the standard form of the n-fold cross-validation result. You use this technique to find overfitting: look at the tenfold results, look at the standard deviation around that 90% accuracy — what was the variation — check the errors on every fold, and then select the features you need. Identify the top five most promising models for the problem. It is like buying a phone: you selected Samsung, you selected Google Pixel, you selected another phone, and you compare them on your parameters — which is better, what type of errors. You can also combine the models: ensemble methods like bagging, boosting, or stacking, or a simple majority vote of which model gives the right answer. Then you evaluate the model and test the model.

7.7.5 Model Serialization Formats

A working model has to be distributable, and the serialization formats should be language agnostic — the model should not be restricted to one package; it should be usable from Python, Ruby, or any other code. The formats:

• PMML — Predictive Model Markup Language. Serialize the model to a .pmml file; it is standardized by the data mining group, but it does not support all ML algorithms.
• PFA — Portable Format for Analytics — a JSON-formatted text.
• ONNX — Open Neural Network Exchange format.
• pickle — saves a model as Python objects in a .pkl file.
• H2O — converts the model to either POJO, a plain old Java object, or MOJO, a model object optimized for the model.
• TensorFlow and PyTorch — give a .pt file.

Use whichever is good for your case. You can also choose online training or offline training, and batch predictions or real-time prediction: for batch prediction you forecast and predict; for offline you can use a REST API.

The common thread is interoperability: a model serialized in a portable format can be loaded and executed by a different runtime than the one that trained it — a TensorFlow model served from Java, a PMML scoring engine, an ONNX model moved between frameworks. The trade-offs between the formats: PMML and PFA are standardized and language agnostic but support only the algorithm families their spec covers; pickle is simple and complete for Python but Python-only and unsafe to load from untrusted sources; ONNX is the modern exchange format between frameworks; H2O's POJO (plain old Java object) and MOJO (model object optimized for scoring) target Java runtimes; TensorFlow and PyTorch write their own formats (a .pt file) with native serving stacks.

The training and serving choices complete the picture: online training updates the model as new data arrives (close to the learn-and-relearn loop of 7.4.4); offline training runs on a fixed snapshot; batch prediction forecasts for many records at once (nightly scoring of all customers); real-time prediction answers one request at a time — typically served through a REST API, the standard interface for offline-built models.

Real-world connection: these are the exact mechanics behind modern ML platforms — Scikit-learn's cross_val_score package implements n-fold cross-validation directly; MLflow tracks models, metrics, and parameters so comparisons are reproducible; ONNX is the industry bridge between PyTorch research models and optimized production runtimes; and the phone-buying comparison mirrors how Kaggle winners and industrial teams maintain leaderboards of candidate models with their RMSE or accuracy scores.

7.8 Deployment Strategies and the Inference Pipeline

Deployment and testing strategies reduce the downtime and risk when releasing a new or updated version of a model. Four strategies were covered: blue-green, canary, A/B testing, and shadow deployment.

7.8.1 Deployment Strategies

• Blue-green deployment: two identical production environments — blue is the existing infrastructure, and green is an identical infrastructure for testing. You know this in industry as staging versus pre-prod versus prod. When green passes testing, traffic switches over.
• Canary deployment: a new release is deployed to a small group of users, while other users continue to use the previous version.
• A/B testing: similar to canary, but with a larger user group and a longer time scale, typically days or even weeks. You partition the users and give each group a version.
• Shadow deployment: the new version is available alongside the old version; the old version keeps serving while the new one is validated on live traffic without users noticing.

Exam note: you may expect a question on this — what deployment strategy you would follow, maybe the lowest-cost strategy, which environment you would use. It is worth taking a screenshot of the strategy table.

When to pick which: choose shadow when you must validate on live data without any user risk; canary when you want a small, controlled exposure with automatic rollback; A/B when the question is scientific — which model actually moves the business metric; and blue-green when the goal is near-zero-downtime switching and you can afford the duplicate environment.

7.8.2 The Inference Pipeline: Trainer, Scheduler, Drift, XAI

Beyond the initial deployment, the system runs an inference pipeline. A real-time inference pipeline serves predictions for what is happening now. A trainer component trains and retrains. A scheduler runs retraining at different intervals, using the data accumulated since the last training, to minimize the risk of drift between the data and the model.

One important thing: the model has to be explainable. XAI — Explainable AI — is one of the latest trends in industry: whatever parameters and methods you use should be explainable. Wherever there are issues, wherever there is a trust problem, you check the fairness and the accountability of the results: keep checking the fairness results, detect the drift, monitor the system, and detect data drift or concept drift. When the incoming data distribution is no longer correct compared with the previous dataset — suddenly there is a difference in the data — you need to update the pipeline. You explain the model, you find out whether the accuracy problem comes from the data or from the method, and then you update the model through the model update pipeline.

The inference pipeline has four cooperating parts:

• Inference — serves predictions for what is happening right now (the live API answering each request).
• Trainer — trains and retrains the model as new labeled data accumulates.
• Scheduler — runs retraining at set intervals (nightly, weekly) on the data accumulated since the last training run; regular retraining is the simplest defense against drift between the data and the model.
• Explainability and monitoring — XAI makes the model's decisions explainable; drift detection watches whether the incoming data distribution still matches what the model was trained on. Data drift means the input distribution changed; concept drift means the relationship between inputs and answers changed. When the incoming data is suddenly different from the previous dataset, the pipeline must be updated — and the first diagnostic question is whether the accuracy problem comes from the data or from the method.

7.8.3 The Model Update Pipeline

The alarm manager watches the production model; when drift or a violation is detected, the model update pipeline is launched for a retrain. After the retrain, the model is re-evaluated and, if it passes, re-deployed.

The leftmost part of the architecture is the trigger side: the alarm manager is the watchdog that decides when to retrain. It monitors production signals — accuracy dips, drift metrics, fairness violations, latency or availability problems — and when it identifies a violation, it launches the model update pipeline. The update pipeline then retrains on newer data, the retrained model is re-evaluated (against the same quantitative criteria from 7.3), and only a model that passes is re-deployed. This closes the loop that began in 7.4.4: alarm → retrain → evaluate → deploy.

7.8.4 Three Levels of ML Pipeline Software

Machine learning pipelines have three levels of software. Level one is model engineering: a structured method where you perform feature engineering first, then model engineering. Level two is model training: applying a machine learning algorithm on training data to train the model, which is where you train many models from different categories with standard parameters and compare them. The third level is the deployment and monitoring machinery around the model — the serving, the scheduling, the drift detection, and the update pipeline that keep the trained model working in production.

• Level 1 — Model engineering. The structured method that runs feature engineering first, then model engineering: the features built in 7.5 and 7.6 become the model built in 7.7.
• Level 2 — Model training. Applying the learning algorithm to the training data — the cross-validation comparison from 7.7.4, training many models from different categories with standard parameters and comparing them.
• Level 3 — Deployment and monitoring machinery. Everything around the trained model that keeps it working: serving, scheduling, drift detection, the update pipeline. This level is where the current section lives, and where most production ML teams actually spend their effort.

Real-world connection: these strategies are the daily practice of MLOps platforms — Kubernetes-based serving with canary traffic splits, AWS CodeDeploy and Azure Deployment Slots for blue-green, feature-flag platforms for A/B, and "shadow mode" built into SageMaker endpoints. XAI tooling (SHAP, LIME, and model-card reporting) is mandated in regulated industries for fairness and accountability checks, and the alarm-manager role is filled by monitoring stacks (Prometheus-style metrics plus dedicated drift-detection libraries) that page engineers when a violation appears.

7.9 Model Serving Architecture Patterns

Architecture patterns are reusable solutions to software problems, and ML systems use the same idea. Why do we need patterns? Because the problems we see in the software industry are repetitive in nature: design patterns like the observability pattern or the factory pattern exist to solve a problem that is already solved and repeatable. Patterns are easy to maintain, people understand the pattern, and when other problems come in the pipeline it is easy to detect them.

7.9.1 Why Architecture Patterns

You all have used the iterator pattern without knowing it. A for loop — start i at 1, while i is less than or equal to 10, increment i — appears in C, C++, Java, .NET, and Python. It is needed whenever you have a set of data: a market basket holds 20 items, and to find out how many items there are and what they cost, you check them one by one — take one, scan it, put it in the chart. That is the iterator pattern. Other well-known patterns are model-view-controller (MVC), the broker pattern, and the factory pattern. The question for us: which patterns are useful for machine learning?

The market-basket walkthrough makes the iterator concrete: the basket holds 20 items; the loop starts at item 1, takes it, scans its price, records it in the chart (the running total), moves to item 2, and repeats while the counter stays at or below 20. The pattern is the visiting of every element in a collection exactly once, and it is the reason the same loop shape works in C, C++, Java, .NET, and Python — the languages differ, the pattern is the same. Because problems are repetitive, patterns are easy to maintain: people understand the named pattern immediately, and new problems arriving in the pipeline are detected faster because they map onto known shapes. MVC, the broker pattern, and the factory pattern are the same idea applied to user interfaces, message routing, and object creation.

7.9.2 Pattern Review: Iterator and Publisher-Subscriber

ML is about predicting and relearning. Machines are not gods — they just learn from data, relearn, learn and relearn. For automated systems, AutoML is the sophisticated version of online learning: you give it a model, an interpreter, and input data. For serving the model, there are specific patterns: model as dependency, model as service, pre-compute, model on demand, and hybrid serving.

Why the class's guess — a user graph — seemed right, and why it was rejected: Instagram and Facebook do maintain friendship graphs internally, so "user graph" is a plausible architecture guess. But the pattern they present to the outside world — and the pattern this lecture is about — is publisher-subscriber: content publishers post updates, and subscribers who follow the channel receive them. A YouTube channel is the publisher; following it makes you a subscriber; every new upload arrives as a notification without you asking for it. The replacement mental model: think about the flow of updates (who publishes, who receives), not the network of friends.

ML's own framing of the same idea: "machines are not gods — they just learn from data, relearn, learn and relearn." AutoML is the sophisticated, automated version of online learning: you give it a model, an interpreter, and input data, and it runs the learn-and-relearn loop for you.

7.9.3 Model Serving Patterns: An Overview

The five serving patterns for ML models:

• Model as a service — the model is wrapped and exposed as an independent service.
• Model as dependency — the model is packaged inside the software application.
• Pre-compute — predictions are computed ahead of time and stored.
• Model on demand — the model is loaded at runtime, typically behind a message broker.
• Hybrid serving — a combination, of which federated learning is the famous example.

7.9.4 Model as a Service and Model as Dependency

Model as a service is the common pattern for wrapping an ML model as an independent service. You give your input to a web application, the web application passes the input to a web service that wraps the ML model with an interpreter, the model predicts, and the result comes back to the web application. It is software as a service applied to models: the model might be predicting the gold price, and the web app just consumes the prediction through the API. You use this pattern every day with AI assistants such as GYANMATI and Copilot — you give them input through a web application and they predict the response.

Model as dependency is the opposite packaging: the ML model is considered a dependency within the software application, and the application must use it. It is the most straightforward way to package the ML model — you use an import statement in your code.

The two patterns are opposites in packaging but both simple:

• Model as a service. The model lives behind its own web service: the web application takes the user's input, calls the service, the service's interpreter runs the model, and the prediction returns. Because the model is an independent service, it can be scaled, versioned, and replaced without touching the web application — software as a service applied to models. A gold-price model wrapped this way is consumed through an API by any number of client apps. Everyday examples: GYANMATI and Copilot — you type input into a web application, and a model service predicts the response.
• Model as dependency. The model is packaged inside the application and used directly — typically one import statement in the code. It is the most straightforward packaging, at the cost of coupling: updating the model means rebuilding and redeploying the application.

7.9.5 Pre-compute Serving

Pre-compute serving is for forecasting. You use an already trained model to pre-compute predictions for the incoming data ahead of time, persist those predictions in the database, and serve them directly when asked. Think of a table booking in a hotel: 50 people are coming, so the table is made ready — water, juice, sandwiches — before they arrive. When you invite people home, the water and snacks are ready immediately, and the cooking starts afterward. It is the same idea as pre-fetching in databases: every database — Oracle, Sybase, DB2 — performs pre-fetching, pulling data into a buffer so it is already in memory when you refer to the table. That is the data buffer and data cache — the redo or transaction log buffer — and the optimization pulls the result from what is available. When the request is real time and cannot be pre-computed, you move to the next pattern.

7.9.6 Model on Demand and the Message Broker

Model on demand means the ML model is available at runtime on request. The message broker architecture is typically used for such on-demand model serving. It has two main types of components. The broker component contains the event channels that are utilized within the event flow — those event channels are message queues; a message broker allows one process to write prediction requests to an input queue. The event processor component contains the model serving runtime and the ML model: it connects to the broker, reads the requests in batch from the queue, sends them to the model to make the predictions, and the model serving process writes the resulting predictions to the output queue.

A story makes it concrete: the professor acts as a broker for stories — someone sends him a message that a story needs to be read, and he hands it to the right reader, a different person for a Hindi story, a Tamil story, or another language. You are the mediator. The chat in the class worked the same way: a student asked a question in the chat at 11:19, and the professor answered at 11:29 — the question sat in a message queue, and the answer came after ten minutes based on priority.

The components in one picture: the broker holds the event channels — the message queues — and lets any process write prediction requests to the input queue; the event processor holds the model serving runtime and the ML model, connects to the broker, reads requests in batch, runs the predictions, and writes the results to the output queue. The processor can also route by request type — choosing which model serves a Hindi story versus a Tamil story, or which model scores a credit request versus a fraud request. The broker story is the professor in the mediator role, handing each message to the right reader.

7.9.7 Federated Learning and Hybrid Serving

Federated learning is also called hybrid serving. The idea: individually, anyone can learn, and then they share; finally the results are aggregated and summarized. On the server side, the model is trained only once, with real-world data, and it sets the initial model for each user — a relatively general trained model that fits the majority of users. Then as many models as users exist, in addition to the one held on the server: each user's device keeps refining its own model, and the results are aggregated back. This goes into personalization — a server-side model that is general enough for everyone, plus per-user models that learn from each individual.

Federated learning is hybrid because it combines two patterns at once: the server model serves the general case (model as a service at the center), while each user's device serves its own personalized copy (model on demand / as dependency at the edge). The training flow is what makes it federated: the server trains a general model once on real-world data, ships that model to every user as a starting point, each user's device keeps refining the model on that user's private data — as many models as users exist, on top of the server's copy — and the individual learnings are aggregated back to improve the shared model. The privacy benefit is the point: raw user data never leaves the device; only model updates travel. The result is personalization at scale — a server-side model general enough for everyone, plus per-user models that learn from each individual.

Real-world connection: these five patterns are what production ML platforms implement — SageMaker endpoints and Vertex AI Prediction as model-as-a-service; embedded TensorFlow Lite models in mobile apps as model-as-dependency; recommendation caches and daily forecast tables as pre-compute; Kafka- or RabbitMQ-backed scoring pipelines as model-on-demand; and federated learning frameworks such as TensorFlow Federated for keyboard prediction on phones. The 11:19-to-11:29 chat story is exactly the async pattern behind every request queue in a real serving architecture.

7.10 Data Sources and Data Types

ML systems can work with many data sources, and you have to categorize the data properly. The Netflix case study is the canonical example of how many data sources a recommendation system consumes.

7.10.1 The Netflix Recommendation Case Study

According to the Netflix technology blog, the data sources for their recommendation system are:

• Several millions (billions) of ratings from its members, with more than a million new ratings added every day.
• Popularity metrics, computed in many aspects and at different intervals — hourly, daily, and weekly.
• Clusters of members, formed geographically or by using other similarity metrics — different dimensions over which popularity is computed.
• Stream-related data, such as duration, time of playing, time of device, day of week, and other context-related information.
• The patterns and titles that subscribers add to their queues each day — millions in number.
• All the metadata related to a title in their catalog, such as director, actor, channel, rating, and reviews from different platforms.
• Social data of users, added recently, so that social features related to them and their friends can be extracted to provide better suggestions.
• Search-related text information from Netflix subscribers or members.
• External data, such as critic reviews.
• Other features: demographics, culture, language, and other temporal data used in their predictive models.

Every step has its source. The billion ratings are structured data already stored in databases — raw data, structured. The social media data is semi-structured — that is the HTML kind of data you tokenize and parse. The search-related text is unstructured: a user searches for "what is the movie recently acted by Kamal Haasan or Shah Rukh Khan or Aamir Khan this year" — free text, unstructured.

Exam note: you may get a case study like this in the examination. You may be asked to identify what the source is and what the type of data is — structured, semi-structured, or unstructured.

The case study's lesson is that a single ML system is fed by many sources, and the classification per source:

• Ratings (billions, a million new ones daily) — structured: stored in databases as rows of member, title, score.
• Popularity metrics (hourly, daily, weekly intervals) — structured, computed from raw logs.
• Member clusters (geographic or similarity-based) — structured descriptions of groups.
• Stream-related data (duration, time of playing, day of week) — structured event data.
• Queue patterns and titles (millions per day) — structured user-action data.
• Title metadata (director, actor, channel, rating, reviews) — structured catalog data.
• Social data — semi-structured: the HTML-like data you tokenize and parse.
• Search text ("what is the movie recently acted by Kamal Haasan or Shah Rukh Khan or Aamir Khan this year") — unstructured free text.
• External critic reviews — text, largely unstructured.
• Demographics, culture, language, temporal data — structured feature tables.

7.10.2 Structured, Semi-structured, and Unstructured Data

The professor's own example of the unstructured-to-semi-structured boundary is a Flipkart review mining program: he built a complete program for building recommendation systems for Amazon and Flipkart. For the OnePlus Nord CE 3 Light phone, he went to the Flipkart website, pulled all the reviews, did some scraping and HTML parsing and extraction, and collected all the tokens. Then he built word clouds: a unigram word cloud and a bigram word cloud. The bigrams showed phrases like "camera is good", "battery fast", "mobile quality", "charging nice" — whichever phrases had high frequency tell you what buyers care about. Social media data and HTML data become semi-structured as soon as you parse them; the raw text is unstructured.

7.10.3 Data Categories: First, Second, and Third Party

Data can be categorized by origin: first party data, second party data, or third party data; some is internal, some external, some hybrid. It is similar to intranets: an intranet is only the company, an extranet extends to partners, and the internet is hybrid — part of the data can be accessible to people. There is also user input — data explicitly given by users, often text, images, or tabular data. User data is the uncontrollable kind: the point is made with a trivia question (the longest railway station name in the world is in Wales, UK, and the longest in India is the M.G. Ramachandran Central Railway Station) — users type anything. Some user data exists, some does not. So user input needs heavy-duty validation, and because results are expected immediately after input, it needs faster processing.

• First party data — collected by your own company from your own users and systems (your customers' purchase history).
• Second party data — another organization's first-party data that they share or sell directly to you (a partner's catalog).
• Third party data — aggregated data bought from data brokers or platforms (demographic segments for advertising).

The intranet analogy maps the ownership spectrum: an intranet is only the company (first party), an extranet extends to partners (second party), and the internet is hybrid — partly accessible to the public (third party and open data).

User input is the special, uncontrollable category: data explicitly given by users — text, images, tabular data. The professor's trivia proof that users type anything: the longest railway station name in the world is in Wales, UK, and the longest in India is the M.G. Ramachandran Central Railway Station. Some user data exists, some does not — it arrives unvalidated, in free form, and in unpredictable volumes. Two consequences: user input needs heavy-duty validation, and because results are expected immediately after input, it needs faster processing.

7.10.4 Automated Data, User Behavior, and Regulations

Automated data is the easy kind: logs record the systems — who did what, when. On your own machine you can open the Task Manager and look at app history, or the Event Viewer for application logs; you can export that data and process it. Logs get processed fast — Oracle logs, for example, are captured quickly depending on the checkpoint, in milliseconds. User behavior data is about what people actually do: when their mind changes, they click, they scroll, they zoom, they look for suggestions, they ignore pop-ups — and sometimes a good offer pops up and we miss it. You check whether the user is always clicking, always scrolling, whether they give attention to pop-ups, how much time they spend.

Then come the legal questions: is it legal to capture the user location? Healthcare data and nuclear data have regulations — patient data cannot be exposed, so we have HIPAA compliance; payment card information and data has PCI-DSS, so they mask everything. Regulations control what data you may collect, store, and use.

• Automated data — system-generated logs: who did what, when. The easy kind, because it is regular and structured. On your own machine, the Task Manager's app history and the Event Viewer's application logs are exactly this; you can export and process them. Logs are processed fast — Oracle logs are captured within milliseconds depending on the checkpoint.
• User behavior data — what people actually do, not what they say: clicking, scrolling, zooming, looking for suggestions, ignoring pop-ups. The analyst's questions: is the user always clicking or always scrolling, do they pay attention to pop-ups, how much time do they spend? Note the professor's aside — sometimes a good offer pops up and we miss it — behavior data records what really happened, not what the user intended.
• Regulations — the legal boundary on collection, storage, and use. Healthcare data is protected by HIPAA compliance — patient data cannot be exposed. Payment card data falls under PCI-DSS, which requires masking. The question "is it legal to capture the user location?" is asked before the pipeline is built, not after.

7.10.5 Storage Formats and Data Systems

Data generation systems can produce data in many formats, and those systems could even be the outputs of other systems. The storage options:

• SQL databases.
• NoSQL — strictly no structured query language: key-value stores, document stores (MongoDB is a document database), column databases, and search databases.
• Time series databases — data saved by time; Windy.com uses that kind of data.
• GIS databases — geographic information system databases.
• Graph databases and graph query languages — Neo4j is one of the most important graph databases; when you are trying to create a lot of relations, especially complex and heterogeneous relationships, graph databases are very powerful where SQL does not work effectively.

The rule of thumb: SQL is the default for structured tabular data; document stores like MongoDB handle flexible, self-describing records; time series databases serve sensor and weather data (Windy.com); GIS databases serve location data; and graph databases such as Neo4j win when the value lives in the relationships — complex, heterogeneous relationship structures that SQL joins express poorly.

7.10.6 APIs, Webhooks, gRPC, and GIS Services

Data can be collected through APIs. The PNR status app, the Google live flight tracker that shows exactly where a flight is, and the railway tracker apps all push your input — the location, your longitude and latitude — to an API that pulls the data back. Open source libraries exist, and many SaaS products can generate data. Webhooks are the streaming-friendly variant:

RPC and gRPC are also used — Google has deployed a lot of remote procedures, and you can use gRPC. For geographic information, the Indian government has built a separate service: ArcGIS for India, used for maps and location-based services; its village geo-proximity API lets you integrate geographic data into your own application, and one former student used it in a company for satellite-based maps.

The data-collection toolbox, in one view:

• APIs — request-driven: your app pushes input (a PNR number, your longitude and latitude) to an API, and the API pulls back the data. The PNR status app, the Google live flight tracker, and railway tracker apps all work this way. Open source libraries and many SaaS products generate data through the same channel.
• Webhooks — event-driven, the streaming-friendly variant: instead of asking repeatedly, you keep yourself connected to the service — you stay hooked — and the service delivers events to you automatically. WhatsApp is the everyday example: the connection is already open, and data — when you woke up, when you turned on your mobile, when you saw the messages — is collected and read automatically. The mental model: API = you ask; webhook = the service tells you.
• RPC and gRPC — remote procedure calls let one program invoke another's functions across a network; Google has deployed a great many remote procedures, and gRPC is the modern, high-performance protocol to use.
• GIS services — geographic information services: the Indian government's ArcGIS for India provides maps and location-based services, and its village geo-proximity API lets developers integrate geographic data into their own applications — one former student used it in a company for satellite-based maps.

7.10.7 Third-Party Data and Data as a Currency

Some people share data publicly, some privately — you can have data sharing through third-party data sources. The US government publishes statistics about the labor market; NASA publishes lots of data for researchers; Facebook also shares data. Data is a new currency now — a lot of people on the black market supply large volumes of data to many customers. Data also arrives through message queues, through streaming — subscribe to events, as in the publisher-subscriber pattern — and through the webhook-style hooks described above.

The third-party landscape: open data published for public use — US government labor-market statistics, NASA research data, and data shared by Facebook. The dark side of data's value: data is a new currency, and a black market supplies large volumes of stolen data to many customers. Delivery channels beyond direct download: message queues, streaming subscriptions (the publisher-subscriber pattern from 7.9.2), and webhook-style hooks.

Real-world connection: Netflix's architecture is documented publicly and is the textbook example of multi-source ML; the Flipkart review-mining pipeline is the same NLP pattern used across e-commerce for product intelligence; compliance regimes (HIPAA, PCI-DSS) are enforced with data masking and access logging in every regulated data platform; and the PNR/flight-tracker API model is the everyday pattern behind government and transport data services worldwide.

7.11 Data Ingestion and Pipeline Layers

Data ingestion was covered in an earlier session, so this is the recap plus the architecture. You collect the data — data comes from sources — you do the data collection and ingest the data, and then you decide where it lands: a cloud data warehouse, a cloud data lake, or a lakehouse. Once that is done, you start the data transformation steps.

7.11.1 The Ingestion Path

The full flow runs: data sources to data ingestion, to the ETL or ELT layer, to the cloud data warehouse or data lakehouse, to data transformation, then you build a BI layer, then a data science layer, then data governance — and the governance layer can be an autonomous layer, depending on the company: some companies care about data privacy, some care about governance.

The complete data store and data pipeline flow, in order:

1. Data sources — the systems from section 7.10: databases, APIs, webhooks, logs, user input, third-party feeds.
2. Data ingestion — data collection: capturing the data and moving it off the source systems (the earlier-session material: batch, streaming, push, pull, CDC).
3. ETL or ELT layer — the extract-transform-load (or load-then-transform) processing that shapes the raw data.
4. Cloud data warehouse, data lake, or lakehouse — the landing zone: warehouse for structured analytics, lake for raw files, lakehouse combining both.
5. Data transformation — modeling, cleaning, and feature preparation for the consumers.
6. BI layer — dashboards and reporting for business users.
7. Data science layer — the ML lifecycle from 7.1: features, models, evaluation, deployment.
8. Data governance — the cross-cutting layer that can be autonomous: some companies care about data privacy first, some about governance; either way it sits over the whole flow.

7.11.2 Designing the Data Layers

Exam note: there is a set question from this material — you will be asked to design the layers: do you have layer 1, layer 2, layer 3, then layer 4, 5, 6? You have to design your own layers based on the scenario, and the question is worth about 8 marks. Related questions cover the types of data — discrete data versus continuous data — and the big data characteristics: describing big data, possibly as an easy match-the-following question. The professor's emphasis is strong: remember the entire data store and data pipeline flow — data sources to ingestion, ETL/ELT layer, cloud data warehouse or data lakehouse, data transformation, BI layer, data science layer, data governance. This is very, very important.

Two related exam topics:

• Discrete versus continuous data. Discrete data takes separate, countable values (number of customers, churn yes/no, star ratings); continuous data can take any value in a range (temperature, price, time). The distinction decides which models and chart types apply.
• Big data characteristics. The classic description of big data — possibly as an easy match-the-following question — is the V's: volume (amount), velocity (speed of arrival), variety (types and formats), plus the extended V's (veracity, value, and others) from the earlier sessions on big data.

7.11.3 Deployment Options and Governance

Deploying the pipelines themselves can be done in different ways: code it yourself manually; build or use a single-purpose tool; use a data integration platform; or go with the DataOps approach. The pipeline closes with data contract, lineage, and governance — knowing what data was promised, where it came from, and who governs it.

Four ways to deploy the pipeline:

• Manual coding — build and run the pipeline yourself, script by script. Maximum control, maximum maintenance.
• Single-purpose tools — one tool for one job (a scheduler, an orchestration tool, a transformation engine).
• Data integration platforms — platforms that handle the whole integration workflow with connectors, monitoring, and management built in.
• DataOps approach — applying DevOps discipline (automation, CI/CD, monitoring, collaboration) to the data pipeline, the philosophy the earlier sessions introduced.

The pipeline closes with three governance artifacts: the data contract (what data was promised — schema, quality, freshness), lineage (where it came from — traceable from source to model, the lineage of 7.4.3), and governance (who governs it — ownership, access, privacy, compliance). Every layer of the flow is wrapped by this governance layer, which may be autonomous depending on the company.

Real-world connection: this layered flow is exactly the modern data stack deployed in industry — Fivetran and Airbyte for ingestion, dbt for transformation, Snowflake, BigQuery, and Databricks for the warehouse/lakehouse, Looker and Power BI for the BI layer, MLflow and SageMaker for the data science layer, and Collibra or DataHub for governance — and DataOps platforms orchestrate the whole stack with Airflow or Dagster. The six-layer answer to the exam question is also the standard diagram a data engineer draws on day one of a new architecture.

Exam Guidance Summary

• Question format. The past-year question paper is shared in the course folder; this term's questions will probably be different, but mostly questions give a scenario and ask you to come up with an answer.
• Study emphasis. The entire data store and data pipeline flow — data sources, ingestion, ETL/ELT layer, cloud data warehouse or lakehouse, data transformation, BI layer, data science layer, data governance — is very important; remember it.
• Review advice (Section 7.7). In project reviews, do the literature review and try multiple algorithms before choosing — same discipline helps in the exam.
• Exam logistics. The exam is in person — there is no online option — and international students were given extra options (multiple dates and locations) in the professor's other papers, since flexibility was offered for students affected by travel problems.

The whole lecture compresses into four exam-ready answers:

1. Deployment strategy question — name the strategy (blue-green, canary, A/B, shadow), say which environment it uses, and justify with cost and risk.
2. Layer design question — write the six layers from the scenario: sources and ingestion, ETL/ELT, storage (warehouse/lake/lakehouse), transformation and BI, data science, governance.
3. Case study question — for each data source, name the source and classify the data as structured, semi-structured, or unstructured.
4. Big data question — state the big data characteristics (the V's) and the discrete-versus-continuous data distinction.

Key Industry Applications

• Cloud deployment. Models and pipelines are deployed on AWS, Google Cloud, and Azure, or on websites and mobile apps.
• Model persistence and versioning. pickle and joblib save trained models (model.pkl); GitHub, ClearCase, SCCS, VCS, and RCS are the version-control lineage of model registries; Oracle point-in-time recovery with RPO is the database analog of ML lineage tracking.
• Validation tooling. Scikit-learn provides n-fold cross-validation with cross-validation scores for comparing models.
• Model interchange. PMML, PFA, ONNX, H2O (POJO and MOJO), and TensorFlow and PyTorch (.pt) files make models language agnostic; REST APIs serve offline predictions.
• Recommendation systems. Netflix's recommendation system consumes billions of ratings, popularity metrics, queue patterns, metadata, and social data; the Flipkart and Amazon review mining program shows unstructured review text becoming semi-structured through HTML parsing and tokenization, summarized in word clouds with unigrams and bigrams.
• Data systems. MongoDB as a document database, Neo4j for graph and heterogeneous relationship queries, Windy.com-style time series data, and GIS databases like ArcGIS for India with its village geo-proximity API.
• Real-time data integration. PNR status apps, live flight trackers, and railway tracker apps work through APIs; webhooks keep systems connected so events stream in automatically, the WhatsApp-style hook; gRPC from Google; publisher-subscriber feeds for Instagram, Facebook, and YouTube.
• Compliance. HIPAA governs healthcare data and PCI-DSS governs payment card data, with masking applied to card information.
• Open and third-party data. US government labor statistics, NASA research data, and Facebook data are shared sources; data has become a currency, including on the black market.
• No-code and low-code. AI assistants and no-code platforms such as GINNI AI, ChatGPT, Perplexity, and Copilot build pipelines quickly with prompts.

The industry picture behind the lecture: the same stack appears at every modern data-driven company — a cloud platform (AWS, Google Cloud, Azure) hosting the pipeline; a feature store and model registry keeping features and models versioned; Scikit-learn and cross-validation for honest evaluation; portable model formats (PMML, PFA, ONNX, H2O POJO/MOJO, TensorFlow and PyTorch .pt) moving models between languages; recommendation and review-mining pipelines (Netflix, Flipkart, Amazon) turning structured ratings and unstructured text into predictions; data systems matched to the data (MongoDB, Neo4j, time series, GIS); real-time integration through APIs, webhooks, gRPC, and publisher-subscriber feeds; compliance enforced by HIPAA and PCI-DSS; open data from NASA and government sources; and no-code platforms plus AI assistants (GINNI AI, ChatGPT, Perplexity, Copilot) compressing pipeline construction to minutes.