Machine Learning Foundations for Software Engineering

2.1 Review of Data Science and the Machine Learning Landscape

2.1.1 Data Science Roles and Hierarchy

The data science project ecosystem involves multiple distinct roles, each with a clear responsibility along the data-to-decision pipeline. Understanding these roles is foundational because ML systems fail not just from bad models, but from unclear ownership at each stage of the pipeline.

The data engineer owns the data pipeline — the extract, transform, and load (ETL) process. They build and maintain the infrastructure that moves raw data from its source (databases, APIs, log files, streaming services) into a clean, structured format in a data warehouse (a central repository optimized for analysis). Think of a data engineer as the plumber of the data world: without them, nothing flows. Once data lands in the data warehouse, business intelligence can be performed on it. This is the infrastructure layer that makes everything else possible. If the data pipeline is broken or slow, every downstream role — analyst, scientist, engineer — is blocked.

The data scientist and data analyst handle data cleansing and exploratory data analysis (EDA) — univariate analysis (examining one variable at a time, looking at distributions and outliers), bivariate analysis (examining relationships between pairs of variables, such as correlation), and multivariate analysis (examining interactions among three or more variables simultaneously). The data scientist specifically drives machine learning model development. Depending on the nature of the problem — classification (is this email spam or not?), regression (what will the house price be?), clustering (what groups exist in this customer data?) — they apply multiple algorithms to the dataset and create the predictive model. This is a one-time forecast, a model built in the lab. The data scientist's workflow is typically exploratory and science-like: they experiment with different approaches in computational notebooks like Jupyter, evaluate models on held-out test data, and iterate until they achieve satisfactory accuracy.

The machine learning engineer takes the model from the data scientist and deploys it into a production environment, managing the ongoing lifecycle. This is a critical transition: the model that performed well in a controlled lab environment must now handle real-world data, scale to thousands of concurrent users, and operate reliably 24/7. The ML engineer bridges the gap between experimental data science and production software systems. They handle model serving infrastructure, monitoring, versioning, and retraining pipelines. As the textbook Machine Learning in Production (T1) emphasizes, the skills required for this role extend well beyond model training — they require software engineering discipline, systems thinking, and operational awareness.

This structure is formalized in the data science hierarchy of needs — a framework that mirrors Maslow's hierarchy but for data projects. At the base is data collection (you cannot do anything without raw data). Then the data engineer handles ETL, transforming raw data into usable form. The data analyst and data scientist clean, prepare, and train on the data. And at the top, the ML engineer manages the model in production. Each layer depends on the layers below it. You cannot train a model without cleaned data, and you cannot clean data without collecting it first.

2.1.2 Where Machine Learning Fits

Machine learning is a subset of artificial intelligence. The relationship is straightforward: AI is the broad goal of making machines intelligent, and ML is one approach to achieving that goal — specifically, by learning functions from data rather than hand-coding rules. In the textbook Machine Learning in a Nutshell (T1 Ch03), this is defined precisely: "Machine learning is the subfield of artificial intelligence that deals with learning functions from observations (training data)."

In the early paradigm, we gave labeled datasets to the machine, applied algorithms, and ended up with a model capable of making predictions. The core principle has not changed: classification and regression algorithms applied to data produce a predictive model. A machine-learning algorithm defines the training procedure — how the function is learned from observations. The learned function is called a model. The action of feeding observations into the algorithm to create a model is called model training. The process of computing a prediction for a new input is called model inference.

What has changed is scale and capability. The first generation of ML — roughly from the late 1990s through 2018 — covers basic ML and deep learning. Deep learning describes a specific class of machine-learning approaches based on large neural networks. This era produced models that could classify diabetes risk from a thousand-row Kaggle dataset or predict sentiment from labeled text. The models were probabilistic but narrow in scope. Training was moderate — a thousand-row dataset could produce decent predictions, and you could run it on a CPU, even in Google Colab without a GPU.

Today we have expanded into generative and agentic paradigms that build on these foundations. Foundation models are large-scale models trained on massive, diverse datasets, and they can be instructed to perform specific tasks with prompts rather than task-specific training. This shift — from training custom models to prompting general-purpose models — represents a fundamental change in how ML is practiced, as we will explore in Sections 2.5 and 2.8.

2.2 Code, Data, and Model: The Three Changing Variables

2.2.1 The Fundamental Distinction

One of the most important mental models in software engineering for machine learning is understanding what changes. When you look at a pure software engineering project, code is everything. You have a CI/CD pipeline, you follow principles and practices, you maintain Git repositories — but ultimately, code is the thing. Debugging, bug fixing, unit testing, integration testing, configuration — all revolve around code. The software engineer thinks about code only. The variable deployed into production is the code. If something breaks, you fix the code, run the tests, and redeploy.

When you move to data science, you add a second changing variable: data, alongside code. The schema may change over time. You could be dealing with structured data (rows and columns in a relational database), semi-structured data (JSON, XML), or unstructured data (text, images, audio). You worry about data volume, batch versus real-time processing, data economics. So data science introduces data as an additional dimension of change. The data pipeline itself becomes a first-class concern — if the data is wrong, no amount of good code can fix the output. As the textbook Machine Learning in Production (T1 Ch01) puts it, data scientists "tend to focus on building models but also spend a lot of time on gathering and cleaning data."

When you enter machine learning, you add a third variable: the model. Now you have to handle code, data, and the model. Which algorithms are you using? What is the dataset? How much training data? What experiments are you running? What hyperparameters are you tuning? All three are changing simultaneously. This makes machine learning inherently more complex than either software engineering or data science alone.

Think of it like cooking. A software engineer follows a recipe exactly — same ingredients, same steps, same dish every time. A data scientist is like a chef who also picks and prepares the ingredients (data), and the dish varies depending on what is available. A machine learning engineer is like a chef who invents new recipes (models), selects ingredients (data), and writes cooking procedures (code) — all at the same time. The more things that can change, the harder it is to get a consistent result.

2.3 The Machine Learning Pipeline

2.3.1 The Basic Pipeline Structure

The ML pipeline has these stages, each building on the previous one:

1. Managed Data: Collect data, explore it, cleanse it, prepare it, split into training and test sets. This resonates with the data science portion. The textbook (T1 Ch03) describes this as: "Before the model can be trained, we have to acquire training data (data collection), identify the expected outcomes for that training data (data labeling), and prepare the data for training (data cleaning and feature engineering)." The preparation often includes steps to identify and correct mistakes in the data, fill in missing data, and generally convert data into a format that machine-learning algorithms can handle well.
2. Train Model: Identify the task type — classification, regression, clustering. Do feature engineering (transforming raw data into numerical representations the algorithm can consume). Select appropriate algorithms. For basic ML: Naive Bayes, SVM (Support Vector Machine), Random Forest for classification; linear regression for regression. For deep learning: LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit) for sequential tasks, CNN (Convolutional Neural Network) for vision. For generative: transformer architectures. The textbook (T1 Ch03) emphasizes: "The choice of the machine-learning algorithm can drastically influence model capabilities and various quality attributes."
3. Evaluate Model: Assess performance using metrics appropriate to the task. For classification: confusion matrix, precision (how many predicted positives are actually positive), recall (how many actual positives were caught), F1 score (the harmonic mean of precision and recall). Compare alternatives. A data scientist typically compares multiple model metrics and selects the best model to push to production.
4. Deploy Model: Push to production. This involves wrapping the model in an inference service, setting up the infrastructure to handle real-world requests, and integrating it with the rest of the application. As T1 Ch10 explains, deployment can take many forms: as a library embedded in the application, as a microservice behind a REST API, as batch processing, or as cached/precomputed predictions.
5. Monitor and Improve: The lifecycle does not end at deployment. Continuously monitor outcomes and take steps to improve the model. This includes watching for data drift (when real-world data starts looking different from training data), concept drift (when the relationship between inputs and outputs changes), and performance degradation. The textbook (T1 Ch03) notes: "The entire process of developing models is highly iterative, incrementally tweaking different parts of the pipeline toward better models."

Managed Data → Train Model → Evaluate Model → Deploy Model → Monitor & Improve
     ↑                                                              |
     └──────────────── feedback loop ──────────────────────────────┘

2.3.2 Pipeline Execution: Sequential and Parallel

The pipeline is not rigidly linear. Some stages enforce sequential execution — you cannot do exploratory data analysis without cleaned data; you cannot create a model without understanding the data first. But other stages can run in parallel. You can train multiple algorithms simultaneously (say, Random Forest, SVM, and a neural network), evaluate all of them, and compare before selecting the best model.

There is also an iteration loop: after deployment, monitoring feeds observations back, and the cycle continues. New understanding from production data comes back to influence the entire pipeline. This is the machine-learning flywheel described in T1 Ch01: "with more users, the system can collect data from those users and use that data to train better models, which again may attract more users."

2.3.3 Train/Test Splits and Validation

There are two common splitting strategies. The traditional approach is train and test: take a dataset with, say, 100 rows, allocate 70 for training and 30 for testing. The model learns patterns from the training set, and you evaluate it on the test set — data it has never seen before. This gives you an honest estimate of how the model will perform on new, unseen data. If you evaluate on the training data, you will get inflated accuracy because the model has already memorized those examples — this is called overfitting (like memorizing test answers instead of learning the material).

The second approach is train, test, and validate: the validation set typically comes from real-world data. For example, if you are building a model on labeled hospital data and it performs well on classification metrics, you may still want to validate against actual patient data — the validation set bridges the gap between experimental testing and real-world performance. Which approach you use depends on the application context. The textbook (T1 Ch04) adds an important nuance: "Machine learning should only be used in applications that can tolerate mistakes" — the validation step helps you assess whether the mistake rate is acceptable for your use case.

2.4 ML Domains: Natural Language, Vision, and Speech

2.4.1 The Three Modalities

Humans communicate through language, speech, and vision — and AI models mirror these modalities. The three primary machine learning domains are:

• Language: Natural Language Processing (NLP), where grammar and vocabulary are predefined and well-documented. NLP was the first domain where large-scale models succeeded precisely because the structure of language — grammar rules, vocabulary, syntax — provided a rich, well-defined foundation for learning.
• Speech: Audio analysis, converting spoken language into text (and vice versa). Speech is harder than text because it adds acoustic variability — accents, background noise, speaking speed, emotional tone — on top of the linguistic content.
• Vision: Computer vision, processing images and video. Vision is challenging because images are high-dimensional (a single 1280×720 color image has 2,764,800 pixel values) and the same object can look completely different depending on lighting, angle, occlusion, and scale.

The first transformer models that emerged were large language models precisely because English grammar and vocabulary are well-defined — it was the domain where good synthesis could happen first. As the field matures, the principles and practices of software engineering apply across all three domains. When we discuss requirements engineering, architecture, and design patterns in later sessions, we will draw examples from one or more of these.

2.4.2 Natural Language Processing Tasks

NLP encompasses a wide range of tasks including text classification (is this review positive or negative?), text summarization (condense this document to three sentences), named entity recognition (find all person names and companies in this text), text generation (write a product description), machine translation (translate English to French), language modeling (predict the next word), and question answering (answer a question based on a passage). When ChatGPT responds to a query, it is performing question answering. These tasks are the building blocks, and throughout the course we may explore one or more of them as use cases for applying software engineering principles.

2.4.3 Computer Vision Tasks

Computer vision uses algorithms to process images and videos. Key tasks include:

• Object detection: Identifying and locating objects within an image, drawing bounding boxes around them. Used in surveillance, robotics, and autonomous systems. The textbook (T1 Ch10) describes an OCR model that "might take an image represented as a vector of numbers and return a probability score and bounding box for each detected character."
• Face recognition: The level of advancement is remarkable. Eight to ten years ago, face recognition required 10 to 15 photographs per person to train a CNN model. Today, applications like Digi Yatra at Indian airports require only a single photograph — sometimes not even a straight headshot — and can recognize a person as they walk through the terminal. This leap came from advances in deep learning architectures and massive training datasets.
• Feature extraction: Extracting multiple features from an image — edges, textures, shapes, colors — that can be used as inputs to other models or analysis systems.
• Image classification: The classic dog-versus-cat problem, which can scale to much more complex classification tasks (identifying hundreds of dog breeds, detecting cancerous cells in medical images, classifying satellite imagery).
• Image restoration: Taking images from the 1940s or 1950s and applying pixel interpolation, deblurring, and colorization — applicable to both photos and vintage films. This is a generative task where the model creates plausible visual content to fill in missing or degraded information.

2.4.4 Speech Recognition

Automatic speech recognition (ASR) converts spoken language into text. It powers transcription services, call center analytics, Siri, and Alexa. This is the domain that takes audio waveforms and produces linguistic output. The textbook (T1 Ch01) uses a transcription startup as its motivating example: the founder "has spent the last couple of years at a university pushing the state of the art in speech recognition technology" using transfer learning to specialize models for specific domains — medical conversations, academic conferences, programming meetups. This scenario illustrates both the potential and the production challenges of speech ML systems.

2.4.5 Sophia Robot: Multimodal Integration

The Sophia robot by Hanson Robotics is a case study in multimodal AI integration — a system that uses all three modalities simultaneously. It uses computer vision for facial tracking and object recognition, NLP for understanding and generating language (connected to an LLM behind the scenes), and speech recognition for real-time conversation. It can tell jokes, teach STEM courses, and walk — a locomotive function alongside cognitive ones. Hanson Robotics is at least a 15-20 year old company, and the compute powering Sophia is inbuilt into the robot itself, likely with its own GPU infrastructure, possibly with cloud communication.

2.5 Foundation Models and Large Language Models

2.5.1 What Foundation Models Are

Foundation models are large-scale machine learning models trained on massive, diverse datasets — essentially all the data available on the web. They serve as a base from which specialized models can be built. The professor's analogy captures this perfectly: constructing a house. You lay a foundation, and on top of it you can build any number of stories — one story for language, another for vision, another for speech. The foundation itself is general-purpose; the stories are specialized.

The textbook (T1 Ch03) formalizes this: "Rather than learning a model for each task, organizations train very large general-purpose models, called foundation models as an umbrella term for large language models and other large general-purpose models. Those foundation models can be instructed to perform specific tasks with prompts." This is a fundamental shift in how ML is practiced — from training custom models to prompting general-purpose ones.

In some technical literature, "foundation model" and "LLM" are used interchangeably, but there is a meaningful distinction. A large language model (LLM) is a subset of foundation models that is specialized for language tasks. When OpenAI launched ChatGPT in November 2022 with GPT-3.5, it was an LLM — it only handled text. It could summarize, generate, translate — but it was purely language-oriented. Trained on approximately 175 billion parameters, it excelled at language tasks. A parameter is a learned constant within the model's internal structure — the values that were identified during training. The "large" in LLM is not just a label; it reflects the fact that complex language understanding capabilities only emerge beyond certain parameter thresholds, often in the billions.

Foundation Models (general-purpose, trained on diverse data)
├── Large Language Models (LLMs) — specialized for text/language
│   └── Examples: GPT-3.5, GPT-4, Claude, Llama
├── Large Vision Models — specialized for images/video
│   └── Examples: DALL-E, Stable Diffusion
└── Speech Models — specialized for audio
    └── Examples: Whisper, Sarvam AI models

A foundation model, by contrast, is trained on diverse data across modalities. On top of it, you can build an LLM for language, a large vision model for images and video, or a speech model for audio. In the last two to three years, research has expanded massively into vision, resulting in large vision models that cater to image and video tasks while being powered by the same foundational training. The textbook (T1 Ch03) describes two strategies for specializing foundation models: (1) fine-tuning — training a copy of the model with custom data — and (2) in-context learning — providing additional information or instructions as part of the prompt.

2.5.2 Popular Foundation Models

Examples include GPT-3.5, GPT-4, GPT-4o, GPT-5.5, Claude Opus, and Llama models from Meta. The exact classification of each as open-source or commercial varies, but all represent the foundation model paradigm. Some key distinctions:

• GPT series (OpenAI): Closed-source, API-based. GPT-3.5 (~175B parameters) was text-only; GPT-4/4o (~330B parameters) expanded to multimodal (vision + text).
• Claude (Anthropic): Closed-source, API-based. Known for longer context windows and safety-focused training.
• Llama (Meta): Open-source. Organizations can download, fine-tune, and host locally — important for confidentiality constraints where data cannot leave the organization's environment.

The textbook (T1 Ch03) notes: "Usually, third-party foundation models are used over an API, but some (open-source) foundation models can also be hosted locally." This API-vs-local distinction has major implications for cost, privacy, latency, and customization — we will explore it further in Section 2.9.

2.6 The AI Engineer Role

2.6.1 Skills and Responsibilities

The AI engineer — sometimes called generative AI engineer — has emerged as one of the most in-demand roles. For software engineers with years of SDLC experience who want to transition into AI/ML, this role is the natural bridge. The skill set combines traditional software engineering with ML-specific knowledge:

• Solid software engineering skills: The course's core purpose — applying SE best practices refined over four to five decades to the ML domain. This includes requirements engineering, architecture design, testing, deployment, and maintenance. The textbook (T1 Ch01) emphasizes that "building an accurate model with machine-learning techniques is already difficult, but building a product and a business requires also collecting the right data, building an entire software product around the model, protecting users from harm caused by model mistakes, and successfully deploying, scaling, and operating the product."
• Python and SQL: The primary programming and query languages for ML work, plus familiarity with vector databases (databases optimized for storing and searching high-dimensional vectors, essential for RAG systems). Python dominates because of its ecosystem — libraries like scikit-learn, TensorFlow, PyTorch, and Hugging Face Transformers all have Python-first APIs.
• CI/CD: DevOps has now branched into DataOps and MLOps — specialized CI/CD flavors for data science and ML experiments respectively. MLOps extends traditional CI/CD to handle data versioning, model versioning, experiment tracking, and automated retraining pipelines. The textbook (T1 Ch01) describes MLOps as "efforts to automate machine-learning pipelines and make it easy and reliable to deploy, update, monitor, and operate models."
• Git/GitHub: All ML code, even for microservices architectures, lives in version control. This includes not just application code, but also model configurations, training scripts, data preprocessing pipelines, and experiment configurations.
• LLMs and transformer knowledge: Understanding the models you work with — how transformers process attention, what tokenization does, how context windows work, what temperature and top-p control. You do not need to implement a transformer from scratch, but you need to understand the architecture well enough to make informed decisions about model selection and prompting.
• Domain knowledge: RAG architecture, prompt engineering, foundation models, fine-tuning, MCP (Model Context Protocol) — and the list grows continuously. The field moves fast, and continuous learning is not optional.

Domain Knowledge (RAG, prompt engineering, fine-tuning)
├── LLM & Transformer Understanding
├── MLOps (CI/CD for ML)
├── Python + SQL + Vector Databases
└── Solid Software Engineering Fundamentals (requirements, architecture, testing, deployment)

2.6.2 Industry Demand

A sample job description from Accenture for an AI/ML engineer highlights that cloud knowledge has become essential. Engineers must be able to leverage cloud AI services into the applications they build — NLP services, computer vision services, speech services. The ability to integrate pre-built cloud AI capabilities alongside custom models is now a baseline expectation. While each organization's JD differs, the core requirements are converging around this combination of SE discipline and ML expertise.

The textbook (T1 Ch01) lists the broader skill ecosystem needed for production ML systems: "We need business skills to identify the problem and build a company. We need domain expertise to understand the data and frame the goals for the machine-learning task. We need the statistics and data science skills to identify a suitable machine-learning algorithm and model architecture. We need the software engineering skills to build a system that integrates the model as one of its many components." The AI engineer role focuses primarily on the software engineering and integration side of this ecosystem, working closely with data scientists who build the models.

2.7 Software Engineering vs Machine Learning: Fundamental Differences

2.7.1 Deterministic vs Probabilistic

Software engineering is deterministic. If you write an if-else statement, the logic executes a specific branch. A function behaves the same way every time given the same inputs. It is rule-based. Given the same input, you always get the same output. This predictability is what makes traditional software testable — you can write a test case with known inputs and assert exact expected outputs.

This is why AI models today — Claude, Codex, GPT — are so good at code generation: coding is structured, rule-based, and well-documented. It was one of the first tasks to be almost entirely automated by modern models. The rules are explicit, the syntax is constrained, and the expected behavior can be verified.

Machine learning is probabilistic. Take a trained model given an image of a pug dog: it will not say "pug 100%." It will say "pug 99%, terrier 0.07%, other dog 0.01%." Every answer is a probability distribution, and the highest-probability output is what gets returned. Even a RAG-based chatbot answering questions from a set of slides is producing probabilistic answers — when multiple slides contain similar content, the retrieval ranks them and the LLM selects based on probability.

The textbook (T1 Ch01) frames this as a shift "from deductive reasoning (mathy, logic-based, applying logic rules) to inductive reasoning (sciency, generalizing from observation)." In traditional SE, we can say whether a component is correct against its specification. In ML, "we can no longer say whether a component is correct, because we do not have a specification of what it means to be correct, but we evaluate whether it works well enough (on average) on some test data."

2.7.2 Specification and Requirements

In software engineering, specifications are clear and specific. You can write a flowchart: step one, step two, step three. The requirement is well-defined, and the implementation follows directly. The textbook (T1 Ch01) provides a concrete example: a compute_deductions function with a docstring that points to the tax code — a developer can implement this function according to the specification without needing to understand the rest of the system.

In machine learning, specifications are inherently vaguer. Consider a requirement like "detect objects visible in the image." The image might contain a tree, a person, a dog, a bicycle — or it might contain things your model was never trained to recognize. A dog that was absent from the training dataset cannot be detected. The textbook (T1 Ch01) captures this precisely: "With machine learning, we have a hard time coming up with good specifications. We can generally describe the task, but not how to do it, or what the precise expected mapping between inputs and outputs is."

The failure of many ML projects traces back to poorly defined requirements exactly because data science and ML are exploratory by nature, while SE is more prescriptive. As the professor warns, this is one of the most common reasons ML projects fail — the requirements are vague, the success criteria are unclear, and nobody agrees on what "good enough" means. Requirements engineering for ML is a critical challenge, and from the third session onward, we examine how to capture functional and non-functional requirements for ML applications properly.

2.7.3 Summary of Contrasts

2.8 The Three AI Paradigms: Predictive, Generative, and Agentic

2.8.1 Predictive AI

Predictive AI uses past data to predict future outcomes. This is the oldest and most established paradigm. Take a labeled Kaggle dataset — the Pima Indian Diabetes dataset with 1,000 rows, split 50-50 between diabetic and non-diabetic patients. Train on multiple algorithms (SVM, decision trees), and the resulting model can take a new patient's parameters — age, blood sugar, BMI — and classify them as diabetic or non-diabetic. The model is probabilistic — it outputs a probability — but its purpose is prediction.

The predictive AI era spans roughly from the late 1990s through 2018, encompassing basic machine learning and deep learning. Models like GPT-2 in 2020, which could perform next-word prediction from a 10-word prompt without transformer architectures, represent the peak of this paradigm. Training is moderate — a thousand-row dataset can produce decent predictions. You can run it on a CPU, even in Google Colab without a GPU. The textbook (T1 Ch04) describes when predictive ML is appropriate: "problems that are intrinsically hard, big, and time-changing" — tasks where hand-coding rules would be infeasible.

2.8.2 Generative AI

Generative AI creates new content based on learned patterns. Since Google's seminal "Attention Is All You Need" paper in 2018, transformer models — BERT, GPT, ChatGPT — have defined this paradigm. Give a thousand-line Shakespeare poem as input, and the model generates fifty new lines in Shakespeare's style. The model has learned the patterns of Shakespeare's language — his vocabulary, rhythm, imagery — and can produce new content that follows those patterns.

Under the hood, generative AI uses prediction — next-word prediction, sentence prediction. The same mechanisms that power predictive AI are the engine, but the scale and purpose differ dramatically:

The professor's book analogy captures the difference beautifully: predictive AI reads one 20-page book and answers questions from it. Generative AI reads 20 books, each 20 pages. The grammar, vocabulary, and understanding are vastly deeper. Predictive AI learns from a narrow dataset; generative AI learns from a broad corpus that spans much of human knowledge.

2.8.3 Agentic AI

Agentic AI represents the current frontier. An AI agent is an autonomous program with a specific purpose — it has memory, access to tools, and reasoning capability. Agentic AI is the broader framework in which multiple agents coordinate to accomplish complex tasks.

ChatGPT is an example of generative AI: you give a prompt, it reasons internally, possibly uses tools, and returns a response. Claude Opus or Codex running in VS Code is an example of agentic AI: behind the scenes, it runs multiple coordinating agents that together produce the code or response you see.

Key components of agentic AI:

• Memory: Storing context across interactions — what the user asked before, what results were found, what decisions were made. Without memory, each interaction starts from scratch.
• Tools: Web search, API calls, code execution, database queries. Tools extend what an agent can do beyond pure language generation.
• Reasoning: Planning and decision-making across agents — deciding which agent to invoke, how to combine results, when to ask for clarification.

2.9 The Generative AI Stack and RAG

2.9.1 The Stack Layers

Building a generative AI application involves a stack of components, any of which can be included or skipped depending on requirements. The stack is modular — you can start with just an API call and add layers as your needs grow:

1. Compute: CPU, GPU, or TPU infrastructure. This is the hardware foundation. With an 8 GB RAM laptop and 50 GB free disk space, you can download and run models like Gemma 2B or Mistral 7B locally using Olama. With a GPU, you can run larger open-source models from Meta or OpenAI. The compute requirement scales with model size — a 2B parameter model runs on a laptop; a 70B parameter model requires a high-end GPU.
2. Foundation models: Pre-trained models accessible via API or downloaded locally. These are the base. You do not train these yourself — you use models that organizations like OpenAI, Anthropic, Google, or Meta have trained at enormous cost.
3. Model hubs: Hugging Face is the primary platform for discovering, downloading, and sharing models. Think of it as the GitHub of ML models — it hosts hundreds of thousands of models, datasets, and spaces (demo applications).
4. Fine-tuned models: Models adapted for specific tasks. Fine-tuning takes a pre-trained model and further trains it on task-specific data. For example, a bank might fine-tune a language model on 40,000 human-verified FAQs about loan processing and account services.
5. Application development: LangChain for orchestration (coordinating multiple models and tools), vector databases (ChromaDB, Pinecone, FAISS) for retrieval (finding relevant information from your documents), and UI libraries (Streamlit, Gradio) for the front end (the interface users interact with).

User Interface (Streamlit, Gradio)
├── Application Layer (LangChain, DSPy)
├── Vector Database (ChromaDB, Pinecone, FAISS)
├── Fine-tuned Models / Prompted Foundation Models
├── Model Hub (Hugging Face)
├── Foundation Models (GPT, Claude, Llama, Gemma)
└── Compute (CPU, GPU, TPU — local or cloud)

2.9.2 Two Ways to Run ML Programmatically

There are two fundamental approaches to using foundation models, each with distinct trade-offs:

1. API-based: Create an API key from OpenAI for GPT-5.5, mini, or nano; from LiteLLM to access multiple models through one key; or from AWS Bedrock to use whatever models it provides. The same applies to Gemini or any other provider. The model runs remotely on the provider's infrastructure; you pay per token or request. Advantages: No GPU needed, instant access to state-of-the-art models, automatic scaling. Disadvantages: Data leaves your environment (privacy concern), ongoing per-token costs, dependency on provider availability, limited customization.
2. Local download: Download open-source models to your own infrastructure. This is especially important when you have confidentiality constraints — you run the model locally and your data never leaves your environment. Almost every major player now has open-source models. Advantages: Full data privacy, no per-token costs after setup, full customization. Disadvantages: Requires GPU hardware, requires ML infrastructure expertise, models may be smaller/less capable than commercial API models.

2.9.3 RAG Architecture

Retrieval Augmented Generation (RAG) lets you build a model that answers questions based on internal documents or context that the model was never trained on. This solves the fundamental limitation of foundation models: they do not have access to your private or proprietary data.

User Query → Embedding → Vector Search → Retrieved Chunks → LLM + Query + Chunks → Answer

2.10 From Prototype to Production: ML Systems Engineering

2.10.1 The Reality Gap

As a researcher, you may develop a brilliant domain-specific speech recognition model using transfer learning. It performs beautifully on medical conversations, academic talks, and technical meetups — in your lab. You have identified a market need: manual transcription is error-prone, expensive, and slow; existing automated tools have poor accuracy on technical terms. You feel ready to launch.

The textbook (T1 Ch01) uses this exact scenario as its motivating example — a researcher named Sidney who "has spent the last couple of years at a university pushing the state of the art in speech recognition technology" and decides to commercialize it. The challenges Sidney faces are universal to ML products.

Then reality hits. The technical challenges include:

• Noisy real-world data: In the lab, your speech samples were noise-free. In the wild, a politician speaking at a rally has hundreds of people in the background — and your accuracy plummets. The textbook notes: "audio files received from customers are often noisier than those used for benchmarking in academic research."
• Performance constraints: In the lab with your GPU setup, the model returned results in seconds. With a thousand concurrent users in production, it takes minutes. Users expect real-time response. The textbook adds: "customers get impatient if their audio files are not transcribed within 15 minutes. Worse, live captioning needs to be essentially instantaneous."
• Live captioning difficulties: Real-time processing at scale introduces latency and synchronization issues that small-scale testing never revealed.
• Escalating costs: Cloud computation, training, inference on AWS or Azure — the costs keep climbing. LLM API usage and GPU infrastructure expenses multiply rapidly. The textbook describes this vividly: "Attempts at using new large language models to improve transcripts and adding new features, like automated summaries, run quickly into excessive costs paid to companies providing the model APIs."
• Scalability issues: Handling user growth strains every component.

Beyond technical issues, engineering and operational challenges emerge:

• Application development: You cannot just release a model. You need mobile apps, web apps. Users must authenticate, make payments — Razorpay or Stripe integration is required. The textbook notes the team "now needs to build a website where users can upload audio files and see results — with which the team members have no experience and which they do not enjoy."
• Pipeline fragility: A speech API library used today may become outdated. Dependencies shift. "Nobody has updated the Tensorflow library in almost a year, out of fear that something might break."
• Monitoring and fairness: If the model translates speech containing racial discrimination or abusive language into another language, you face serious liability. Continuous monitoring is non-negotiable, especially for medical or legally sensitive translations. The textbook warns: "One customer sent a complaint with several examples of medical diagnoses incorrectly transcribed with high confidence, and another wrote a blog post about how the transcriptions for speakers with African American vernacular at their conference are barely intelligible."
• Dialect issues: Regional dialects that your model was never trained on degrade translation quality.

The key insight: moving from a research prototype to a production system requires significant engineering beyond machine learning. This is where software engineering principles become essential.

2.10.2 The Model Is Only One Component

In a production ML system, the model itself is a small fraction of the whole. Consider an object detection application. The system must include:

• A user interface (Streamlit, Gradio)
• User management and authentication
• Photo upload capability with cloud storage (S3)
• Payment processing
• One or more databases (relational, NoSQL, vector)
• Cloud processing infrastructure
• Logging and monitoring

The non-ML components — UI, user management, payment, database, cloud, logging, monitoring — vastly outnumber the ML component. When you write a model in Google Colab, you are only building a small piece of the system.

    ┌─────────┐
    │  Model  │  ← This is what data scientists build
    │ (small) │
    ├─────────┤
    │ Serving │  ← Inference service, API, load balancing
    ├─────────┤
    │  Data   │  ← Data pipelines, feature stores, storage
    ├─────────┤
    │  Ops    │  ← Monitoring, logging, alerting, CI/CD
    ├─────────┤
    │  App    │  ← UI, auth, payments, business logic
    └─────────┘
      (massive)

This is captured in the 2015 paper "Hidden Technical Debt in Machine Learning Systems", which famously illustrates that the ML code itself is only a tiny central box in a much larger diagram. The surrounding blocks — serving infrastructure, monitoring, machine resource management, data verification, feature extraction, data collection, configuration — dwarf the ML code. The serving infrastructure alone is typically the largest block. The textbook (T1 Ch01) echoes this: "the machine-learned model is clearly the essential core of the entire product. Yet...there are many more concerns beyond training an accurate model."

2.10.3 The Production ML Pipeline

The pipeline transforms when a model moves from experimentation to production:

Experimentation phase (static): Labeled data → cleaning → feature engineering → train/test split → model training → evaluation. Everything is offline and controlled. You choose the data, you control the environment, you evaluate at your convenience.

Production phase (dynamic): New challenges arise that the experimentation phase never encounters:

• Re-evaluate the model on live production data (not static collected data) — production data has different distributions, edge cases, and noise patterns
• Continuously monitor 24/7 — the model must work at 3 AM on a Sunday, not just during your working hours
• Select production data for periodic retraining — the world changes, and the model must keep up
• Update the model regularly — deploy new versions without downtime
• Redeploy — roll back if something goes wrong, roll forward when fixes are ready

The production environment introduces new data distributions, concept drift (the relationship between inputs and outputs changes over time), and operational concerns that the experimentation phase never encounters. Software engineering practices — CI/CD adapted as MLOps, containerization with Docker and Kubernetes, monitoring and observability — are the tools that address these challenges.

Real-world: The CI/CD systems, Git workflows, Docker, and Kubernetes that power Swiggy, Zomato, Ola, and Uber — food delivery, ride-hailing, e-commerce — were originally built for pure software applications. Today, these same infrastructure components are leveraged to deploy and manage ML systems at scale. The principles that made traditional software deployment reliable are being adapted for the unique demands of machine learning.

Exam Guidance Summary

• This is a conceptual, theory-oriented course — no mathematical derivations or numerical problems
• From the fourth session onward, webinars and in-class demos provide experiential learning components
• Post-midterm: observability tools, evaluation frameworks, and best practices for production ML
• Sessions 14 and 15 are dedicated entirely to agentic AI with hands-on work
• The course structure follows the software engineering lifecycle — requirements engineering (session 3), architecture, design patterns, implementation, testing, deployment, monitoring — applied to ML systems
• Key textbook reference: The course draws from foundational SE for ML literature

Key Industry Applications

• ChatGPT / GPT series: Transitioned from LLM (GPT-3.5, text-only) to multimodal (GPT-4/4o, supports vision). GPT-3.5 trained on ~175B parameters, GPT-4o series around ~330B parameters. Uses agentic AI web search for real-time data gaps. Launched November 2022, trained up to September 2021 data — illustrates the stale-data problem that agentic search solves.
• Claude Opus / Codex: Coding-focused agentic AI used within IDE environments (VS Code). Demonstrates agentic AI in practice — multiple coordinating agents producing code responses.
• Digi Yatra: Face recognition at Indian airports — single-photo recognition replacing the old 10-15 photo requirement. A concrete example of computer vision advancing from requiring many training samples to few-shot recognition.
• Sophia Robot (Hanson Robotics): Multimodal AI integrating NLP, CV, and speech with inbuilt GPU compute. 15-20 year old company. Case study in orchestrating multiple AI modalities simultaneously.
• Sarvam AI: Speech-focused models — machine translation, speech generation. Domain-specialized foundation model for the speech modality.
• Hugging Face: Model hub for sharing and discovering pre-trained models. The "GitHub of ML" — hosts hundreds of thousands of models, datasets, and demo spaces.
• LangChain + ChromaDB + Streamlit: A complete open-source stack for building RAG applications. LangChain handles orchestration, ChromaDB provides vector storage, Streamlit delivers the user interface.
• Olama: Running models like Gemma 2B, Mistral 7B locally on CPU. Enables local model deployment for confidentiality-constrained environments.
• LiteLLM / AWS Bedrock: Unified API access to multiple foundation models through a single key. Simplifies multi-model architectures.
• RunPod: GPU cloud partner (under negotiation for course labs). Illustrates the compute infrastructure layer of the generative AI stack.
• Accenture AI/ML engineer JD: Cloud AI services integration is a baseline requirement. Demonstrates industry convergence on the AI engineer skill set.
• "Hidden Technical Debt in Machine Learning Systems" (2015): Foundational paper on the infrastructure surrounding ML code. Key insight: ML code is a tiny fraction of the total production system.
• "Attention Is All You Need" (2018, Google): The transformer paper that launched generative AI. Introduced the self-attention mechanism that powers all modern LLMs.
• "Cloud Native Artificial Intelligence": Reference paper on AI paradigms and cloud-native approaches to ML systems.