Big Data Ecosystems, Cloud Platforms, and LLM Pipelines

# Big Data Ecosystems, Cloud Platforms, and LLM Pipelines

This session ties the big data story together and opens the door to large language models. It starts with a hands-on path from a bare virtual machine to a running single-node Hadoop cluster, then recaps the distributed systems core — HDFS, MapReduce, and YARN — and tours the Apache Hadoop ecosystem with its cloud equivalents (AWS and Google) and cloud-native platforms (Snowflake, BigQuery, Databricks). The second half turns to LLM pipelines: the foundations (memory, corpus, text preprocessing, embeddings), the three pipelines (data, inference, Ops), the five training stages, evaluation and feedback with PSI and KL divergence, testing, and deployment. Throughout, the professor pairs every technical idea with an everyday analogy — HDFS as the body, MapReduce as the brain, fault tolerance as a mother cooking with whatever is in the kitchen, and retraining as writing the multiplication table five times.

13.1 Setting Up a Big Data System: From Virtual Machine to Hadoop

There are two paths to a working big data system:

• Local path: run everything yourself on a virtual machine (or WSL).
• Cloud path: rent a ready-made cluster through Ambari, the Hadoop cluster management tool.

Both end at the same place: a Hadoop system where you can store files in HDFS, run MapReduce jobs, and watch the cluster work.

13.1.1 Choosing a Virtual Machine and Operating System

A virtual machine (VM) is a computer simulated in software: it borrows CPU, memory, and disk from your real machine and runs a full operating system inside a window, isolated from the rest of your setup. Any VM tool works — download VirtualBox or VMware Workstation. Then download an operating system image: Ubuntu, Red Hat Linux, or any other Linux distribution. The image is just a file; the VM tool boots it.

Before starting the machine, allocate its hardware — this is the machine's "shopping list":

Start the virtual machine and the Linux system appears. Log in with the default account vboxuser (password changeme), then add that account to the sudo users so you have administration rights — sudo lets you run commands as the system administrator, which installing software requires. From there you are inside your own Unix system: a terminal plus a browser. You can also do the entire setup through WSL — the Windows Subsystem for Linux — which gives you a Linux terminal directly inside Windows without a separate VM window.

Exam note: none of this setup is graded material, but following the guide once makes the ecosystem topics concrete — you will see the name node and data nodes you study later as real running processes.

13.1.2 Installing Java, SSH, and Hadoop

Hadoop is built in Java, so the first software to install is OpenJDK — the open-source Java runtime. Then set up SSH for passwordless authentication: SSH (Secure Shell) is the standard way to log into a remote Unix machine, and you do not want to type a login password again and again while running cluster commands. The setup has three moves:

1. Generate a key pair — a private key (kept secret, on your machine) and a public key (shared).
2. Register the public key under the Hadoop user account (the guide uses a dedicated hadoop user).
3. Test it: after registration, a plain ssh localhost drops you straight into your big data system with no password prompt.

Next, download the Apache Hadoop software — use wget with the link in the guide — and extract the archive with gunzip (the download is a gzip-compressed tar file). The guide's instructions are very clear; each command has its purpose and can be copied directly.

After extraction, six files matter:

Edit them with any Unix editor; the guide recommends nano, which is the easiest for beginners. In .bashrc you set the environment — give the path to the Java you installed, copy the entries, and you are set.

13.1.3 Name Node, Data Nodes, and Starting the Cluster

This ties back to the earlier session's discussion of the name node and data nodes. Recall the roles:

• The name node is the master node, the metadata node: it collects all the information about all the data nodes — which files exist, and which data nodes hold each block. It does not store the data itself; it stores the map of where the data lives.
• Data nodes are the children, the slaves — they hold the actual data blocks. Picture a tiny three-node cluster where three students are the data nodes and the professor is the master node: the professor keeps the class roster (metadata), and each student holds a piece of the material (data).

A machine can host both a name node and a data node at once, but for convenience we create a separate name node.

Starting the cluster is a short sequence:

1. Create directories for the name node and the data node — the on-disk homes where each will keep its state.
2. Understand YARN — Yet Another Resource Negotiator — which handles resource management: it decides which running job gets which slot of memory and CPU. Everything here is Java.
3. Format the disk — because the disk is basically blank, you format it, create the name node, and create the cluster. Formatting initializes the file system's metadata structures.
4. Run start-dfs.sh to start the distributed file system: the name node runs and the data node runs.
5. Run start-yarn.sh to start the resource manager and node manager.

Type the jps command — "Java process status" — and it lists every running Java process: the name node, the secondary name node, the data node, the node manager, and the resource manager. All five processes present means your cluster is up. Then open Hadoop in the browser through its URL and you can browse the file system graphically.

13.1.4 Running a Word Count Job

With the cluster up you can run HDFS commands — for example hadoop fs -cat — to view any kind of data stored in the distributed file system. The demonstration job was word count, the classic MapReduce program.

Real-world connection: in industry, people working on AWS clusters spend their day doing exactly this — monitoring whether nodes have started, watching where the data is going, checking the disk. Hive and the other libraries in the ecosystem all run on top of the big data system this way; the cluster is the shared foundation they all sit on.

13.1.5 Student Questions and Answers

13.2 Distributed Systems Recap: HDFS, MapReduce, and YARN

13.2.1 Shared Resources and the HDFS–MapReduce Analogy

The recap frames a distributed system as parallel processing: parallel processors working together, sharing one or more resources. Recall the three sharing arrangements covered earlier:

• Shared memory — many processors read and write the same memory bank; fast, but hard to scale and a single physical point of failure.
• Shared disk — processors share a common disk or storage system; slower than memory, but storage is easier to scale out.
• Shared CPU — processors share compute; the workloads themselves are distributed across machines.

There is a standing question to think over: which sharing arrangement do you choose, and why? The answer depends on the workload — how much data moves between processors, how fast it must flow, and how much failure you can tolerate. Memory sharing is fastest, disk sharing survives bigger failures, and CPU sharing spreads the work.

The professor's picture for the big data stack:

• HDFS is like the entire body — all the data lives there. It is the shared storage that every tool reads from and writes to.
• MapReduce is the brain and the intelligence — it takes the data stored in the body and turns it into answers.

A second picture makes the distribution explicit: if you are carrying two heavy suitcases, you distribute them between your right hand and your left hand according to your capability — one suitcase to the stronger hand, one to the weaker. That distribution of load is what MapReduce does with work: it splits a job across many workers, hands each worker a piece matched to its machine, and collects the results. The suitcases map to data partitions, your two hands map to worker nodes, and your judgment about which hand can carry which suitcase maps to the framework's scheduling.

Exam note: the MapReduce material — horizontal and vertical scaling, how the file system gets distributed, data replication across node A, node B, node C, and the MapReduce flow — is easy to study and score well on; it is a strong return-on-effort topic. The professor promised to share the distribution notes, and they are worth having.

13.2.2 YARN: The Resource Negotiator

Between the running workloads and the cluster's resources sits YARN — Yet Another Resource Negotiator. A busy cluster always has more jobs asking for resources than resources to give: which job is better, which one is easy, what is running, what should run? The negotiation happens automatically inside YARN. It is the cluster's traffic controller: it takes the requests from the applications, looks at what memory and CPU the nodes currently have free, and hands out the capacity — deciding which job runs now and which waits.

YARN is separate from the processing model itself. MapReduce is distributed processing that you can implement in Python, Java, C, .NET, or anything else; YARN is the resource layer that decides where and when that processing gets to run. Keeping the two ideas apart — what the job computes versus who gives it the machines to compute on — is the key mental model for the whole ecosystem.

13.3 The Big Data Ecosystem: Tools on Top of Hadoop

Hadoop alone only stores (HDFS) and processes (MapReduce) data. The real power is the ecosystem: a family of open-source tools that sit on top of the cluster and handle every job a data platform needs — getting data in, querying it, scripting jobs, running machine learning, streaming, coordinating components, and indexing search. The professor tours the family one role at a time. HDFS is the body these tools all plug into.

13.3.1 Ingestion Tools: Sqoop and Flume

Before anything can be analyzed, data must get into the system. Two tools do the intake:

• Sqoop pushes structured data into HDFS — typically rows from relational databases (MySQL, Oracle, PostgreSQL). Its name echoes "SQL-to-Hadoop": you give it a database table and a target directory, and it moves the data in bulk.
• Flume handles streaming data and real-time data — log lines, sensor events, web clicks arriving continuously — and can also push structured data into HDFS.

Both exist for one reason: to get external data into the body of the system. Sqoop is the batch courier for tidy tabular data; Flume is the live feed for data that never stops arriving.

13.3.2 Processing and Query: MapReduce, Spark, Hive, HBase

Once data is inside, four tools cover processing and query:

• MapReduce provides distributed processing — the batch engine studied in depth in the previous session.
• Spark is the in-memory data flow engine — very, very fast, because anything processed from memory is much better than anything touched through disk. (Section 13.6 compares the two engines in detail.)
• Hive is the relational database on Hadoop — a SQL database on Hadoop. You write familiar SQL and Hive translates it into MapReduce or Spark jobs running in the background.
• HBase is the NoSQL database on Hadoop — a key-value / wide-column store for random, real-time reads and writes, sitting directly on HDFS.

Together they cover the two main ways to query big data: SQL-style (Hive) and key-value / column style (HBase). If your question is "sum this column over all rows," Hive is the natural fit; if your workload is "fetch these rows by key, fast, millions of times a second," HBase is.

13.3.3 Scripting, Machine Learning, and Streaming: Pig, Mahout, Storm, Kafka

• Pig is a scripting language — like a bash script, a batch script, or PowerShell — you use it to run jobs. You describe the data flow in Pig's script language and it compiles to processing jobs on the cluster.
• Mahout is the machine learning package: pull the data, process it, apply any machine learning algorithm. For data management for machine learning, it is the natural fit — the ML library of the classic ecosystem.
• Storm is the streaming tool, in the same family as Kafka: Kafka is the message/stream backbone (the pipes that carry events), and Storm is the stream processor (the engine that reacts to those events as they flow).

Kafka and Storm split the streaming job the way HDFS and MapReduce split the batch job: one moves the data, the other processes it.

13.3.4 Coordination and Indexing: Zookeeper and Solr

• Solr provides the indexing — the full-text search layer that makes the ecosystem's data findable, the way a library catalog makes books findable.
• Zookeeper keeps all the components of the ecosystem safe, controlled, and properly fenced: it tracks whether things are started or stopped, checks liveness ("is the food there or not" — the heartbeat questions from Section 13.7), and does the allocations that keep the pieces from stepping on each other. It is the ecosystem's coordinator: when many distributed components must agree on who is the leader, who is alive, and who gets which lock, Zookeeper is the referee.

Together, all of these form the Apache Hadoop open source ecosystem. The naming pattern is worth internalizing — one tool per job:

13.4 Cloud Equivalents: AWS and Google

Every Hadoop component has a named equivalent in the cloud. The lesson of this section is the shape: the ecosystem pattern repeats everywhere — you just rename the components. Once you can translate between Apache and Amazon and Google names, a pipeline built on one vendor can be rebuilt on another without redesigning the architecture.

13.4.1 AWS: Kinesis, S3, EMR, Elasticsearch, DynamoDB

Amazon's family maps one-to-one onto the Hadoop ecosystem:

The professor's phrasing is the exam-ready mapping: where we say HDFS, they say the EMR file system; where we say MapReduce, they say Amazon MapReduce; where we say a normal search, they say Elasticsearch; and where we have Hive as the relational data store, they have DynamoDB.

Connectors push data from any source to any source — from Kafka into S3, from S3 into the EMR file system, from a relational database into DynamoDB — so the same pipeline pattern transfers across platforms. The plumbing is different; the architecture diagram is the same.

13.4.2 Google Cloud: Storage, Bigtable, Dataflow, Dataproc, BigQuery

Google has a similar family:

• Cloud Storage — object storage, the S3 equivalent.
• Cloud Bigtable — the NoSQL / wide-column store, the HBase equivalent.
• Cloud Datastore — a document database for app data.
• App Engine — the platform where APIs interface; the APIs let applications talk to the data services.
• Cloud Dataflow and Cloud Dataproc — for data ingestion and processing; Dataproc is Google's managed Hadoop/Spark service, Dataflow is the stream-and-batch processing engine.
• Cloud SQL — managed relational databases; data pushes through Cloud SQL as it moves between stages.
• Google BigQuery — the analytics layer: the serverless data warehouse that answers queries over the platform's data (Section 13.5.2 covers it in depth).

Real-world connection: multi-cloud practitioners map these equivalences so that a pipeline built on one vendor can be rebuilt on another without redesigning the architecture. The mapping table in this section is exactly the tool they carry: every Apache name has an Amazon name and a Google name, and the roles — store, process, query, stream — stay constant.

13.5 Cloud-Native Data Platforms: Snowflake, BigQuery, and Databricks

Beyond raw infrastructure, the market offers cloud-native data platforms: complete systems that bundle storage, compute, and analytics into one managed product. The professor's advice for choosing among them is blunt: every one of these platforms has AI/ML capability, scales, provides multi-cluster support, and helps you build data pipelines — so the decision comes down to the pricing model and whether you want a multi-cloud platform, and then you pick.

13.5.1 Snowflake: Virtual Warehouses Without Lock-in

Snowflake offers a cloud-native platform built around a simple separation: instead of a dedicated on-premise warehouse, you build a virtual warehouse — compute capacity that you can start, stop, and resize on demand — and put services on top of it, with database storage underneath. The storage tier and the compute tier are decoupled, which is what makes the warehouse "virtual": you are not buying a fixed machine, you are renting a pool of processing power that grows and shrinks with your workload.

Snowflake runs on AWS, Azure, and Google Cloud with no vendor lock-in. The lock-in problem is real, and the professor names its mechanism precisely:

13.5.2 BigQuery: Serverless Data Cloud

BigQuery provides a serverless data cloud platform: you do not provision servers at all. It offers automatic scaling and administration — like Amazon, it lets you scale the hardware up or down — memory, CPU, or resources — without you touching a single machine, because the service handles the scaling for you. Standard SQL works, so the learning curve is small: if you already know SQL, you already know how to ask BigQuery questions. It also supports predictive analytics (building models directly inside the warehouse) and integrates seamlessly with AI/ML services.

Its Looker platform adds very powerful visualization on top of the warehouse. Other tools in the same space include RapidMiner and Presto, and some teams build their own tools: leave the data where it is, and keep your own drill-down analysis — the platform-agnostic alternative to adopting a vendor's BI layer.

13.5.3 Databricks: Spark Plus Delta Lake

Databricks combines Apache Spark processing with Delta Lake capability, creating what is called a lakehouse architecture — a single platform that holds both data warehouse and data lake workloads. Where a data lake stores raw files and a warehouse stores curated tables, a lakehouse puts a transactional, queryable layer directly on lake storage: you get the warehouse's reliability and the lake's flexibility in one system. Databricks is the managed, production version of the Spark engine from Section 13.6, with Delta Lake supplying the storage layer, and its MLflow platform provides the evaluation matrices used for LLM scoring in Section 13.16.

The ML Spark (Spark's machine learning library, MLlib) comparison against regular big data processing is flagged as important to remember — the ML-focused engine versus the general processing engine is a distinction that recurs in exams and interviews.

13.5.4 Choosing a Platform: Student Questions

The three platforms side by side:

13.6 Apache Spark vs MapReduce

Spark versus MapReduce is the classic big data comparison — and the professor flags it as important to remember. The two engines attack the same problem (processing massive data across many machines) with different architectures, and the difference is memory.

13.6.1 MapReduce: A Definition

MapReduce is a framework for writing functions that process massive quantities of data in parallel on giant clusters of commodity hardware in a dependable manner. Four parts of that definition carry real weight:

• Functions — you write two functions, a mapper and a reducer; the framework does everything else.
• In parallel — the work runs on many, many clusters at once; no single machine sees the whole input.
• Commodity hardware — MapReduce is mainly meant for ordinary, cheap machines. Many companies have purchased lots of machines that are not in use, so the pattern reuses those machines instead of buying special ones.
• In a dependable manner — failures are expected and handled: if a node dies mid-job, the framework reruns that node's tasks elsewhere.

It is primarily based on Java — the reference implementation and most production deployments are Java.

13.6.2 Apache Spark: A Definition

Apache Spark is a data processing framework that can rapidly operate processing duties on very massive data sets, and can distribute data processing duties across multiple computers — either on its own or in tandem with other distributed computing tools. Where MapReduce's definition stresses commodity hardware and dependability, Spark's stresses speed ("rapidly") and flexibility: it can run standalone or sit on top of an existing cluster (including HDFS and YARN from earlier sections).

13.6.3 In-Memory Processing and the RDD

The defining difference is memory. Spark uses its own memory, so it is much, much faster than MapReduce. The consequences:

• Real-time processing: Spark can deal with real-time processing; MapReduce's disk-bound design is a batch design.
• Security: Spark's security is not as good as MapReduce's yet, but the team keeps improving it — a real trade-off to remember.
• Caching: most importantly, Spark can cache the memory data for processing its tasks, while MapReduce cannot cache in memory — it has to go to disk and do everything through the disk. A job that reads the same data repeatedly (iterative machine learning, interactive query) is transformed: the data stays in memory instead of being re-read from disk every pass.

The in-memory engine architecture is built on the RDD — the resilient distributed dataset: a fault-tolerant collection of data partitions spread across the cluster, kept in memory where possible and rebuilt from lineage (the record of operations that created it) when a partition is lost. Spark rewrote everything for this engine, so it has its own library (MLlib, the ML Spark mentioned in Section 13.5), and work is split into tasks that the engine schedules across the cluster.

Comparison — MapReduce vs Spark:

When to pick which: choose MapReduce when you need maximum maturity and security on commodity batch workloads; choose Spark whenever speed, iterative processing, or real-time work matters — which, in practice, is most modern workloads.

13.6.4 Setting Up and Running Spark

Spark is very easy to set up, including on WSL. Follow the setup instructions — install Python, install the JVM (Spark requires a JVM because it runs on the Java Virtual Machine) — and you can run your own Spark environment:

• Open the Spark shell from the command prompt (spark-shell) for interactive Scala work, or use PySpark for your Python coding.
• You submit a job the way you run a regular Python program: where you would type python for normal Python, you use spark-submit and it runs.
• Streaming jobs work the same way — a streaming program is submitted with the same command; the engine handles the continuous data flow.

13.6.5 Student Questions and Answers

13.7 Fault Tolerance and High Availability

13.7.1 The Question That Started It

The whole discussion is triggered by one student question, and the professor's answer frames everything that follows.

The father-mother framing is the lecture's anchor: both parents handle the family's needs, but they do it differently — one works through the problem inside the household, the other guarantees the household stays running. A system almost always needs both, but they are different designs with different price tags.

13.7.2 Fault Tolerance: Withstanding Failures Inside the System

A concrete industry picture makes the "keep serving with less" idea plain: a service has five modules, and one module goes down. The other modules collect and keep that module's information, so if there is a problem with module A, the others still provide some basic information. If you are waiting for a tax refund and the refund calculation module misbehaves, a fault-tolerant system at least tells you whether the refund was calculated or not — something is going on, instead of silence.

In critical systems the stakes are higher: flight systems control all the sensors through a Flight Management System, and the aircraft carries two of them from day one, running on a real-time operating system such as VxWorks, because there can be zero downtime — a failure is catastrophic or has significant impact. To make that possible you build systems and algorithms that guarantee a duplication factor — a second copy of anything whose loss would stop the mission.

13.7.3 High Availability: The Uptime Contract

High availability is also a contract: a cloud provider signs an SLA promising, say, zero downtime; if the infrastructure is responsible for an outage, the provider pays the customer. That is how the contract works — a commitment you make to the client, backed by clusters running their systems. Availability is measured as the percentage of time the service is in an operable state, and it is governed by recovery objectives:

• The recovery point objective (RPO) says how much data you are willing to lose at any given time — the maximum acceptable data loss after an outage. An RPO of zero means no lost data: every transaction must be captured before it is acknowledged.
• The recovery time objective (RTO) says how fast you must come back — the maximum acceptable outage duration. An RTO of one day might be fine for an internal reporting system; for an online retailer, even a short outage can cost serious money.

13.7.4 Load Balancing vs High Availability: A Correction

The next exchange is a vocabulary correction — a student's plausible-sounding account of high availability is rejected and replaced with the right term.

Why the student's story seemed plausible: adding machines to a cluster is a real reliability technique, and "if half the machines are down, half the capacity is served" sounds like a meaningful degradation story. Where it goes wrong: spreading work across more machines is load balancing — a performance technique, not an availability guarantee. High availability says nothing about how many machines serve a request; it promises that the service stays reachable. The preferred term to remember is load balancing for capacity questions, high availability for uptime promises.

The veracity tie-in is the professor's re-explanation: veracity (one of the seven Vs of big data) is the data-quality trait of continuing through small mistakes — a pipeline that self-corrects when the data is slightly wrong, the way a human listener infers "centigrade" from context when you say a temperature in the 30s. Fault tolerance is the same idea at the system level: detect the error, recover, keep serving.

13.7.5 The Uptime Math: 99.9% and Beyond

No one in this world — not Oracle, not Sundar Pichai, not even God — can give 100 percent high availability. There is always a tiny second or two. The professor has seen this across a career in the US and at Cisco: nobody delivers a full 100 percent. So the conversation turns into numbers.

The general relationship is simple: downtime equals one minus the availability, times the time period:

\[ \text{downtime} = (1 - a) \cdot T \]

where \(a\) is the availability fraction (so 99.90% availability means \(a = 0.999\)) and \(T\) is the period you care about (a day, a week, a month, a year). The \( (1 - a) \) factor is the share of time the system may be down; multiplying by \(T\) converts that share into real time.

For that kind of setup you definitely need high availability — you need a cluster, extra infrastructure, and money.

13.7.6 Disaster Recovery, RTO, and RPO

A disaster recovery plan (DRP) is the organized answer to a large-scale failure: which systems come back first, from which backups, with how much data loss and how fast. High availability is part of it — but the plan is built around the two objectives: the RPO sets the backup frequency (if losing one hour of data is acceptable, back up hourly; if not, back up continuously) and the RTO sets the recovery machinery (fast restoration needs standby systems ready to start). High availability without RTO/RPO targets is a slogan, not a plan.

The professor's closing analogy keeps the two terms distinct:

• Fault tolerance is a mother with no vegetables who does something to prepare the food from what is available — cooking dinner from whatever is in the kitchen, even if the planned ingredients are missing.
• High availability is the extra frozen vegetables kept in the fridge for the worst case — backup stock standing by, so dinner happens even when the kitchen is empty.

The ability to manage with what you have is fault tolerance; the standby stock is high availability. Both matter, and they are related terms, not rivals — the father and the mother of the opening question, handling the same family differently.

13.8 Batch vs Stream Processing

The lecture already covered batch and stream processing; this session's job is the decision: when do you run a job on a finite pile of data, and when do you process data as it arrives? The answer is all about latency — and what the business can tolerate.

13.8.1 Batch Processing: Finite Data, Rerunable Jobs

Batch processing runs a little behind reality — sometimes hourly behind, mostly daily, often 6 to 8 hours behind the real-time system. The professor's own data warehouse work is the classic picture: a batch script filtered the data, checked every acknowledgement, checked the counts, and then produced the output stored in a table. The job runs on schedule, takes the data that has accumulated since the last run, and writes a result — then waits for the next schedule.

The defining property is finiteness: the input data of a batch process is finite. The file exists, complete, when the job starts. That single property buys the batch world its great operational luxury — a failed job can simply be rerun: nothing is lost, you just run it again with whatever data is available. If a nightly job crashes at 2 a.m., you fix the script and rerun it on the same input; the answer comes out the same.

13.8.2 Stream Processing: Constantly Arriving Data

Stream processing is continuous. With Apache Kafka in a good architecture, the moment sensor data arrives you are consuming it — immediately taking the data in. There is no "complete file" waiting; data shows up in an endless series of events, and the system processes each one as it lands.

That immediacy introduces fault tolerance concerns: if something goes wrong, what happens when the sensor goes down? In batch, you may not have a problem — you continue with whatever data is available, because the input was already captured. But when you depend on real-time data, a down sensor matters: the stream goes silent, and your model has nothing fresh to work on. Time-series forecasting models like ARIMA — the autoregressive integrated moving average model for predicting future values from a series of past values — and SARIMA, its seasonal version that adds repeating patterns (daily, weekly, yearly), show up in this world, because stream data is time-ordered by nature. (The class audio said "Arima, SARMA"; the standard model names are ARIMA and SARIMA — the seasonal extension.) Stream jobs work on data that is constantly arriving; the stream works when something starts coming.

13.8.3 Choosing by Latency

The whole batch-versus-stream decision is about latency — how long between an event happening and the system knowing about it. What latency are we talking about — time, minutes, hours, seconds? You also look at the business: what does the business need, and how much delay can it tolerate? That combination decides between batch and stream.

When to pick which: if the business can wait for the answer and the input is a complete pile of data, batch is cheaper and simpler — rerunnable and forgiving. If the decision depends on what happened seconds ago, stream is the only option — and with it come the fault tolerance obligations of Section 13.7.

13.9 LLM Foundations: Memory, Corpus, and Text Preprocessing

A large language model (LLM) is an AI system built to predict and generate human-like text. Before the professor opens the LLM pipeline material, he lays the foundations — what an LLM needs, what you must know first, and how text becomes numbers.

13.9.1 What an LLM Needs: Parametric and Non-Parametric Memory

The answer splits into four needs:

• Parametric memory — knowledge stored inside the model's parameters (the weights learned during training). When the model answers from its own weights, it is using parametric memory.
• Non-parametric memory — knowledge stored outside the model, in an external system such as RAG (retrieval augmented generation: fetch relevant text at query time and feed it to the model). Web crawling fills this external store with fresh material the model never memorized.
• Ground truth — resources that let the end user see where the data came from and judge how trusted it is; this is the measure of faithfulness of an answer.
• A good quality corpus — a diversified corpus, where some sources are trustworthy and reliable and some are not; the model's raw material for learning.

The word corpus is important — if you have good data, you have a good corpus, and that is where LLMs work. The industry is hunting for quality data at any cost: OpenAI has started taking rare books and feeding them to its models, and destroying the rare books in the process — because they want quality data, and scarcity makes the source valuable.

13.9.2 The Foundations: Math, Python, NLP, Transformers, and Ethics

For the LLM to work you need foundations, and each one is itself a big topic: mathematics, Python programming, machine learning basics, deep learning, NLP, transformers, cloud tools, and ethics and bias. There are lots of permutations and combinations: we work on text, on poetry, with lots of calculations — what comes first, what comes next, the order of precedence. There are algebraic rules, statistics and calculus, and lots of Python code involved internally — many packages, not only scikit-learn (everyone working with Python uses packages like Pandas, NumPy, and Scikit-learn), and deep learning with TensorFlow.

The story told a few minutes earlier is the running example: the story, the king, the curry, the movie — these are all stored in different places and at different times, and recalling them means connecting related information, deciding what needs to be stored and what needs to be processed, based on context and what is next to what. That is exactly what a recurrent network does with a sentence: each word's meaning depends on the words before it. So we need text processing, web crawling, web scraping, the self-attention mechanism, and the encoder-decoder architecture — the key models central to modern LLMs. Understanding how the transformer works is the doorway to the parametric and non-parametric levels of model training.

Exam note: the foundations list — mathematics, Python programming, machine learning basics, deep learning, NLP, transformers, cloud tools, ethics and bias — is flagged as the key background for understanding LLM questions; keep it in mind whenever an LLM scenario appears.

13.9.3 Text Cleaning and Tokenization

In any machine learning problem, data cleaning and preprocessing are as important as building the model — especially for unstructured data like text. Text has so many issues. Data issues are easier to fix than text issues: spelling mistakes, quotations, repetitions, and meanings that shift with context. "50 kilometers per liter" and "50 kilometers per hour" convey the same meaning if the context is understood, but without context the meaning changes totally — what is the subject, what is the object, whose voice is it, what is the sentiment? Systems do not understand people's sentiments, voice, noise, and context. If somebody writes "okay" in a conversation, we do not know what "okay" means without the conversation. So we manipulate the text for contextual understanding: a lot of lowercasing, removing HTML tags, stemming to find the root word, lemmatization, text wrangling, and text cleansing.

There is a natural hierarchy inside the corpus:

corpus (set of documents)
  → document → paragraph → sentence → clause → phrase → word → root word → morpheme

• A corpus means a set of documents.
• Documents are made of paragraphs, paragraphs of sentences, sentences of clauses, clauses of phrases, phrases of words, words of root words — "act" is a root word, "reaction" is a word built from it.
• A word which cannot be split further is a morpheme: the smallest meaning-bearing unit in a language, down to the character. "Re-" and "-action" in "reaction" are morphemes; each carries meaning, and neither splits further without losing it.

These are the building blocks. Then we do tokenization — splitting paragraphs into small words, tokens — remove the unnecessary words, perform stemming, cut to the root word, take the right word, and build a word cloud as a visualization of the data. Finally we give the words word embeddings, because the computer cannot understand spoken language the way humans do — everything must be mapped into vectors, every feature mapped into a vector, and that is where the count vector comes into play.

Real-world connection: this is where the industry is shifting. Many organizations are building their own LLMs, their own corpora, their own models — mini LLMs, domain-specific LLMs, company-specific LLMs, customer-specific LLMs — because they do not want to depend on a general provider. In fintech work, teams have been advised on GenAI adoption for the financial domain, and the corpus design is the first decision: what documents go in, and how they are cleaned, decides what the company's model can know.

13.9.4 Word Embeddings and Vector Representation

Once we have the tokens, we do word embedding and give each embedding a weight: the numerical representation of the same text. The simplest form is the count vector (one-hot encoding): each word in the reference vocabulary gets one position, and a word's vector places a 1 in its own position and 0 everywhere else.

That is the embedding idea in its simplest form — a count vector: each word is one position, one bit. The same machinery underlies TF-IDF, where we find the term frequency and the inverse document frequency, and n-grams. The standard form of the TF-IDF score is:

\[ \text{tfidf}(t, d) = \text{tf}(t, d) \cdot \log\left(\frac{N}{\text{df}(t)}\right) \]

where \(\text{tf}(t, d)\) is the term frequency of term \(t\) in document \(d\) (how often the term appears), \(\text{df}(t)\) is the document frequency (how many documents contain the term at all), and \(N\) is the total number of documents. The fraction \(N / \text{df}(t)\) down-weights terms that appear in almost every document ("the", "is") because they carry little information, and the logarithm keeps the ratio from exploding. These are all happening internally in the LLMs. Claude goes further with discourse analysis and pragmatic analysis — interpreting language in context, knowing how to connect to other systems — and that is why LLMs are becoming so good.

13.9.5 Multimodal Models and the Industry Shift

Modern models are not limited to text. The professor's own forthcoming work is a multimodal model: input is not necessarily text — it can be text, numbers, audio, video, or image — and output can be the reverse: text to text, text to audio, text to image, text to video. An earlier paper built a chatbot on graph databases, published in a Springer soft computing venue, with another paper in a supercomputing journal. The core task stays the same: learn to predict the next word, the sequence of words, based on context — regardless of whether the input arrives as text, sound, or pixels.

13.10 The Three LLM Pipelines: Data, Inference, and Ops

An LLM application is not one program — it is three pipelines working together, each with its own center of gravity. The professor explicitly flags all three as important: knowing which pipeline does what is core material. The three pipelines are the data pipeline (data-centric), the inference pipeline (inference-centric), and the Ops pipeline (operation-centric).

13.10.1 The LLM Data Pipeline

The LLM data pipeline is data-centric. It focuses on preparing data for LLMs: cleaning, tokenization, and embedding generation for vector databases. It ensures the data is in the right format for training and for retrieval augmented generation — RAG.

Everything covered in the previous section — cleaning, splitting paragraphs into tokens, generating embeddings — is exactly the work of this pipeline; it is why vector databases are being built in large numbers. The pipeline's outputs are the artifacts the model and the retrieval system consume: clean tokens for training corpora, and embedding vectors for the vector database that RAG searches at query time.

13.10.2 The LLM Inference Pipeline

The LLM inference pipeline is inference-centric. It chains together prompt engineering, model calls — through APIs or locally — and output parsing. It connects the user prompt to the final generated output, often involving multiple steps or tools.

13.10.3 The LLM Ops Pipeline

The LLM Ops pipeline is operation-centric: it manages the entire lifecycle — version control, continuous integration and continuous deployment (CI/CD), monitoring for drift and hallucinations, and cost optimization, which means token usage. Every time something comes in, we ask: how are we managing it? Was the previous version needed for the next one? Are we looking at data drift or model drift?

Version management is real: people using the Gemini API have seen errors appear because the version changed; you have to specify the version properly and then it works. Whether it is model drift, data drift, or hallucination, you need proper optimization of the tokens you are using — and that is where extended prompt engineering, version control, and CI/CD come into play. The Ops pipeline is where an LLM application stops being a demo and becomes a service: reproducible versions, monitored behavior, and controlled cost.

13.10.4 Student Questions and Answers

Exam note: the three LLM pipelines — the data pipeline, the inference pipeline, and the Ops pipeline — were explicitly flagged as important. Knowing which pipeline does what is core material: data pipeline prepares data (cleaning, tokenization, embeddings for vector databases and RAG); inference pipeline chains prompt → model call → output parsing; Ops pipeline manages versioning, CI/CD, drift and hallucination monitoring, and token cost.

13.11 Traditional Machine Learning vs LLM

The professor contrasts the classical ML world with the LLM world on the dimensions that decide how you build, supervise, and debug a system.

13.11.1 Task-Specific Models vs General-Purpose Models

A traditional ML model can be built without any LLM tools — no ChatGPT, no GPT-3, no GPT-4. You specify your own parameters; the model is task-specific, built for a specific user or task, and smaller in size. The LLM is the opposite: it is big data, a big corpus, and can do many things; it is larger in size. The data flow passes through the same stages, but traditional ML and LLM use different data sets in different pipelines.

The key structural claim: the flow through stages is the same for both — data in, model, evaluation, deployment — what changes is the size and where the feedback comes from.

13.11.2 Human-in-the-Loop and Interaction Logs

Traditional ML has continuous feedback, though it gets less importance. In LLM work, the key is reinforcement learning feedback with the human in the loop — HIL, human in the loop, one of the current buzzwords. The human needs to be involved to interpret the inference and give the correct direction to continue further.

We also need to know what kind of interaction log we are having: if the LLM gives something wrong — say a chatbot recommending a stock purchase and something goes wrong — you look at the interaction logs and the sentiment of the interface. Sometimes the user also makes mistakes or does not understand the domain well, so you need to correct them and give them the right options. The interaction log is the audit trail of every exchange: what the user asked, what the model answered, and how the human steered the conversation — the raw material for the evaluation and feedback pipeline (Section 13.16).

13.11.3 The Allergies Example: Model Problem or Data Problem?

The live example that ties the section together comes from a student's real experience with a medical chatbot.

13.12 Data Quality for LLM

13.12.1 Quality Beats Size

Data quality is critical. The professor's one-line law: high quality data allows smaller models to outperform larger ones trained on messy data. The picture is a water analogy: a small cup of clean water is better than a large drink of something that is not clean — a fresh fruit juice of 50 ml is equivalent to 500 ml of plain water. Same for models: a small model on clean data beats a big model on dirty data.

The mechanism behind the analogy: a model learns patterns from what it sees, and wrong patterns learned from messy data cannot be unlearned later — the noise becomes the model's "knowledge." A smaller model that saw only clean, curated data has a far higher ratio of signal to noise in everything it knows, so its outputs are more trustworthy per unit of size. That is why the corpus-design work of Section 13.9 — cleaning, deduplication, curation — is the main point of control for the whole LLM stack: it is cheaper to fix data than to buy more compute.

13.12.2 Traceability and Freshness

Two more properties complete the quality picture:

• Traceability — you must know from where the data is coming, so the end user can judge how trusted the answer is. Traceability is especially important for RAG systems: when an answer is assembled from retrieved passages, the user needs the source of each passage to verify it. This is the operational side of the ground truth and faithfulness requirements from Section 13.9.1 — an answer without a source is an assertion; an answer with a source is evidence.
• Freshness — data must be up to date: up-to-date, up-to-minute data provides accurate and timely information. A model trained on stale data answers yesterday's questions; a RAG store that ingests continuously answers today's. Freshness is what separates a decision-support system from an archive.

13.13 Pipelines: Structure and the Five Training Stages

13.13.1 What a Pipeline Is

Pipelines are configurable, multi-stage funnels that represent a process at your institution or organization, with the goal of achieving a desired outcome. The funnel picture is literal: many inputs enter at the top, and each stage narrows and transforms the flow until the desired output emerges at the bottom.

Each pipeline has its own data and its own orchestration; it can run independently, and it can use different tools for different things based on the SLA — the SLA for data retrieval is different from the SLA for data quality checks. The stages represent different milestones: each stage is a checkpoint where a piece of work is completed, verified, and passed on to the next.

A series of LLM training follows exactly the same logic: there is no difference between a data pipeline and an LLM pipeline. The professor's standing saying: computer science is common sense — old wine in a new bottle, acting nicely with different variations. The shapes you already know (funnel, stages, milestones, SLAs) are the same shapes used for LLM training; only the tools inside the stages change.

13.13.2 The Five Stages of an LLM Pipeline

The stages of the LLM pipeline are: data preparation, pre-training, fine-tuning, evaluation, and deployment.

data preparation → pre-training → fine-tuning → evaluation → deployment

1. Data preparation — we collect the data: sourcing, cleaning, and formatting the corpus.
2. Pre-training — we do the pre-training: the model learns language fundamentals from the corpus (Section 13.14).
3. Fine-tuning — the model is adapted to a domain or task (Section 13.15).
4. Evaluation — the model is measured, scored, and compared (Section 13.16).
5. Deployment — the model goes live (Section 13.18).

Concepts like synonym, antonym, and hyponym belong to the NLP territory you will study properly next semester; for now, understand the flow. The course itself is a prelude and interlude — like cement: you must understand the cement mixture process before you use it for a building. So study machine learning nicely, study deep learning nicely, study big data nicely, study text and speech analytics nicely — and finally you deploy. The five stages are the mortar that holds the whole course together: each lecture maps onto one stage of this funnel.

13.14 Pre-training and Retraining

13.14.1 Building Broader Linguistic Knowledge

The pre-training pipeline develops broader linguistic knowledge. The professor's test: if he wants to speak fluent Hindi with a student, he needs good knowledge of Hindi grammar, pronunciation, the sounds, the nouns, the proverbs — only then can he hold a fluent conversation. Language skills are the base, and that is natural language processing — computational linguistics: NLP's linguistic understanding, encoded in algorithms and models.

It is an art: understanding the language and understanding the context, especially when the same sound conveys different meanings. In Tamil, one word that literally means "killing" is used as praise for excellent work — "he killed it" — the meaning flips entirely with context. The conversation, the context, carries the sense. Pre-training is what builds that context sensitivity at scale: by seeing billions of words in context, the model learns not just what words mean but how their meaning shifts with the words around them.

13.14.2 Retraining: Keeping the Model Current

Language models are dynamic. New information is made available, and the model must be kept current; user feedback shows areas of improvement. When the student caught the allergies mistake (Section 13.11), the catch itself showed that something was wrong with the pre-training — the feedback feeds improvements back in.

The normal flow is training, continuous training, and retraining: pre-train, continue to pre-train, and retrain, retrain, retrain — the retraining keeps making the model better. Each retraining round takes the current model, feeds it the new and corrected material, and produces an improved version. Retraining is not a one-time event; it is the operational rhythm of a living model — the same feedback loop that the evaluation pipeline triggers in Section 13.16.

13.14.3 The Teacher Analogy

The school analogy makes it concrete:

1. Pre-training is the first exposure. The teacher first gives you the topic — like learning the multiplication table. You practice it until the fundamentals are in place.
2. The test. Then the teacher writes a number on the board and asks you to fill in the blank: 5 × 6.
3. Retraining is the correction drill. If you get it wrong, you write it five times — that is retraining.
4. Repeat. Your retraining did not work? Do a retraining, repeat, repeat, repeat.

The loop maps cleanly onto the model: exposure builds the base (pre-training), the test exposes the gap (evaluation), and the drill corrects it (retraining) — then the cycle repeats until the gap closes. The multiplication table is never "done"; it is maintained.

13.15 The Fine-tuning Pipeline

13.15.1 The Stages of Fine-tuning

Fine-tuning is what makes a general model context-aware: proper transfer learning, proper parameters. You check accuracy, you check style, you check the indent percentage. The stages:

1. Take a pre-trained model — like YOLO for object detection, or any other base model. The base model already carries the broad knowledge from pre-training; fine-tuning does not start from scratch.
2. Add the hyperparameters — the knobs that control how training behaves.
3. Train by passing epochs — so you can check how close the model is to the goal; an epoch is one full pass over the training data, and several epochs let the model's error descend toward the target.
4. Control overfitting — keep the model strong and resilient so it generalizes beyond the training data.
5. Fine-tune the LLM for the domain-specific task or the particular task you want to achieve.

The fine-tuned parameters work for this particular domain — and for this particular version. A model fine-tuned for medical QA is not a model fine-tuned for legal QA; the adapted weights carry the domain's task knowledge, and versioning (Section 13.15.3) tracks which adaptation is which.

13.15.2 Hyperparameters, Epochs, and Overfitting

Hyperparameters are the knobs: learning rate (how big a step the model takes toward the goal on each update), error rate, bias, gradients — you tune them. They sit above the learned parameters: the parameters emerge from training, the hyperparameters are chosen before it.

Then you train by passing epochs, checking how close you are getting — each epoch's error tells you whether the model is still improving. Then comes overfitting control: you need a strong, resilient model, which means handling outliers — outliers are removed using the IQR, the interquartile range.

The IQR is a spread measure built from the quartiles: sort the data, cut it into four equal quarters, and take the middle half. If \(Q_1\) is the value at the 25th percentile (one quarter of the data below it) and \(Q_3\) is the value at the 75th percentile (three quarters below it), then:

\[ \text{IQR} = Q_3 - Q_1 \]

The standard outlier rule flags any point below \(Q_1 - 1.5 \cdot \text{IQR}\) or above \(Q_3 + 1.5 \cdot \text{IQR}\). A tiny check with real numbers: data \(\{2, 4, 6, 8, 10, 12, 14\}\) has \(Q_1 = 4\), \(Q_3 = 12\), so \(\text{IQR} = 8\) and the fences are \(4 - 12 = -8\) and \(12 + 12 = 24\) — none of the points are outliers. Add a stray 100 and it sits far above the upper fence 24, so it is removed before fine-tuning. Remove the outliers, fine-tune the LLM, and the model is ready.

13.15.3 Versioned Fine-tuned Models

You will have version 1, version 2, version 3 of the fine-tuned model. Which version works best? You do a comparative study — that is where boosting and bagging come into the picture, combining models to find the best result.

• Bagging (bootstrap aggregating) trains several versions in parallel on different samples and combines their votes — each version is an independent expert, and the majority (or average) decides.
• Boosting trains versions sequentially, each new one focusing on the mistakes of the previous — the ensemble builds a stronger learner by stages.

Versioning plus ensembles is the practical answer to "which fine-tuned model do we ship?": you do not trust version numbers, you measure them against each other in a comparative study, and you can combine the best candidates into a stronger result.

13.16 The Evaluation and Feedback Pipeline

The evaluation and feedback pipeline is where the LLM loop closes: measure the model, score it, decide whether to retrain, and feed the results back. The professor calls it very important — the exam guidance at the end of this section shows why.

13.16.1 Model Drift vs Data Drift

In the evaluation and feedback pipeline, two things are tracked: model drift and data drift.

• Data drift is a change in the distribution of the data itself — the world the model sees has moved (customers changed behavior, sensors changed readings, a source system changed format).
• Model drift is a change in the model's performance when the data has not moved — the model's behavior degrades while the input distribution stays consistent.

In the evaluation we consider recall and accuracy as part of the model metric, and we conclude the PSI — the population stability index, comparing the actual distribution against the expected distribution over buckets:

\[ \text{PSI} = \sum_i (p_i - q_i) \cdot \ln\left(\frac{p_i}{q_i}\right) \]

where \(p_i\) is the actual distribution share in bucket \(i\) (the share of current observations that fall in bucket \(i\)) and \(q_i\) is the expected distribution share in bucket \(i\) (the share the baseline or training distribution predicted). Each bucket contributes how far the two shares are apart, scaled by the log of their ratio, and the index sums across buckets. (The formula was not written in class — the professor only named the index — so this is the standard definition, matching the distribution-comparison method used in banking and fintech: bucketize the metric, compare the bucket percentages between distributions, and sum the log-weighted differences.) A commonly used industry rule of thumb: PSI below 0.1 means no significant shift, 0.1 to 0.25 moderate shift, above 0.25 major shift.

KL divergence provides the measure for data drift:

\[ D_{\text{KL}}(P \parallel Q) = \sum_i P(i) \cdot \log\left(\frac{P(i)}{Q(i)}\right) \]

where \(P\) is the actual probability distribution and \(Q\) the expected (baseline) distribution, and \(P(i)\), \(Q(i)\) are the probabilities assigned to outcome \(i\) by each distribution. (Again, the professor named the divergence without writing the formula; this is the standard definition.) KL divergence is similar to PSI but asymmetric — \(D_{\text{KL}}(P \parallel Q) \neq D_{\text{KL}}(Q \parallel P)\) — so it can detect distribution order switching, not just the size of the gap.

13.16.2 The Scorer and Weighted Evaluation

That is where the scorer comes into the picture. An LLM test case has four parts: the input, the LLM output, the retrieval context (retrieval means the context we get from RAG), and the arguments from the LLM application. The LLM evaluation matrix is provided by MLflow; from it we get the scores. The scorer asks: 100 marks — what is the reason? 90 marks — what is the reason? How much precision, how much recall, how much accuracy? We also tune hyperparameters, but the scorer comes in on top.

Earlier we gave no weightage; now we give weightage for each metric, and the scorer defines whether we need to retrain the model. The scoring is not a single number from one metric; it is a weighted blend — precision, recall, and accuracy each contribute according to the weight you assign, and the weighted score crosses a threshold before the pipeline reacts.

13.16.3 The Feedback Loop That Triggers Retraining

When the scorer says retrain, the pipeline is triggered: the inference data becomes again the training data — we store this data, and then we retrain, and then we evaluate again, and the loop comes back through. Production traffic becomes the next training corpus: every inference that was scored and judged is saved, fed into the next training run, re-evaluated, and scored again.

The college-test analogy makes the weightage idea plain: tests T1, T2, T3, T4. A student scores well in T1 and T3, poorly in T2 and T4 — did they do well overall? We do not know until we assign weightage. If T1 is 90 percent and T3 is 60 percent, the weighted result decides whether it is good enough. That is the pass/fail threshold: accumulating the various scores and designing the final decision. The scorer is the examiner who knows that a 90 on the hard test and a 60 on the easy test tell different stories until weights say otherwise.

13.16.4 Evaluation Metrics

The evaluation metrics are:

• Perplexity — how surprised the model is by held-out text; lower means better prediction.
• BLEU — the bilingual evaluation understudy score, measuring overlap between generated and reference text (originally for machine translation).
• ROUGE — recall-oriented measures for summarization, measuring how much of the reference content the generated text covers.
• The fairness and bias check — the FAT principle from Section 13.9, applied to every evaluation.
• Robustness — how well the model holds up under noisy or adversarial inputs.

The fairness and bias check is why the NLP material on fairness and bias matters — bias checks ride along with every evaluation. A model is not evaluated on accuracy alone; it is evaluated on whether its answers are fair, accountable, and transparent.

13.16.5 Exam Notes

Exam note: in the comprehensive exam, questions may come with the LLM — a scenario will be given, and you will be asked what type of metrics you look at and how you pass the pipeline. Metadata questions follow the same pattern: metadata store versus repository versus registry — based on the scenario, which metadata types you choose and how you build the metadata systems. These are real-life questions, not disconnected from practice.

13.17 Testing LLM Pipelines

13.17.1 Unit, Functional, and Load Testing

We can automate the testing. There are many tests out there, but the important ones we normally do are the unit test on each pipeline, ensuring that the functionality works, and the load test — putting a lot of load into the data pipeline. There are lots of other tests hidden in the process; for a pipeline project, unit, functional, and load coverage is the baseline.

• Unit test — take one small, isolated piece of the pipeline and verify it behaves correctly on its own.
• Functional test — verify an end-to-end function (a use case) produces the expected outcome.
• Load test — push the pipeline with volume, checking that it keeps performing when the input is heavy.

In the pipeline, the scores and the data drift are split into multiple smaller pieces, and each one can be unit-tested: a scoring module, a drift-detection module, a parsing step — each gets its own small test that pins its behavior.

13.17.2 The Unit Test Example

13.17.3 Testing Automation as a Research Area

This is a great research area: ML testing automation of a RAG pipeline. For example, testing automation of a RAG pipeline for healthcare diseases — a concrete project shape. There are at least three project ideas visible right away in this area, and it is an area many people do not focus on.

For a final-year MTech project, combining your domain and your company, you can build test automation for RAG pipelines, and the work can grow into research papers — the professor offers guidance and co-authorship to students who push the work forward. The shape of such a project: pick a domain (healthcare diseases), build a RAG pipeline, and automate the testing of retrieval quality, answer correctness, and drift detection — the evaluation machinery of Section 13.16 turned into an automated test suite.

13.18 Deploying with Confidence

13.18.1 Commit, Roll Back, and Feature Flags

Deployment uses the same common sense as version control: commit or roll back — checkpoint, commit, roll back. The deployment of a model or a pipeline is treated like a code change: you make a checkpoint of the working state, you commit the new version, and if anything breaks you roll back to the checkpoint. There is no drama in deployment because every step is reversible.

The second tool is the feature flag: flag which feature is good, which database, which flag — feature flags let you turn capabilities on and off per user. A feature flag is a switch in the configuration: the code is deployed, but the capability is switched on for some users and off for others. That separation of deploying code from enabling features is what makes rollback surgical — you can disable one flag without reverting the whole system.

13.18.2 A/B Testing with Human Validation

We can do A/B testing — this version or that version — and get user input and feedback. A human is involved: an unattended robot performs the process execution, then human validation happens, then human approval happens. The human-in-the-loop pattern carries into production.

The A/B test is the deployment experiment: two versions (A and B) serve real users in parallel, the system collects behavior and feedback, and the comparison decides which version stays. The robot executes; the human validates and approves. The HIL pattern from Section 13.11 does not stop at training time — it is part of how production changes are approved.

13.18.3 Monitoring for Anomalies and Drift

In production you monitor logs and anomalies. Watch for model drift, outlier anomalies, and anomalous behavior — data anomaly, model anomaly, or environment anomaly. The three-way split is the monitoring checklist:

• Data anomaly — the input stream looks wrong: missing fields, shifted ranges, new categories.
• Model anomaly — the model's outputs behave unexpectedly even though the inputs look normal.
• Environment anomaly — the infrastructure around the model misbehaves: a dependency fails, latency spikes, a node dies.

Each one needs a different fix — data pipeline repair, retraining, or infrastructure work — so the first job of monitoring is to classify the anomaly, then act. The RAG pipeline builds on all of this and is picked up next.

Exam Guidance Summary

• Comprehensive exam questions may arrive wrapped in LLM scenarios: given a scenario, what type of metrics do you look at, and how do you pass the pipeline? Expect to reason about evaluation metrics — perplexity, BLEU, ROUGE, fairness and bias check, robustness — and about the scorer's weightage and thresholds.
• Metadata questions follow the same scenario pattern: metadata store versus repository versus registry — based on the scenario, which metadata types you choose and how you build the metadata systems.
• The three LLM pipelines — data, inference, and Ops — were explicitly flagged as important; know which pipeline does what, including version control, CI/CD, drift and hallucination monitoring, and token cost optimization.
• The LLM foundations list — mathematics, Python programming, machine learning basics, deep learning, NLP, transformers, cloud tools, ethics and bias — is the key background for understanding LLM questions.
• Spark versus MapReduce differences are important to remember: in-memory processing, RDD, caching, security trade-offs, and the any-data nature of Spark.
• MapReduce material — horizontal and vertical scaling, file system distribution, data replication across nodes, and the MapReduce flow with shuffling and sorting — is easy to study and score well on.
• Uptime math may be worth practicing: converting availability percentages into downtime (99.90% gives 43.2 minutes; 99.99999% gives about 86 seconds per day, 6 seconds per week, 5 minutes per year) — the formula is downtime = (1 − a) · T, and the figures come from different periods, so practice each conversion.
• The batch-versus-stream decision is all about latency — time, minutes, hours, seconds — plus what the business needs.
• The seven Vs of big data (one of them, veracity) and the fault-tolerance characteristics of pipelines appear in conceptual questions about big data systems.

Key Industry Applications

• Real-world: AWS cluster monitoring — people in industry monitor nodes, checking whether they started and where data is going; Hive and other libraries run on top of big data systems.
• Real-world: cloud equivalents map one-to-one — AWS Kinesis (Kafka), S3 (storage), EMR file system (HDFS), Amazon MapReduce, Elasticsearch (search), DynamoDB (relational store); Google Cloud Storage, Cloud Bigtable, Cloud Datastore, Cloud Dataflow, Cloud Dataproc, Cloud SQL, and BigQuery. Multi-cloud practitioners keep these maps so a pipeline can be rebuilt on another vendor without redesigning the architecture.
• Real-world: cloud-native platforms in production — Snowflake (virtual warehouses, no vendor lock-in), BigQuery (serverless, automatic scaling, standard SQL, predictive analytics, Looker visualization), Databricks (Spark plus Delta Lake, lakehouse architecture, MLflow evaluation matrices), plus tools like RapidMiner and Presto.
• Real-world: uptime contracts — Oracle Cloud OCI offers clients a zero-downtime commitment with SLA penalties when infrastructure is at fault; aerospace systems run dual Flight Management Systems on real-time operating systems like VxWorks with zero allowed downtime.
• Real-world: LLM data engineering — OpenAI's rare-book harvesting shows the industry's hunger for quality corpora; organizations are building mini, domain-specific, company-specific, and customer-specific LLMs; fintech teams adopt GenAI with corpus design first.
• Real-world: LLM operations — Gemini API version changes break applications until the version is pinned; companies struggle with token usage minimization and hallucination monitoring; WhatsApp-style lock-in is the cautionary tale for model and tool dependence.
• Real-world: retrieval-augmented generation (RAG) is the non-parametric memory of LLM applications, with vector databases built by the LLM data pipeline; traceability and freshness of the retrieved data decide trustworthiness.
• Real-world: research and tools — ML testing automation of RAG pipelines (e.g., healthcare diseases) is an open research area; NotebookLM converts documents into notebooks, note cards, and flashcards for study; Spark MLlib (ML Spark) is the machine learning library of the in-memory engine.