Design Patterns for ML Systems: Registry, Serving, and Agentic Orchestration

7.1 The Model Registry Pattern

7.1.1 Definition and Motivation

A model registry is a centralized repository that stores every artifact associated with a trained machine learning model — the model binary, the dataset used to train it, the algorithm, the hyperparameters, the evaluation metrics, and system-level performance data. The registry assigns a unique version to each entry and timestamps it so that any past configuration can be retrieved and compared.

7.1.2 Historical Lineage: From Code Versioning to ML Versioning

The registry pattern did not appear from thin air — it extends a lineage of version-control thinking that software engineering has been refining for decades.

7.1.3 What Problem Does the Registry Solve?

A registry makes "which model is in production right now?" a one-click question rather than a forensic investigation.

7.2 MLflow Registry in Practice

7.2.1 Stage Management: Staging and Production

MLflow is an open-source platform that implements the model registry pattern. It provides a web interface running on a configurable port (default: 5000) through which teams can browse, compare, and promote models. The key organizational concept is stage management — models move through lifecycle stages that control which version is exposed to production traffic.

7.2.2 What Gets Stored in a Registry Entry?

When a model is registered, MLflow captures a rich set of metadata. This is what turns a loose collection of experiments into a queryable, auditable inventory.

7.2.3 The Bigger Picture

MLflow will be revisited during deployment discussions where Docker, containers, and Kubernetes enter the picture. The registry demo here is only to show the pattern — the deployment mechanics come later.

7.3 The Registry Pattern Beyond MLflow

7.3.1 Tool-Agnostic Pattern and Alternative Implementations

While MLflow is a popular open-source implementation, the registry pattern is tool-agnostic. Several production-grade alternatives exist, each with a different emphasis:

7.3.2 What All Registries Share

Regardless of the tool, every model registry provides a common set of capabilities. Understanding these shared features helps you evaluate any new tool that enters the market.

7.3.3 Antipatterns the Registry Addresses

A registry makes "which model is in production right now?" a one-click question rather than a forensic investigation.

7.3.4 Choosing a Registry

7.4 Separating Models from Business Logic

7.4.1 Models as Replaceable Components

A registry encourages treating the model as a replaceable component. The business logic — API routing, input validation, response formatting — stays fixed. The model behind it can be swapped by updating a version pointer in the registry. This separation means an A/B test can point 5% of traffic at model version 6 while the remaining 95% sees version 5, with no code deployment required.

7.4.2 Practical Benefits of Separation

7.4.3 The Stable Interface Contract

Think of this like an API contract in microservices. The model is a service with a defined request/response schema. As long as the contract holds, the consumer (business logic) does not care what happens inside.

7.4.4 Connection to Model Composition

7.5 Feature Stores and Training-Serving Skew

7.5.1 Definition and Registry Connection

A feature store is a centralized catalog of feature definitions and pre-computed feature values. It emerged from the same registry thinking that produced model registries — if we version models, we should also version the features that feed into them.

7.5.2 Training-Serving Skew in Depth

7.5.3 Feature Store Tools and Architecture

Several production-grade feature store implementations exist:

A typical feature store architecture has two tiers:

┌─────────────────────────────────────────┐
│           Feature Store                  │
│  ┌───────────────┐  ┌────────────────┐  │
│  │  Offline Store │  │  Online Store   │  │
│  │  (data lake /  │  │  (Redis /       │  │
│  │   warehouse)   │  │   DynamoDB)     │  │
│  │  Batch queries │  │  Low-latency    │  │
│  │  Training data │  │  Serving data   │  │
│  └───────────────┘  └────────────────┘  │
│         Same feature code computes both  │
└─────────────────────────────────────────┘

The offline store holds historical feature values for training (batch queries, no latency constraint). The online store holds the latest feature values for serving (millisecond retrieval). Critically, the same feature computation code populates both stores — this is how skew is prevented.

7.5.4 Connection to the Registry

7.6 The Two Serving Paradigms

When a trained model is put into production to make predictions, the architecture of how those predictions are delivered falls into one of two categories. These are deployment patterns — they describe how the model serves predictions, not how it was built.

7.6.1 Batch Prediction (Offline Inference)

In batch prediction (also called offline inference), data accumulates over a period — hours or a full day — and predictions are generated on a schedule. A typical setup works like this:

A common implementation uses AWS SageMaker with an S3 bucket as the data store. The SageMaker cron job pulls data from S3, runs the model, and writes predictions back to S3 or DynamoDB. Applications then read the stored predictions.

7.6.2 Real-Time Prediction (Online Inference)

In real-time prediction (also called online inference or on-demand inference), the model stays loaded in memory continuously — 24 hours a day, 7 days a week, 365 days a year. A user sends a request, the model processes it immediately, and the response returns within milliseconds or seconds. Behind the scenes, an API receives the request, invokes the model, and returns the prediction.

User request
    │
    ▼
┌──────────────┐
│ API Gateway  │  ← Authentication, rate limiting, routing
└──────┬───────┘
       │
       ▼
┌──────────────────┐
│  Model Service    │  ← Model always in memory, ready to serve
│  (REST / gRPC)    │
└──────┬───────────┘
       │
       ▼
  Prediction response
       │
       ▼
┌──────────────────┐
│  Logging &        │  ← Every request/response logged for monitoring
│  Monitoring       │
└──────────────────┘

Real-time prediction is necessary when the response must be immediate:

• Fraud detection: A credit card transaction hits the bank's system. Within milliseconds, a model must decide whether to approve or block it. Waiting until the next batch run is not an option — the fraud would already have succeeded.
• Dynamic pricing: When you open Ola or Uber, a pricing model considers current demand, available drivers, weather, and time of day to compute a fare. The price changes in real time as conditions change.
• Spam filtering: Every incoming email must be classified before it reaches the inbox. A batch job that runs once a day would let spam through for up to 24 hours.
• Generative AI applications: ChatGPT, Gemini, and similar systems give responses as soon as you type a prompt. They run APIs behind the scenes that invoke models in real time.

The architecture for real-time serving typically involves an API gateway, a model service, and auto-scaling infrastructure. The examples discussed in earlier lectures — the microservices pattern with an API gateway on localhost:8001, the pipe-and-filter pattern — are all real-time serving architectures once deployed to production.

7.6.3 Model Misbehavior: Risk and Remediation

7.6.4 The "Razor Knife" Analogy and Gemini 1.0

7.6.5 Side-by-Side Comparison

7.6.6 Real-World: Both Paradigms Coexist

7.7 Architectural Implications

7.7.1 Infrastructure and Cost Trade-offs

The batch and real-time distinction is not just a technical choice — it drives infrastructure decisions, cost models, and operational complexity. Choosing one over the other has cascading effects on everything from hardware provisioning to team skill requirements.

7.7.2 Operational Complexity

7.7.3 The Middle Ground: Hybrid and Cached Approaches

7.7.4 Choosing the Right Paradigm: Decision Framework

7.7.5 Cost Modeling Example

7.8 When to Use Which?

7.8.1 Decision Heuristic

The decision heuristic is straightforward. If the user or downstream system needs the prediction now — within seconds of the request — choose real-time. If the prediction can wait hours or until the next business cycle, batch is more economical. Some systems use both: real-time for latency-sensitive decisions, batch for analytics and reporting.

Think of it like ordering food. Ordering from a meal-prep service that delivers a week's worth of meals on Sunday is batch — cheap per meal, but you wait until the scheduled delivery. Ordering from a food-delivery app is real-time — you get it in 30 minutes, but you pay a premium per order. The food quality (model accuracy) might be the same; what changes is when you get it and how much it costs.

7.8.2 Hybrid Architectures

Many production systems do not choose one or the other — they combine both paradigms.

7.8.3 Quick Decision Checklist

Ask these five questions in order. The first one that gives a clear answer decides the paradigm:

1. Does the consumer need the result within seconds? → Real-time.
2. Can the prediction be precomputed and stored? → Batch.
3. Is the input data only available in batches (e.g., nightly ETL)? → Batch.
4. Does the prediction depend on live user context (location, session)? → Real-time.
5. Do you need both historical analysis and live decisions? → Hybrid.

Real-world: The patterns discussed throughout the course — pipe-and-filter, CQRS, RAG, microservices — all describe real-time serving architectures once deployed. Batch serving has not been demonstrated yet in the course examples because those patterns are fundamentally about handling live requests.

7.9 From Predictive to Generative to Agentic

7.9.1 The Three Paradigms of AI Systems

Machine learning systems have evolved through three paradigms. Predictive AI answers "what will happen?" — will this customer churn, is this transaction fraudulent. Generative AI answers "create something" — write an email, generate an image, summarize a document. The third and newest paradigm is Agentic AI.

An agent is any piece of software that can perform a task — a prediction model, a classifier, a regressor. Agentic AI is different: it refers to AI systems that can pursue complex goals with limited direct supervision. At its highest level of autonomy, you give a single prompt and the system figures out every step — which tools to use, what sub-tasks to decompose into, when to ask for clarification. We are not fully there yet, but building-block autonomy is already deployed: within a well-defined step, an agentic system can make decisions, call APIs, and iterate without step-by-step human instructions.

7.9.2 Request-Response vs. Goal-Oriented

Traditional AI (predictive and generative) follows a request-response pattern. You ask a question; the system answers. With multi-turn interaction, it remembers the conversation window. But it has no tool access — everything it knows is from its training data. It is stateless: each request is treated as independent unless wrapped in a conversation context.

Think of the difference like this: a calculator is request-response — you type "2 + 2" and get "4." A human project manager is goal-oriented — you say "ship the feature by Friday" and they break it into tasks, assign people, track progress, and handle blockers. Agentic AI aspires to be closer to the project manager than the calculator.

7.9.3 What Makes Agentic AI Architecturally Different

Agentic AI pursues goals over time. It uses multi-step reasoning, actively selects and invokes tools (web search, code execution, database queries), and maintains memory and context across interactions. The systems you interact with today — Codex generating entire codebases, Claude writing and testing code — are agentic applications. Behind the scenes, multiple agents run: one behaves like a developer, another like a tester, a third like a reviewer. They communicate, collaborate, and produce a finished result.

7.10 Chatbot versus Agent: A Comparative View

7.10.1 Capability Comparison with Examples

Consider the prompt: "Plan my trip to Goa."

A chatbot (generative AI) will respond with text: "Here are the best beaches — Baga, Anjuna, Calangute. You should visit these restaurants. The weather in March is pleasant." It gives you information, but you must then separately search flights, compare hotels, and make bookings.

An agent (agentic AI) will search for flights, check hotel availability, compare prices, book the reservation (if you have granted booking permissions), and add the itinerary to your calendar. It chains multiple actions together to accomplish a goal, not just describe one.

Think of the difference like asking a librarian versus hiring a personal assistant. The librarian (chatbot) can tell you which books are relevant to your research — you still have to read them, take notes, and write the paper. The personal assistant (agent) reads the books, drafts the paper, formats the citations, and submits it — you review the output.

7.10.2 Controlling Agent Behaviour: Preventing Unbounded Loops

7.11 The Five Core Components of an AI Agent

Every AI agent is built from five components. The specific implementation varies by tool and use case, but the conceptual architecture is universal.

┌─────────────────────────────────────────────────────────┐
│                     PLANNING                            │
│          Task decomposition & sub-task sequencing       │
├─────────┬───────────────────────────────┬───────────────┤
│  MEMORY │         LLM CORE              │    TOOLS      │
│ (4 types)│    (Reasoning Engine)         │ (Web, APIs,   │
│         │                               │  Code, DB)    │
├─────────┴───────────────────────────────┴───────────────┤
│                  ACTION LOOP                            │
│          Observe → Think → Act → Reflect                │
└─────────────────────────────────────────────────────────┘

7.11.1 LLM Core — The Reasoning Engine

The large language model is the brain. It can be any foundation model — GPT-5.4, Gemini 3.5, Grok, Claude, or an open-weight model like Gemma or Mistral. In a multi-agent setup, different agents can use different LLMs. You might assign code-generation tasks to a model known for coding (like Claude or specialized GPT variants) while using a lighter model for classification. During experimentation, use smaller models (GPT-4.1 nano, Gemma 2B) to keep costs low. In production, switch to more capable models.

Chain-of-thought reasoning is what makes LLMs effective as reasoning engines. When asked "Give me Python code for binary search," the model does not jump to the answer. It thinks: "We need a sorted array. If it is not sorted, the first step is to sort it. We then divide it into two halves. Compare the middle element with the target. If equal, return the index. If the target is smaller, search the left half; otherwise, search the right half." This intermediate reasoning — visible in tools like DeepSeek R1 — is the same process a human interviewer expects when asking the same question. Each reasoning step connects logically to the next.

7.11.2 Memory — Persistence Across Interactions

Memory in agentic systems is not a single storage bucket. There are four distinct types, each serving a different need.

In-context / working memory is the conversation window you see when you open ChatGPT. You give a prompt; the system responds. You follow up with "make it shorter" — it knows what "it" refers to because the conversation history is held in working memory. This is temporary: it persists only for the duration of the conversation session. It is the memory you, as a user, can see and scroll through.

Semantic knowledge base is an externally curated, static dataset. A medical domain knowledge base, for instance, contains policy documents, clinical guidelines, and research findings. Anyone asking a medical question gets the same answer drawn from this shared knowledge. It is global (not personalized) and relatively static (updated only when the knowledge base itself is updated). A fine-tuned model is an example: you take GPT-5.4 and fine-tune it on medical data. The resulting model carries that domain knowledge for all users.

External / episodic memory is personalized per user and dynamic. Consider a banking chatbot serving 100 customers. Customer A asks about a transaction two days ago. Today, Customer A asks a follow-up question. The chatbot looks up Customer A's previous interactions in a vector database and tailors the response. Customer B asking the same question would get a different answer because their transaction history is different. Episodic memory is persistent, personal, and updated with every user interaction.

Procedural memory / skills is the most transient form. When an agent calls a weather API, the API's response is temporarily held so it can be passed to the next agent in the workflow. It is not permanently stored; it is the working state of a function call. The distinction from in-context memory is that procedural memory specifically concerns data flowing through tool invocations — information the agent could not produce from its trained model and had to fetch from an external system.

7.11.3 Action Loop — Observe, Think, Act, Reflect

The agent continuously cycles through four phases: observe the current state of the task, think about what step to take next, act by invoking a tool or generating output, and reflect on whether the action moved the goal closer. This loop runs until the task is complete or a stopping condition is met.

7.11.4 Tools — The Interface to the External World

Tools are what let an AI agent reach beyond its training data. When ChatGPT 3.5 launched in 2022, asking "Who is the Prime Minister of India?" would return "As of my knowledge cutoff in September 2021, Narendra Modi is the Prime Minister." It could not access real-time information. Today, when you ask the same question, the model performs a web search, retrieves current information, and answers with the latest data.

Common tools include:

• Web search: retrieve current information beyond the model's training cutoff
• Code execution: run Python, SQL, or shell commands
• API calls: invoke external services (weather APIs, payment gateways, prediction endpoints)
• Database queries: read from or write to structured and vector databases
• File operations: read uploaded documents, generate output files

JSON has become the standard data interchange format for tool communication because it is lightweight, universally supported, and human-readable. Every major cloud provider — AWS, Google Cloud, Azure — uses JSON for policy documents and API payloads.

7.11.5 Planning — Task Decomposition

Planning is how an agent breaks a complex goal into manageable sub-tasks. A "plan my trip to Goa" request becomes: search flights → compare prices → check hotels → book flight → book hotel → add to calendar. Each sub-task may be handled by a different agent (a specialized LLM instance), and the results are assembled into the final output. Planning is the "divide and conquer" layer of agentic systems.

7.12 Agentic AI Applications in Production Today

7.12.1 Current Production Use Cases

Production-grade agentic systems already exist. Codex and Claude for software development are the most visible examples: you describe a feature, and multiple agents collaborate — one writes the code, one generates test cases, one runs the test suite, one opens a pull request. The user reviews and merges.

7.12.2 An Expanded View of Production Patterns

Additional agentic application patterns:

• Fix my code: an agent receives a bug report, analyzes the codebase, writes the fix, runs the existing tests, sees any failures, iterates, opens a PR. The left side is what a developer does manually (copy code into ChatGPT for debugging advice). The right side is the agent doing it autonomously.
• Research a topic: give an agent a research paper. It validates the claims, proofreads the text, identifies strengths and weaknesses, compares with existing literature, and suggests improvements.
• Manage an online store: an agent monitors inventory, adjusts prices dynamically, responds to customer queries, and processes orders.
• Monitor my inbox: an agent continuously reads incoming emails, drafts replies, flags urgent messages, and unsubscribes from junk — though most people still prefer human-in-the-loop for replies, especially in professional contexts.

7.12.3 The Spectrum of Agent Autonomy

7.13 Multi-Agent Communication and the Need for Patterns

7.13.1 Communication Protocols for Agentic Systems

When multiple agents collaborate on a task, they need communication protocols. The patterns studied earlier in the course — microservices, pipe-and-filter, event-driven, CQRS — were designed primarily for predictive and generative AI systems. In the agentic world, where multiple LLM instances act as independent agents passing work to each other, two patterns have emerged as particularly effective: the Saga pattern (for managing distributed, multi-step agent workflows) and the Blackboard pattern (for shared-state collaboration). The Saga pattern is the focus of the remainder of this lecture.

7.13.2 The Parallel: Microservices → Agents

Think of it like a team of specialists in a hospital. The radiologist takes X-rays, the surgeon operates, the pharmacist dispenses medication, and the nurse monitors recovery. Each specialist is autonomous (like an agent), but they must coordinate. If the surgeon cancels the operation, the radiologist does not need to re-take the X-ray, but the pharmacist must cancel the medication order. That compensating action when something goes wrong is exactly what the Saga pattern formalises.

7.13.3 Two Patterns for Multi-Agent Collaboration

7.13.4 When to Use Which Pattern

7.14 Model Composition Patterns

Before diving into the Saga pattern, it is worth noting that multi-model systems already exist in predictive AI. Three composition strategies are common.

7.14.1 Ensembles and Metamodels

Multiple models vote or a metamodel combines their outputs. Random forest is an ensemble of decision trees; stacking uses a secondary model to learn how to weight base model predictions.

Think of ensemble prediction like asking three doctors for a diagnosis. If all three agree, you are confident. If two agree and one disagrees, you trust the majority. A metamodel is like a chief physician who knows that Doctor A is excellent at cardiac cases but unreliable for neurological ones, and weighs their opinions accordingly.

7.14.2 Sequential Decomposition

One model's output becomes the next model's input. In a speech-to-text pipeline, an acoustic model converts audio to phonemes, a language model converts phonemes to words, and a punctuation model adds structure.

Audio → [Acoustic Model] → Phonemes → [Language Model] → Words → [Punctuation Model] → Structured Text

7.14.3 Cascade / Two-Phase Prediction

A fast, cheap model handles the common case; a slower, more accurate model handles the edge cases. A spam filter might use a lightweight rule-based model for obvious spam and invoke a deep learning model only for borderline messages.

Think of it like airport security. Most passengers walk through the metal detector (fast, cheap screening). Only those who trigger an alarm get pulled aside for a full pat-down and luggage inspection (slow, thorough, expensive). The metal detector handles 95% of cases; the detailed inspection handles the remaining 5%.

7.14.4 From Predictive Composition to Agent Composition

7.15 Why Memory Matters

7.15.1 From Stateless to Stateful Systems

All the microservices examples shown earlier in the course were stateless: each request triggered the same pipeline — receive input, run model, return response, forget everything. In agentic systems, this no longer works. An agent needs to remember the user's previous queries, the intermediate results of its own sub-tasks, and the outputs of other agents it collaborates with. This is why memory — in its four forms — is a first-class architectural concern in agentic design.

Think of the difference between a vending machine and a barista. A vending machine (stateless) does not remember you. Every time you press "coffee," it dispenses the same drink. A barista (stateful) remembers that you like oat milk, no sugar, extra shot — and asks "the usual?" when you walk in. The barista's memory transforms the interaction from a transaction into a relationship.

7.15.2 Memory as an Architectural Concern

7.15.3 The Four Types at a Glance

7.16 The Four Types of Memory

7.16.1 Reference Summary

Memory is the mechanism that lets an agent retain information across steps, sessions, and users. Without it, every interaction starts from scratch. There are four distinct types, each with its own storage strategy, lifespan, and purpose.

This section consolidates the memory discussion from §7.11.2 into a reference table.

• In-context / working memory: Temporary, visible to the user, lasts for one conversation session. This is the "scratch pad" — what the model sees right now in its prompt window. When you open a new chat in ChatGPT, the context history you see is the working memory.
• Semantic knowledge base: Externally curated, static, global — the same for all users. Example: a fine-tuned medical model that "knows" drug interactions because it was trained on medical literature. This memory does not change from user to user.
• External / episodic memory: Personalized per user, persistent, dynamic — updated with every interaction. Stored in vector databases or similar retrieval systems. This is how a customer service bot remembers that you prefer email over phone calls.
• Procedural memory / skills: Transient state from tool invocations, passed between agents during a workflow execution. Think of this as the "sticky notes" an agent passes to itself while working through a multi-step task.

The distinction between different memory stores raises another common question about their architecture.

Memory types also interact with workflow execution logic, which leads to a related question.

Real-world: If you build agents using a visual builder (like the OpenAI Agent Builder), you have limited control over memory — you cannot attach a custom vector database or configure retention policies. To build a persistent, production-grade agent, you write your own Python code where you control the storage layer (ChromaDB, Pinecone, PostgreSQL with pgvector) and the memory logic.

7.17 Prompt Engineering as a Control Mechanism

7.17.1 The Five-Step Prompt Methodology

Prompt engineering is not just about getting better answers — it is the primary mechanism for constraining agent behavior and preventing runaway execution. When an agent has access to tools, memory, and multi-step reasoning, an unconstrained prompt is like handing a teenager the car keys with no destination, no speed limit, and no fuel gauge.

Let us now apply this methodology to a concrete scenario.

Real-world: Production teams at companies deploying AI agents maintain prompt registries — versioned collections of prompts with A/B testing. When a new model version ships, they test each prompt against a benchmark suite before deploying. This is prompt engineering treated as a first-class engineering discipline, not an afterthought.

7.18 Architectural Thinking: Why Patterns Matter

7.18.1 Patterns as Reusable Solutions

An architectural pattern is a reusable, named solution to a common structural problem in software design. It is not a specific library or framework — it is a blueprint that tells you how to arrange components, how they communicate, and how they handle failure. Patterns have names so teams can communicate complex designs in a single phrase: "Let's use CQRS" carries more meaning than a paragraph of explanation.

Even when you accept that patterns matter, choosing the wrong one creates its own problems.

Real-world: Companies like Netflix, Uber, and Amazon use architectural patterns extensively. Netflix's microservices architecture handles thousands of services; Uber's trip workflow is a real-world distributed transaction (match rider → assign driver → process payment → track ride). When these systems fail partially, they rely on exactly the kind of compensating logic that the Saga pattern formalizes.

7.19 The Problem: Distributed Transactions

7.19.1 All-or-None Multi-Step Operations

A distributed transaction is a multi-step operation where each step runs as an independent service with its own local data store, and the overall transaction is "all or none" — every step must succeed for the transaction to be considered complete. If any step fails, all preceding steps must be undone.

Step 1: Order Service      → Create order (select restaurant and items)
Step 2: Payment Service    → Process payment (charge the user's account)
Step 3: Inventory Service  → Update inventory (decrement restaurant's stock)
Step 4: Delivery Service   → Deliver order (assign partner, complete delivery)

Step 1: Order Service     → ✅ Order created (order_id: #4521)
Step 2: Payment Service   → ❌ Payment declined (insufficient funds)

Current system state:

Order DB:  Order #4521 exists (status: "pending")
Payment DB: No record (payment was never completed)
Inventory DB: Stock already reserved (was decremented optimistically before payment)
Delivery DB: No assignment (never reached this step)

Step 1: Order Service     → ✅ Order created
Step 2: Payment Service   → ✅ Payment processed (user charged ₹500)
Step 3: Inventory Service → ✅ Stock decremented
Step 4: Delivery Service  → ❌ No delivery partner available

Current system state:

Order DB:  Order exists (status: "pending delivery")
Payment DB: ₹500 charged to user
Inventory DB: Stock decremented
Delivery DB: No record

When a distributed transaction does exist, the natural next question is: "Why not just use a global transaction protocol?"

Real-world: Every ride-hailing app (Uber, Ola) executes a distributed transaction when you book a ride: match rider → assign driver → estimate fare → process payment → track ride. If the driver cancels after assignment, the system must undo the fare hold, release the driver's schedule slot, and re-enter matching — all without a global rollback. This is a real-world Saga in action.

7.20 The Saga Solution: Compensating Transactions

7.20.1 Forward Transactions and Reverse Compensation

A saga is a sequence of local transactions. Each local transaction (T₁, T₂, …, Tₙ) updates the data store of a single service and publishes an event or message to trigger the next transaction. If the sequence completes, the saga is successful.

If transaction Tₙ₊₁ fails, the saga does not magically roll back — because T₁ through Tₙ have already committed to their local databases. Instead, the saga executes compensating transactions (Cₙ, Cₙ₋₁, …, C₁) in reverse order. Each Cᵢ undoes the effects of the corresponding Tᵢ.

The saga pattern has clear inputs and outputs for each transaction instance.

Here is the step-by-step sequence for how compensation executes when a failure occurs.

Forward chain (already committed):
  T₁ ✅ Order #4521 created
  T₂ ✅ Payment of ₹500 processed
  T₃ ✅ Inventory decremented (2 items removed from restaurant stock)

Failure:
  T₄ ❌ No delivery partner available

Compensation chain (executes in reverse):
  C₃ ✅ Restore inventory (add 2 items back to restaurant stock)
  C₂ ✅ Refund payment (₹500 returned to user's account)
  C₁ ✅ Cancel order (Order #4521 status → "cancelled")

Final state: Consistent — as if the order was never placed.

Real-world: Amazon's order processing uses a saga-like pattern. When an order fails at the fulfillment stage, the system triggers reverse compensations: refund the payment, restock the items, and update the order status — all coordinated through their internal event system. During peak events like Prime Day, they process millions of such compensations per hour.

7.21 Saga Participants and Communication

7.21.1 Local Data Stores and Message Passing

Each service in a saga is a saga participant. A saga participant is an independent service that executes one step of the saga — it performs a local transaction, has its own data store, and communicates with other participants through messages. Every participant has its own local data store — this is the defining property. The data store can be relational (PostgreSQL, MySQL), NoSQL (MongoDB, DynamoDB), or even a simple JSON file during prototyping.

┌──────────────┐    ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
│ Order Service │───▶│ Payment Service  │───▶│Inventory Service │───▶│Delivery Service  │
│              │    │                  │    │                  │    │                  │
│ DB: orders   │    │ DB: payments     │    │ DB: stock_levels │    │ DB: assignments  │
│ T₁: create   │    │ T₂: charge       │    │ T₃: decrement    │    │ T₄: assign       │
│ C₁: cancel   │    │ C₂: refund       │    │ C₃: restore      │    │ C₄: (terminal)   │
└──────────────┘    └──────────────────┘    └──────────────────┘    └──────────────────┘

1. Order Service creates order → publishes "OrderCreated" event
   Message: {order_id: 4521, user_id: "U100", restaurant_id: "R55", items: [...], total: 500}


Payment Service (subscribed to "OrderCreated") → receives message

   Processes payment → publishes "PaymentProcessed" event
   Message: {order_id: 4521, payment_id: "P789", amount: 500, status: "success"}


Inventory Service (subscribed to "PaymentProcessed") → receives message

   Decrements stock → publishes "InventoryUpdated" event
   Message: {order_id: 4521, items_decremented: [...], remaining_stock: {...}}


Delivery Service (subscribed to "InventoryUpdated") → receives message

   Assigns delivery partner → publishes "DeliveryAssigned" event
   Message: {order_id: 4521, partner_id: "D42", eta: "25 min"}

A subtler but equally dangerous mistake involves how you set up the data stores themselves.

Real-world: Kafka is the backbone of sagas at companies like LinkedIn (which created Kafka), Uber, and Netflix. When you complete an Uber ride, a "TripCompleted" event flows through Kafka to trigger payment processing, driver rating, receipt generation, and analytics — all as independent subscribers reacting to the same event.

7.22 Orchestration-Based Saga

7.22.1 Centralized Controller Model

In orchestration, a centralized controller — the orchestrator — manages the entire saga flow. The orchestrator knows the sequence: call service A, receive response, call service B, receive response, call service C, and so on. It also knows the compensation sequence: if service C fails, call C's compensation, then B's compensation, then A's compensation.

         🎵 Conductor (Orchestrator)
        ╱        |         ╲
       ╱         |          ╲
  🎻 Violins   🥁 Drums    🎤 Vocals
 (Order)     (Payment)    (Delivery)

With the orchestrator's role defined, here is how a complete orchestration-based saga executes step by step.

Saga Instance #4521:
┌─────────────────────────────────────────────────────────────┐
│ Step │ Service     │ Forward │ Compensation │ State         │
│──────│─────────────│─────────│──────────────│───────────────│
│  1   │ Order       │ T₁      │ C₁           │ ✅ completed  │
│  2   │ Payment     │ T₂      │ C₂           │ ✅ completed  │
│  3   │ Inventory   │ T₃      │ C₃           │ ❌ failed     │
│  4   │ Delivery    │ T₄      │ C₄           │ ⏹ skipped     │
└─────────────────────────────────────────────────────────────┘

Compensation triggered:
  → Executing C₂ (refund payment)... ✅
  → Executing C₁ (cancel order)...  ✅
  → All compensations complete. Saga status: COMPENSATED

Now let us see how this looks in a real orchestration tool.

{
  "StartAt": "CreateOrder",
  "States": {
    "CreateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:order-service",
      "Next": "ProcessPayment",
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "HandleFailure" }]
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:payment-service",
      "Next": "UpdateInventory",
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RefundPayment" }]
    },
    "UpdateInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:inventory-service",
      "Next": "DeliverOrder",
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RestoreInventory" }]
    }
  }
}

Another common source of confusion is the relationship between orchestration and communication style.

Real-world: Netflix uses orchestration for its content delivery pipeline. When new content is uploaded, an orchestrator coordinates transcoding, subtitle generation, quality checks, and global distribution — each step handled by an independent service. If transcoding fails, the orchestrator triggers compensation (clean up partial files, notify the content team, reset the upload status).

7.23 Choreography-Based Saga

7.23.1 Decentralized Event-Driven Model

Hook: Imagine a relay race without a coach standing on the sidelines. Each runner has trained for their specific leg of the race. When the previous runner hands them the baton, they run their segment and hand off to the next runner. No one is calling out instructions from the edge of the track. The coordination is distributed — built into each runner's preparation, not into a central authority. This is exactly how choreography-based sagas work in distributed systems.

7.23.2 Purpose

The purpose of choreography-based saga is to coordinate a multi-step distributed transaction without a central authority. The result is maximum decoupling (services do not need to know about each other's existence — they only know the broker). This makes each service independently deployable, independently scalable, and independently maintainable by separate teams.

7.23.3 Inputs and Outputs

• Input to each service: An event arriving through the message broker (e.g., OrderCreated, PaymentProcessed).
• Output from each service: A new event published back to the broker after the local transaction commits (e.g., PaymentProcessed, InventoryUpdated). If the local transaction fails, the service publishes a failure event instead (e.g., InventoryFailed).

7.23.4 Step-by-Step Process

Normal flow (happy path):

1. An external trigger (e.g., a user placing an order) publishes an initial event — OrderCreated — to the message broker.
2. Service A (Order Service) has already committed its local transaction and published this event. Its job is done.
3. Service B (Payment Service), subscribed to OrderCreated, receives the event, processes the payment locally, commits, and publishes PaymentProcessed to the broker.
4. Service C (Inventory Service), subscribed to PaymentProcessed, receives the event, updates inventory locally, commits, and publishes InventoryUpdated.
5. The chain continues until all local transactions have completed. The full saga is complete.

Failure flow (compensation path):

1. Suppose Service C (Inventory Service) fails — the requested item is out of stock.
2. Instead of publishing InventoryUpdated, Service C publishes InventoryFailed to the broker.
3. A compensating consumer (a service subscribed to failure events for that step) picks up InventoryFailed and triggers the reverse compensation chain.
4. Payment Service, receiving the compensation trigger, executes its compensating transaction (refund) and publishes PaymentReversed.
5. Order Service, receiving PaymentReversed, executes its compensating transaction (cancel the order) and publishes OrderCancelled.

7.23.5 Tracing the Flow

In choreography, tracing is harder than in orchestration because no single component holds the complete picture. To trace a saga:

• Each event must carry a correlation ID (a shared identifier that links all events belonging to the same saga instance).
• Observability tools (distributed tracing systems like Jaeger or OpenTelemetry) aggregate events from the broker using correlation IDs to reconstruct the full saga flow.
• Log aggregation across all services is essential for debugging.

7.23.6 Complexity Dimensions

Choreography-based sagas trade centralized simplicity for distributed flexibility. The key complexity dimensions are:

• Messaging infrastructure: You must set up and operate a message broker (Kafka, RabbitMQ, SQS). The broker becomes critical infrastructure — if it goes down, the entire saga stops.
• Event schema design: Every event type (OrderCreated, PaymentFailed, etc.) must be carefully designed and versioned. Schema changes can break subscribers.
• Distributed tracing: Without a central coordinator, you need correlation IDs and observability tools to follow a saga across services.
• Debugging difficulty: When something goes wrong, you must aggregate logs from multiple services and the broker to piece together what happened.
• Testing complexity: End-to-end tests must simulate event flows across multiple services. Contract testing between publishers and subscribers becomes essential.
• Message ordering: Events may not arrive in strict order. Services must handle out-of-order delivery gracefully (e.g., using sequence numbers or idempotent operations).
• Scaling: The broker scales horizontally to handle more events. Individual services scale independently by adding more consumer instances. There is no single bottleneck.

7.23.7 When to Use Choreography

Choreography is the right choice when:

• Maximum service independence is the priority — teams own individual services and deploy autonomously.
• The flow is linear or simple — a straightforward chain of steps without complex conditional branching.
• Scalability is critical — no central bottleneck means the system scales horizontally.
• Real-time event-driven systems — IoT event pipelines, order processing, and notification systems.

Choreography is not ideal when:

• The flow has complex branching logic with many conditional paths (orchestration handles this more naturally).
• Centralized visibility is required for compliance or auditing.
• The flow is hard to reason about from events alone — if you need a whiteboard diagram to understand the sequence, an orchestrator might be clearer.

7.23.8 Real-World Applications

Food delivery platforms (Swiggy, Zomato) and ride-hailing services (Ola, Uber) use choreography-based coordination for distributed transactions across their microservices architecture. These are multi-billion-dollar companies operating at massive scale — proof that choreography works in production for real-time, high-throughput event-driven systems.

Message brokers like Kafka, RabbitMQ, and AWS SQS provide the backbone for choreography-based sagas. Kafka, in particular, offers durable event storage (events are retained for a configurable period), which enables event sourcing and replay — capabilities that make choreography more robust in practice.

7.24 Orchestration versus Choreography: Trade-offs

7.24.1 Choosing Between Two Coordination Strategies

This section provides a decision framework. Use the comparison table below to evaluate which pattern fits your specific situation.

7.24.2 Comparison Table

7.24.3 The Hybrid Reality

7.24.4 Decision Checklist

Use this quick checklist to guide your choice:

Choose orchestration when:

• The flow has complex branching logic (if-then-else, parallel forks, conditional compensation)
• You need centralized visibility for compliance, auditing, or regulatory reporting
• The flow is long-running (hours or days) and needs checkpoint-based recovery
• A single team maintains the coordination logic

Choose choreography when:

• The flow is a simple linear chain of steps
• Teams own individual services and need autonomous deployment cycles
• High throughput and low latency are priorities (no central bottleneck)
• The system is already event-driven (e.g., built on Kafka)

Choose a hybrid when:

• The critical business flow needs centralized control
• Side effects (notifications, analytics, audit) can tolerate eventual consistency
• You want to minimize coupling for non-critical paths while keeping the core path tightly coordinated

7.25 The Saga Pattern in Agentic AI Workflows

7.25.1 Agents as Saga Participants

Hook: Think of a consulting firm tackling a complex client engagement. A senior partner (the supervisor) breaks the problem into specialized workstreams: market research, financial modeling, legal review, and strategic recommendations. Each specialist works independently with their own data and tools. If the financial modeler's analysis reveals a flaw, the partner does not restart the entire engagement — they reroute that piece to another modeler, ask for a revised analysis, or adjust the downstream work accordingly. This is exactly how the Saga pattern coordinates multi-agent AI systems.

7.25.2 What Is an Agent's "Local State"?

In the Saga pattern for microservices, each service has its own database. In agentic AI, each agent has its own local state — a conceptually equivalent idea:

• Conversation context: The messages exchanged between the agent and its tools or sub-agents so far.
• Retrieved documents: Results from vector database queries, web searches, or knowledge base lookups.
• Tool outputs: Return values from API calls, code execution, or file operations.
• Intermediate reasoning: Chain-of-thought traces, planning steps, and intermediate conclusions.

This local state is what makes each agent an independent participant. If one agent fails, its local state can be inspected, discarded, or used as input for a retry — just like a compensating transaction inspects and reverses a service's local database changes.

7.25.3 The Supervisor Agent as Orchestrator

In an orchestrated agentic saga, one agent takes the role of supervisor (also called an orchestrator agent or manager agent). The supervisor's responsibilities mirror the saga orchestrator:

1. Receive the user's goal — e.g., "Plan a 5-day trip to Tokyo within a budget."
2. Decompose into sub-tasks — search flights, find hotels, check calendar availability, assemble itinerary, validate budget.
3. Assign sub-tasks to specialist agents — a flight search agent, a hotel recommendation agent, a calendar integration agent, a budget analysis agent.
4. Collect outputs — each specialist returns its result (or a failure signal).
5. Validate results — the supervisor checks whether outputs are coherent, complete, and within constraints.
6. Invoke compensating logic on failure — if the hotel agent fails or returns no results, the supervisor does not halt the entire workflow. Instead, it applies a compensation strategy.

7.25.4 Compensation Strategies for Agents

The five compensation strategies for agents are:

These strategies map directly to compensating transactions in traditional sagas: they are application-level functions that undo, correct, or work around the effect of a failed or suboptimal step.

7.25.5 Assumptions and Scope

The Saga pattern in agentic AI assumes:

• Agents are independently executing units — each agent runs its own inference, manages its own context window, and can fail independently.
• Communication is structured — agents pass structured messages (not free-form text) to enable the supervisor to parse and validate outputs.
• Failure is expected — LLM agents are probabilistic. The system is designed for failure recovery, not failure prevention.
• Token budgets are finite — retries and reprompts consume tokens and add latency. Compensation strategies must account for cost and time constraints.

7.25.6 Industry Validation

Real-world: AWS published guidance on adapting the Saga pattern to agentic AI workflows approximately six months before this lecture. The adaptation recognizes that LLM-based agents, like microservices, are independently executing units with their own state, and they need a coordination mechanism that handles partial failures gracefully. This is not a theoretical exercise — it is an active area of industry adoption.

The pattern's core insight — forward steps followed by reverse compensation on failure — applies unchanged to agentic systems. What changes is the form of compensation: instead of database rollbacks, you have output discarding, reprompting, rerouting, and graceful degradation.

7.26 Key Terminology Summary

7.26.1 Glossary of Saga Pattern Terms

Saga

A sequence of local transactions coordinated to achieve a single logical outcome. Each local transaction has a paired compensating transaction that can undo its effect if a later step fails. The saga ensures that either all steps complete successfully, or the effects of completed steps are reversed.

Example: A food delivery order saga: create order → process payment → update inventory. If inventory update fails, the saga triggers compensating transactions: refund payment, then cancel order.

Saga participant

A service (or agent) that performs one local transaction within the saga and maintains its own data store or local state. Participants do not share databases — they communicate only through events or orchestrator commands.

Example: In the food delivery saga, the Order Service, Payment Service, and Inventory Service are each saga participants. In an agentic system, each AI agent (flight search agent, hotel agent, budget agent) is a participant.

Compensating transaction

The reverse operation that undoes the effect of a previously committed local transaction. Compensating transactions are application-level functions — they are not automatic database rollbacks. They execute in reverse order to unwind the saga's effects.

Example: The compensating transaction for "process payment" is "refund payment." For "create order," it is "cancel order." In agentic AI, compensation might mean "discard agent output" or "retry with a different prompt."

Orchestration

A centralized saga coordination strategy where a single orchestrator (a dedicated service or function) knows the entire flow, invokes each participant in sequence, handles responses, and manages compensation. The orchestrator is the single source of truth for the saga's state.

Example: AWS Step Functions executing a workflow: the state machine definition specifies the sequence of steps, error handling, and retry logic. Each step invokes a Lambda function or service.

Choreography

A distributed saga coordination strategy where participants communicate solely through events on a message broker. There is no central controller. Each participant reacts to events it receives and publishes new events when its local transaction completes (or fails).

Example: A Kafka-based order pipeline where the Order Service publishes OrderCreated, the Payment Service subscribes and publishes PaymentProcessed, and the Inventory Service subscribes and publishes InventoryUpdated. No single service orchestrates the flow.

Distributed transaction

A multi-step operation that spans independent services (or agents), each with its own data store, requiring all-or-none completion. The challenge is that traditional ACID transactions cannot span multiple independent databases — hence the need for the Saga pattern.

Example: Booking a flight requires reserving a seat (airline service), charging a credit card (payment service), and sending a confirmation (notification service). Each step touches a different database.

Message broker

Middleware that receives events from publisher services and routes them to subscriber services. The broker decouples publishers from subscribers — neither needs to know about the other's existence. Common brokers include Apache Kafka (distributed event streaming platform), RabbitMQ (message queuing system), and AWS SQS (managed queue service).

Example: In a choreography-based saga, Service A publishes OrderCreated to a Kafka topic. Service B, subscribed to that topic, receives the event and processes it. If Service B is temporarily down, the broker holds the event until Service B recovers.

7.26.2 Terms in Context

Exam Guidance Summary

What the Midterm Covers

The midterm syllabus covers everything from the beginning of the course through the Blackboard pattern. Lecture 8 (the next class) will complete the Saga pattern — covering orchestration and choreography in detail with code examples — and introduce the Blackboard pattern. That will be the final topic included in the midterm.

About the Sample Questions

Sample questions will be released this week. They are indicative only — the format (question types, structure) will be representative, but the exact questions on the exam will differ. This course is being offered for the first time, so there are no past papers. Pay close attention to the sample questions when they arrive to understand the expected format, depth, and style.

High-Value Exam Topics

Key Concepts to Review

Key Industry Applications

Model Development and Registry

• MLflow: Open-source model registry used in the lecture demonstration. Integrates with any ML framework (scikit-learn, TensorFlow, PyTorch). Tracks model versions, parameters, metrics, and artifacts. The industry standard for experiment tracking and model lifecycle management.
• AWS SageMaker: Managed ML platform with built-in batch transform jobs. Uses S3 for data storage and DynamoDB for metadata. Provides end-to-end ML workflows from training to deployment.

Saga Orchestration and Workflow Management

• AWS Step Functions: Managed orchestration service for implementing orchestration-based sagas. Defines workflows as state machines with built-in error handling, retries, and parallel execution. The natural choice for orchestrating distributed transactions on AWS.

Agentic AI Tooling

• OpenAI Agent Builder / Agents SDK: Visual and programmatic tools for building multi-agent workflows. Supports Python and TypeScript deployment. Enables the supervisor-and-specialist agent pattern covered in Section 7.25.

Event-Driven Infrastructure

• Message brokers (Kafka, RabbitMQ, AWS SQS): The backbone of choreography-based sagas and event-driven architectures. Kafka provides durable event streaming with replay capability. RabbitMQ offers flexible message routing. SQS provides managed, serverless queuing.

Canonical Industry Examples

• Food delivery (Swiggy, Zomato): The canonical example of distributed transactions managed by the Saga pattern. Order creation, payment processing, inventory management, and delivery coordination are independent services coordinated via sagas.
• Ride hailing (Ola, Uber): Real-time dynamic pricing models and distributed transaction flows. Driver matching, fare calculation, payment processing, and ride tracking each run as independent services.
• Streaming (Disney Hotstar): An example of massive-scale infrastructure handling concurrent sagas — millions of viewers streaming simultaneously, each session involving multiple coordinated services.

Production Agentic Systems

• Codex, Claude: Production-grade agentic AI systems with multi-agent collaboration behind the scenes. These systems demonstrate the coordination patterns discussed in this lecture operating at scale.

Memory and Storage for Agents

• Vector databases (ChromaDB, Pinecone): Storage layer for episodic and semantic memory in agentic systems. Enable retrieval-augmented generation (RAG) by storing and querying document embeddings.
• Gemma 2B, Mistral 7B: Open-weight models suitable for local agent development and experimentation. Lower resource requirements make them accessible for building and testing multi-agent prototypes.