Software Engineering for Machine Learning

Saga Pattern and Architectures for Agentic AI

Published: 2026-07-26

Level: postgraduate

Audience: Postgraduate students in Machine Learning

Saga Pattern and Architectures for Agentic AI

8.1 The Saga Pattern — Distributed Transactions in Microservices

Why can't we just use a database transaction? When a business operation spans multiple independent services — each with its own data store — there is no single database engine to guarantee atomicity. The Saga pattern was invented to solve exactly this gap.

When a system is split across multiple independent services — each with its own data store — a single business operation (placing an order, booking a ride) can no longer rely on a single monolithic database transaction. Every microservice may use a relational, non-relational, or any other kind of data store, and there is no central database to enforce ACID properties across them all. This is the distributed transaction problem, and the Saga pattern is one of the most popular solutions for it.

Intuition: the food-delivery chain Think of ordering food through an app like Swiggy or Zomato. The transaction is not a single step — it is a chain: the customer places an order, the order reaches the restaurant, the restaurant accepts it, payment is processed, inventory is updated, and the order is confirmed. Only after the customer receives the food is the transaction truly complete. Each step in this chain is handled by a different service, possibly with a different database. The Saga pattern treats this entire chain as one logical transaction, even though no single database spans it.

In a monolithic application with one process and one centralized database, the database engine handles this with ACID guarantees — atomicity (all or nothing), consistency (rules always hold), isolation (concurrent transactions do not interfere), durability (committed data survives crashes). But in a microservices architecture, every service owns its own local data store. There is no single database transaction spanning all the steps. The Saga pattern addresses exactly this gap.

The Saga pattern, formally In the Saga pattern, each microservice is called a Saga participant. Each participant executes a local transaction on its own data store and communicates with the next participant either synchronously (via REST, gRPC, or GraphQL) or asynchronously (through a message broker like Kafka, RabbitMQ, or any publish-subscribe event-driven mechanism). The key idea: the sequence of local transactions, taken together, forms the distributed transaction. If every local transaction succeeds, the Saga succeeds. If one fails, compensating transactions undo the work already done — more on this in Section 8.1.1.

A Saga is formally defined as a sequence of local transactions $T_{1}, T_{2}, \dots, T_{n}$ where each $T_{i}$ has a corresponding compensating transaction $C_{i}$ . The Saga succeeds if all $T_{i}$ complete. If $T_{k}$ fails, then $C_{k - 1}, C_{k - 2}, \dots, C_{1}$ execute in reverse order to undo the effects of the already-committed transactions.

Scope: The Saga pattern applies when a business operation spans multiple services, each owning its own data store. It does not replace local ACID transactions within a single service — each service still uses its own database transactions normally. The Saga coordinates across service boundaries, not within them.

The name "Saga" comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem ("Sagas," ACM SIGMOD). It is not an acronym — there is no expansion. The term was chosen to evoke the idea of a long, multi-chapter narrative where each chapter (local transaction) has a defined ending (commit or compensate).

Real-world: Swiggy, Zomato, and ride-booking apps all depend on distributed transaction patterns like Saga. AWS, Azure, and other cloud providers recommend the Saga pattern for microservices-based systems, and it is increasingly reused for ML applications — particularly multi-agent coordination in Agentic AI. Even the AWS Step Functions and AWS Lambda-based examples for Agent TKI (the multi-agent framework) are built on Saga principles. The pattern was born in microservices but has become one of the two pillars (alongside CQRS — Command Query Responsibility Segregation) of distributed system design and is now leveraged for agent coordination.

The Saga pattern replaces a single ACID transaction with a sequence of local transactions, each with a defined compensating transaction for undo. It is the standard approach for distributed transactions across microservices.

8.1.1 Compensating Transactions — Rolling Back in a Distributed World

The fundamental problem: how do you undo across services? In a monolithic database, a failed transaction simply rolls back — the database engine undoes all changes atomically. In a distributed system, there is no such mechanism. If Service A committed its changes and then Service B fails, Service A's changes are already persisted. We need a manual undo — and that is exactly what a compensating transaction provides.

Not every transaction succeeds. If a downstream service fails, the work already completed by upstream services must be undone. This is handled by a compensating transaction: the logical inverse of the original transaction.

Compensating transaction, formally If there are $n$ transactions $T_{1}, T_{2}, \dots, T_{n}$ and $T_{n + 1}$ fails, compensating transactions $C_{n}, C_{n - 1}, \dots, C_{1}$ execute in reverse order. $C_{n}$ undoes whatever $T_{n}$ did — if $T_{n}$ created a record, $C_{n}$ deletes it; if $T_{n}$ modified a value, $C_{n}$ restores the old value; if $T_{n}$ created a file, $C_{n}$ removes that file. The exact undo behavior depends on the nature of the use case. After $C_{n}$ completes, $C_{n - 1}$ fires, and so on, walking backward until every committed local transaction has been compensated.

Whatever transactions exist after the failure point ( $T_{n + 2}, T_{n + 3}, \dots$ ) are never executed — they are simply skipped since the Saga terminates at the failure.

Worked example: ordering a masala dosa Consider a food-delivery order with four steps:

Step	Forward Transaction	Compensating Transaction
1	`create-order(order_id=101)`	`cancel-order(order_id=101)`
2	`reserve-inventory(item=masala-dosa, order_id=101)`	`release-inventory(item=masala-dosa, order_id=101)`
3	`make-payment(amount=150, order_id=101)`	`refund-payment(amount=150, order_id=101)`
4	`confirm-order(order_id=101)`	(no-op — confirmation has no undo)

Success path: All four transactions commit. The customer gets their dosa.

Failure path (payment fails at step 3):

$T_{1}$ and $T_{2}$ already committed.
$T_{3}$ fails (payment gateway returns error).
Compensation runs in reverse: $C_{2}$ releases the reserved inventory, $C_{1}$ cancels the order.
$T_{4}$ (confirm-order) is never executed.

Result: The system returns to the state it was in before the order was placed. No food was prepared, no money was charged.

Naming convention for transaction pairs Always start a transaction name with a verb followed by a noun — create order, cancel order, update inventory, restore inventory, make payment, refund payment. The verb signals the action; the noun signals the domain entity being acted upon. This convention makes the compensating relationship visually obvious: the forward and compensating operations share the same noun but use opposing verbs.

Pitfall: assuming a compensating transaction is always a simple inverse Compensating transactions are not always trivial. If a forward transaction triggered a notification to a third-party system (e.g., sent an SMS to the customer), the compensating transaction cannot unsend the SMS. In such cases, the compensating transaction may need to send a cancellation notification instead, or mark the record so downstream systems handle it gracefully. The architect must reason about what is logically reversible, not just physically reversible.

A compensating transaction can be implemented in one of two ways:

Two separate methods: an execute() method for the forward transaction and a compensate() method for the rollback. Each method has a single responsibility, which is easier to test and reason about.
One method with two conditions: a single method that checks a flag and performs the forward or reverse operation accordingly. This can reduce code duplication when the forward and reverse logic share significant structure.

Both approaches are valid; the choice depends on the developer's preference and the complexity of the operation.

Q: If the system uses non-relational databases instead of relational ones, how are transactions handled? A: The Saga pattern does not depend on the type of data store. Whether the local database is relational (SQL) or NoSQL (DynamoDB, MongoDB, JSON files), each service executes its local transaction and implements a compensating transaction to undo it. The pattern operates at the service level, not the storage-engine level. For fully autonomous agents (levels 5–6 of autonomy), NoSQL is generally preferred because the schema must be flexible — you cannot predict every data shape an autonomous agent will generate. At earlier autonomy levels (3–4), where the outputs are bounded (e.g., structured with Pydantic models, typed as boolean/literal/string), SQL databases can still work well.

Q: Does the compensating transaction logic belong to the same service or a different service? A: Typically it is part of the same service — either a separate endpoint or a separate method within the same class. In practice, the service that knows how to perform a forward transaction is also the best entity to undo it. Placing compensation logic in a different service would create tight coupling and make the system harder to maintain.

Q: What if the compensating transaction itself fails? A: That is a real possibility. The standard approach is a retry mechanism — retry the compensating transaction a configurable number of times. A compensating transaction must itself be reliable, and in production systems this often involves idempotency guarantees (running the same compensation twice has the same effect as running it once) and dead-letter queues for unrecoverable failures. If all retries fail, the system must escalate to human intervention or alert an operator.

Q: In the real world, with multiple concurrent users, how do we track which compensating transaction belongs to which distributed transaction? A: A unique identifier — such as an order ID or customer ID — must be shared across all participants in the transaction. Every service records this identifier so that compensating transactions can target the correct records. The ID propagates through every event, call, or message in the chain. Without this identifier, a compensating transaction might delete the wrong records — a data corruption scenario far worse than the original failure.

Q: Is a Saga transaction always sequential? A: For a single user's distributed transaction, yes — the steps execute in sequence. For instance, ordering one masala dosa involves create-order, then update-inventory, then make-payment, in that order. But across millions of users operating simultaneously, many such sequential transactions run in parallel — that is the scalability benefit of the microservices architecture. The parallelism is at the user level, not within one user's workflow.

Scope and assumptions The compensating transaction model assumes that each local transaction is committable independently — it does not require a two-phase commit protocol. It also assumes that the system can tolerate temporary inconsistency: between $T_{k}$ committing and $C_{k - 1}$ compensating, the system is in an intermediate state. This is acceptable in most eventual-consistency designs but may be unacceptable in safety-critical systems (e.g., medical devices, autonomous vehicles) where strict atomicity is required.

A compensating transaction is the logical inverse of a forward transaction. It executes in reverse order to undo committed work when a downstream step fails. Every forward transaction must have a defined compensating transaction — even if, in some cases, that transaction is a no-op.

8.1.2 Two Implementation Styles — Orchestration vs. Choreography

The central design question: who coordinates? Once you accept that a distributed transaction involves multiple services, the next question is: who decides which service to call next? This question has exactly two fundamental answers, and the choice between them is one of the most important architectural decisions in a microservices system.

The Saga pattern can be implemented in two fundamentally different ways, distinguished by where the coordination logic lives.

Orchestration: centralized control In orchestration, a centralized component — the orchestrator — decides the sequence of service calls: call Service A, get its status, call Service B, get its status, call Service C, and so on. If anyone fails, the orchestrator invokes their compensating transactions in reverse order. The orchestrator is the brain of the entire flow; individual services are "dumb" — they execute whatever the orchestrator tells them, whether it is the forward transaction or the compensating one. The orchestrator is an additional component separate from the business services.

Think of an orchestra conductor: the musicians (services) play their parts, but the conductor (orchestrator) decides who plays when and what to do if someone misses a note.

Choreography: decentralized coordination In choreography, there is no central coordinator. Each service decides, after completing its own work, which event to emit next. Services communicate through events published to a message broker. Each event triggers the next participant. If a failure occurs, a failure event propagates backward through the same broker, and each upstream participant listens for the compensating event and reacts. The intelligence is distributed; every participant knows both what to do on success and what to undo on failure.

Think of a relay race: each runner (service) hands the baton (event) to the next runner. There is no coach on the track deciding the order — the handoff pattern is pre-agreed, and each runner knows exactly when to start running.

Worked example: comparing the same flow in both styles Consider a three-step Saga: Validate then Review then Summarize.

Orchestration: An orchestrator calls the Validation Agent, waits for the result, then calls the Review Agent, waits, then calls the Summary Agent. If the Summary Agent fails, the orchestrator calls review_agent.compensate() and then validation_agent.compensate(). All coordination logic lives in the orchestrator.

Choreography: The Validation Agent emits a validated event. The Review Agent subscribes to validated and, upon receiving it, performs the review and emits a reviewed event. The Summary Agent subscribes to reviewed and processes it. If the Summary Agent fails, it emits a summary_compensated event. The Review Agent subscribes to this event and compensates itself, then emits review_compensated. The Validation Agent subscribes and compensates. No central coordinator exists.

When to pick which style

Dimension	Orchestration	Choreography
Coordination logic	Centralized in orchestrator	Distributed across services
Easier to understand	Yes — flow is explicit in one place	Harder — flow is implicit across event subscriptions
Easier to debug	Yes — orchestrator logs the sequence	Harder — must trace events across brokers
Single point of failure	Yes — if orchestrator crashes	No — services are independent
Coupling	Services are loosely coupled from each other but coupled to orchestrator	Services are fully decoupled
Works well at scale	10–15+ participants	3–4 participants

For simple flows with 3–4 participants, choreography works elegantly — the event graph is small and easy to trace. With 10–15+ participants, the event graph becomes difficult to trace and maintain; orchestration is usually preferred in those cases. A hybrid approach is also common: an orchestrator manages the overall sequence while individual services communicate through events for specific steps.

Q: Is choreography just event-driven architecture? What distinguishes it from ordinary microservice communication? A: Choreography uses event-driven architecture, but it is more. In choreography, the entire distributed transaction — an "all or none" guarantee — must hold across multiple independent services, each with its own database. Ordinary event-driven communication does not by itself ensure that if one step fails, all prior steps are rolled back in a coordinated compensating sequence. Choreography adds the compensating-transaction semantics on top of the event-driven substrate. The architect must design every participant as both a publisher and a subscriber, with both success and failure event handlers, so that the distributed transaction as a whole either succeeds completely or compensates completely.

Q: In choreography, how does a service know which other services to roll back? A: It does not need to know the entire chain. Each service communicates only with its immediate neighbors through events. If Service C fails, it emits a failure event consumed by Service B. Service B handles that event (compensating itself), then emits its own failure event consumed by Service A. Service A compensates itself, and the chain completes. The architect is responsible for wiring these publisher-subscriber relationships so that the cascade of compensating events flows correctly in reverse.

Q: Can we use a combination of orchestration and choreography? A: Yes. A common hybrid uses an orchestrator to manage the overall sequence but lets services communicate with each other through events for individual steps. The orchestrator retains control while event-driven communication decouples the participants. This is often the pragmatic choice in real production systems.

Q: In the orchestration pattern, the orchestrator appears to have no database. What if it fails mid-sequence? A: The orchestrator must persist state — a JSON file (as in the demo), a database, or a durable workflow engine like AWS Step Functions. It needs to record: (1) the status of every transaction (succeeded/failed), and (2) what compensating action to take if a participant fails. Without persistent state, a crash would lose the entire transaction's progress. In production, this state is stored durably so the orchestrator can recover and resume or compensate on restart.

Orchestration centralizes coordination in a single orchestrator; choreography distributes it across event-driven services. Orchestration is easier to debug and scales to many participants; choreography avoids a single point of coordination but becomes hard to trace with many participants. Choose based on the number of participants and the need for visibility.

8.2 Practical Demo — Saga Orchestration for Research Paper Processing

Why a concrete demo matters The Saga pattern's formal definition is abstract. Seeing it implemented with real agents processing a real document makes the concepts tangible — especially the compensating-transaction logic, which is the hardest part to grasp from definitions alone.

8.2.1 Demo Overview and Agent Design

The problem Given a research paper (a 6-page PDF with abstract, introduction, background, methodology, conclusion, and references), process it through a pipeline of three AI agents coordinated by an orchestrator. This is a classic multi-step Saga: each agent performs a local transaction, and the orchestrator manages sequencing and compensation.

The three agents:

Validation Agent — reads the paper and classifies it as valid or invalid. A valid paper has all expected sections (abstract, introduction, background, methodology, conclusion, references); an invalid one is missing critical content. The agent uses an LLM (GPT-5) behind the scenes to perform the classification.
Review Agent — if the paper is valid, performs a SWOT-style review: identifies strengths, weaknesses, and suggestions for improvement. Another LLM call analyzes the full paper text.
Summary Agent — summarizes the valid paper in exactly 300 words.

The orchestrator's logic (sequential)

Call the Validation Agent. If the result is invalid, stop immediately — there is nothing to review or summarize. If valid, proceed.
Call the Review Agent. Store its output.
Call the Summary Agent. Store its output.
If all three succeed, mark the Saga as success.

This is a linear pipeline: the orchestrator is hard-coded with an if-else sequence. It does not reason about which agent to call next — the order is deterministic.

State management: A shared saga_state.json file tracks the transaction's status (running to completed / rolled_back), a list of completed agents, and metadata (results from each agent). Each agent also writes its output to a .txt file in an outputs/ folder. The state file is the orchestrator's durable memory — without it, a crash would lose all progress.

The orchestrator class — the central coordinator — contains a run() method that calls agents in sequence:

validation_agent.execute(paper_text) — updates state
review_agent.execute(paper_text) — updates state
summary_agent.execute(paper_text) — updates state
If all succeed — status = "completed", Saga success

Each agent class contains two methods:

execute() — the forward transaction (validate the paper, review strengths/weaknesses/suggestions, summarize in 300 words)
compensate() — the compensating transaction (delete the output .txt file, remove the agent's entry from the shared state)

Worked example: the agent interface pattern

class ReviewAgent:
    def execute(self, paper_text: str) -> str:
        # Forward transaction: call LLM, write review.txt, update saga_state.json
        review = llm.review(paper_text)
        write_file("outputs/review.txt", review)
        update_saga_state("review", metadata={"review": review})
        return review

    def compensate(self) -> None:
        # Compensating transaction: delete review.txt, remove from state
        delete_file("outputs/review.txt")
        remove_agent_state("review")

Every agent follows this same execute/compensate pattern. The orchestrator's compensate() method iterates over completed agents in reverse and calls each one's compensate() method.

The demo implements Saga orchestration with three LLM-powered agents, a shared JSON state file, and deterministic if-else sequencing. Each agent has both execute() and compensate() methods.

8.2.2 Execution Scenarios

Success scenario (paper.pdf): A well-formed 6-page research paper is validated as valid, reviewed (strengths, weaknesses, suggestions written to review.txt), summarized (300-word summary written to summary.txt). The saga_state.json shows status: "completed" with all three agents listed under completed_agents and their metadata included.

Failure scenario 1 — Invalid paper (paper1.pdf): A malformed document — only a title, author, introduction, and methodology; no conclusion, no references. The Validation Agent returns invalid. The orchestrator does not call the Review or Summary agents at all. Status changes to rolled_back. No output files are created.

This is a key design decision: the Saga terminates early. Since the paper is invalid, there is no point running the downstream agents. No compensation is needed because no downstream transactions were committed.

Failure scenario 2 — Summary fails (paper2.pdf): A valid paper that contains the trigger phrase "fail summary" (artificially inserted for demonstration). The Validation Agent succeeds (valid), the Review Agent succeeds (writes review.txt), but the Summary Agent raises an exception. The orchestrator then runs compensation in reverse:

Step	Action	Result
1	`summary_agent.compensate()`	Deletes `summary.txt`
2	`review_agent.compensate()`	Deletes `review.txt`
3	`validation_agent.compensate()`	Deletes `validation.txt`
4	State reset	`saga_state.json` to `status: "rolled_back"`, empty agents and metadata

This illustrates the core principle: even though the review agent had already written useful output, the distributed transaction as a whole is treated as "all or none." In a real payment scenario, this would correspond to cancelling the order and refunding the payment at every prior step.

Q: Why roll back the validation and review when the summary fails? The paper was valid and the review was useful. A: In this demo, the rollback deletes everything to illustrate the compensating mechanism. In a real system, the decision of what to roll back depends on the use case. If the review output is still valuable and has no downstream dependency on the summary, the architect could choose to retain it. The critical point is that every participant must have a defined compensating transaction — even if, in some cases, that transaction is a no-op (an empty compensate() method that does nothing).

Q: The orchestrator in this demo is not itself an agent — it just hard-codes a sequence of calls. Should an orchestrator be an agent? A: It depends. When the sequence of steps is deterministic (validate then review then summarize), hard-coded logic (if-else conditions) suffices and is reliable. When the next step depends on the agent's output in a way that requires reasoning — or when multiple different workflows must be chosen dynamically — the orchestrator itself can be an agent armed with an LLM. However, LLM-based orchestrators introduce the risk of hallucinated step sequences — the LLM might invent a step that does not exist or skip a required step. Most current production examples, including those from AWS and Azure, use deterministic if-else orchestration logic. As autonomy levels increase (toward levels 5–6), agent-based orchestrators may become more common.

Q: Would using an agent for the orchestrator be better than hard-coding flows when there are multiple possible workflows? A: For well-defined, sequential problems — like the paper pipeline — the business logic already specifies the order, so agent intelligence is unnecessary and may even be harmful (hallucinated steps). For complex problems where the next state is not predetermined or depends on open-ended outputs, an agent-based orchestrator that reasons about the state and chooses the next action is valuable. The use case drives the choice.

Pitfall: mixing LLM reasoning with deterministic orchestration If the orchestrator uses an LLM to decide which agent to call next, the LLM may hallucinate steps — for example, calling a "formatting" agent that was never defined, or skipping validation entirely. In production, the safest approach is deterministic if-else orchestration for the main flow, with LLM reasoning reserved only for the within-agent tasks (like the validation or review logic itself).

The orchestration demo shows three concrete scenarios: success, early termination (invalid input), and full compensation after mid-pipeline failure. The compensating-transaction loop runs in reverse over completed agents — this is the pattern to remember for exams and assignments.

8.3 Practical Demo — Saga Choreography for Research Paper Processing

Same problem, different coordination style The paper-processing pipeline from Section 8.2 can be implemented with choreography — no central orchestrator. This section shows exactly how the same three agents communicate through events instead of being called by an orchestrator. Comparing the two implementations side by side is the best way to understand the trade-offs.

8.3.1 Event-Driven Agent Coordination

The same paper-processing pipeline can be implemented with choreography — no central orchestrator. Instead, agents communicate exclusively through events persisted in a shared JSON file (a simplified stand-in for a real message broker like Kafka or RabbitMQ).

Event flow (success path):

The research paper is given to the Validation Agent. It validates the paper and emits a validated event.
The Review Agent, subscribed to validated events, picks it up, performs the review, and emits a reviewed event.
The Summary Agent, subscribed to reviewed events, picks it up, generates the summary, and emits a summarized event.
The system detects that all three events have been emitted and declares Saga success.

Each agent is both a publisher (emitting its own success/failure event) and a subscriber (listening for events from its neighbors). There is no direct API call from one agent to another.

Worked example: event flow (failure path — summary fails):

Validation and Review succeed as before (emitting validated and reviewed).
Summary Agent fails and emits a summary_compensated event.
Review Agent, subscribed to summary_compensated, compensates itself (deletes review output) and emits a review_compensated event.
Validation Agent, subscribed to review_compensated, compensates itself (deletes validation output) and the Saga is rolled back.

The compensation cascade flows automatically through the event subscriptions — no orchestrator needed. Each agent handles its own compensation when it receives the appropriate failure event.

Event subscription map:

Agent	Subscribes to	Emits on success	Emits on failure
Validation	(paper input)	`validated`	`validation_compensated`
Review	`validated`	`reviewed`	`review_compensated`
Summary	`reviewed`	`summarized`	`summary_compensated`

Each agent listens only to the events relevant to it — it does not need to know the full chain. This is the decoupling advantage of choreography.

Q: In choreography with a message broker, does every agent receive every event? A: Every event is stored in the broker, but who reads it is a design decision. A consumer subscribes to specific event types or topics. An agent need not read every event — only those it declares interest in. The architect configures which agents subscribe to which event types. In the demo, the Review Agent subscribes only to validated and summary_compensated — it ignores all other events.

Q: Why must choreography always use an event-driven mechanism? Can one agent not call the next agent directly? A: Choreography is defined by the absence of a central coordinator and the use of event-based communication between decentralized participants. Direct API calls between agents would make them tightly coupled and would reintroduce a kind of implicit orchestration (the caller deciding whom to call next). The event broker decouples them completely — the publisher does not know who consumes its events, and the consumer does not know who published them. If direct calling is what you need, orchestration is the more natural fit.

8.3.2 Message Broker Role and Q&A

In a production implementation, these events would flow through Kafka or RabbitMQ. The message broker acts as a dumb pipe — it stores and delivers events but does not contain business logic. Each agent decides independently whether an event is relevant to it (whether it needs to act or ignore it). The broker can support one-to-one, one-to-many, or many-to-one publisher-subscriber topologies.

Scope: choreography vs. ordinary event-driven architecture Choreography uses event-driven architecture, but it is not the same thing. Ordinary event-driven communication does not guarantee that if one step fails, all prior steps are rolled back. Choreography adds compensating-transaction semantics: every participant must have both a success handler and a failure handler, and the failure events must cascade backward through the subscription chain until all committed transactions are compensated. The architect is responsible for wiring these subscriber relationships correctly.

In choreography, agents communicate exclusively through events — no direct calls, no central coordinator. The message broker is a dumb pipe that stores and delivers events. Compensation cascades automatically through failure-event subscriptions in reverse order.

8.4 Compensating Transaction Implementation — A Code Walkthrough

From concept to code Sections 8.1–8.3 described the Saga pattern and its two styles conceptually. This section translates the concept into concrete code: how does compensate() actually work inside each agent class, and how does the orchestrator coordinate the reverse-order compensation loop?

8.4.1 Agent-Level Compensate Methods

The agent class structure Each agent class in the orchestration demo contains exactly two methods:

execute(paper_text) — the forward transaction. Performs the LLM-powered task (validate, review, or summarize) and writes output to a .txt file. Also updates the shared saga_state.json.
compensate() — the compensating transaction. Deletes the output .txt file and removes the agent's entry from the shared state. This is the logical inverse of execute().

This two-method pattern is the minimal viable interface for a Saga participant.

The remove_agent_state(agent_name) helper:

Reads the completed_agents list from saga_state.json.
Removes the named agent from the list.
Removes the corresponding metadata entry.
Writes the updated state back to the JSON file.

This helper is idempotent: if the agent was already removed (e.g., due to a retry), calling it again does nothing harmful.

The delete_output(filename) helper:

Checks if the file exists in the outputs/ directory.
If it exists, deletes it.
If it does not exist, does nothing (safe to call multiple times).

Per-agent compensation — concrete trace:

class ValidationAgent:
    def compensate(self):
        delete_output("validation.txt")
        remove_agent_state("validation")

class ReviewAgent:
    def compensate(self):
        delete_output("review.txt")
        remove_agent_state("review")

class SummaryAgent:
    def compensate(self):
        delete_output("summary.txt")
        remove_agent_state("summary")

Every agent follows the same two-step pattern: delete output, remove state. The only difference is the filename and agent name. In a real system with a database, these would be SQL DELETE or UPDATE statements instead of file operations.

The orchestrator's compensation loop: The SagaOrchestrator class has a compensate() method that iterates over the completed agents in reverse order and calls each one's compensate() method. This is the generic compensating-transaction loop — applicable to any number of participants:

class SagaOrchestrator:
    def compensate(self):
        # Reverse order: last completed agent compensates first
        for agent_name in reversed(self.completed_agents):
            self.agents[agent_name].compensate()
        self.status = "rolled_back"
        self.completed_agents = []
        self.metadata = {}

The reverse iteration is critical: the last committed transaction must be compensated first, because downstream transactions may depend on upstream data. Compensating in forward order could break those dependencies.

Worked example: compensation trace for the summary-fails scenario Starting state: completed_agents = ["validation", "review", "summary"]

Iteration	Agent	Action	State after
1	`summary` (reversed order)	Delete `summary.txt`, remove from state	`["validation", "review"]`
2	`review`	Delete `review.txt`, remove from state	`["validation"]`
3	`validation`	Delete `validation.txt`, remove from state	`[]`

Final state: status = "rolled_back", empty agents and metadata lists. The system is back to the pre-Saga state.

Pitfall: compensating in forward order If the orchestrator compensated in forward order (validation first, then review, then summary), the system would briefly be in an inconsistent state: the validation file would be deleted but the review file would still exist, even though it was created later. While this may not cause problems in the demo (since each compensation is independent), in real systems where downstream compensation depends on upstream data still being present, forward-order compensation can fail.

In a real system with a database, the compensate() method would execute SQL DELETE, UPDATE, or INSERT to reverse the forward transaction's effects, rather than merely deleting a file. The principle remains the same: every forward operation must have a defined reverse operation.

Q: Is the rollback logic also called "compensating transaction" or just "rollback"? A: The proper term is compensating transaction. "Rollback" is a database term implying a single undo within one ACID transaction. In a distributed Saga, there is no single transaction to roll back — each local transaction must be individually compensated. The term "compensating transaction" captures this distributed, application-level undo semantics. The vocabulary distinction matters: saying "rollback" in a distributed context can mislead people into thinking there is a single atomic undo mechanism, which does not exist.

Q: In a real system with a database, what does the compensating transaction actually do — delete, update, or something else? A: It depends on the forward transaction's semantics. If the forward transaction inserted a row, the compensating transaction deletes it. If the forward transaction updated a column from value A to value B, the compensating transaction sets it back to A. If the forward transaction changed a status flag, the compensating transaction restores the previous status. The architect must design the compensating operation to leave the local data store in the same logical state as before the forward transaction — consistent with any audit-trail requirements.

The compensation loop iterates completed agents in reverse order, calling each agent's compensate() method (delete output + remove state). The reverse order is critical. Every forward operation must have a defined reverse operation, even if that reverse operation is a no-op.

8.5 Saga Pattern in the Agent TKI World

From microservices to AI agents The Saga pattern was born in the microservices world — coordinating REST APIs and databases. This section shows that the pattern transfers directly to multi-agent AI systems. The only change is vocabulary: "services" become "agents," "API calls" become "LLM tasks," but the coordination logic — forward transactions, compensating transactions, orchestrator or choreography — remains identical.

8.5.1 Transferring the Pattern to Multi-Agent Systems

The mapping: microservices to agents

Microservices concept	Agent TKI equivalent
Microservice	AI agent (LLM-powered)
Service's local data store	Agent's output file, database, or API call
REST/gRPC call	Supervisor calling agent.execute()
Compensating transaction	agent.compensate() — delete output, undo API call
Orchestrator	Supervisor agent (or deterministic controller)
Message broker (Kafka/RabbitMQ)	Event bus (for choreography-style agent coordination)

The Saga pattern, originally a microservices design pattern, transfers directly to multi-agent AI systems (Agent TKI). Instead of microservices, the participants are AI agents — each powered by an LLM, each performing a specific task, each potentially writing to its own data store or calling external APIs.

AWS reference architecture for Agent TKI Saga orchestration:

A supervisor agent (the orchestrator) coordinates multiple sub-agents.
Sub-agent A might call an AWS Lambda function (serverless compute).
Sub-agent B might query Amazon Redshift (data warehouse).
Sub-agent C might call an external API.
Each agent may or may not have an associated knowledge base or action group.
The supervisor tracks success/failure and triggers compensating transactions when needed.

The same logic applies: if Sub-agent C fails (e.g., external API returns HTTP 404), the supervisor invokes Sub-agent B's compensating transaction, then Sub-agent A's compensating transaction. The pattern is identical — only the vocabulary changes from "services" to "agents."

Worked example: Agent TKI failure cascade Consider three agents in a Saga:

Data Agent — queries a data warehouse for patient records.
Analysis Agent — runs an ML model on the records.
Report Agent — calls an external API to generate a PDF report.

If the external API returns HTTP 404 (not found), the Report Agent fails. The supervisor invokes:

report_agent.compensate() — deletes the partial PDF.
analysis_agent.compensate() — deletes the analysis output.
data_agent.compensate() — removes the cached patient records.

The Saga is rolled back. The system returns to its pre-Saga state, ready to retry with a corrected API endpoint.

8.5.2 AWS Step Functions and Q&A

AWS provides Step Functions as a managed Saga orchestration service. It allows defining a workflow of Lambda functions (each representing a Saga participant), with automatic retry, error handling, and compensating-transaction chaining. Interested readers can explore the AWS Step Functions console (with a free-tier account) to see sample microservices and Agent TKI workflows that create Lambda functions, DynamoDB tables, and other resources behind the scenes.

Real-world: AWS Step Functions The AWS Step Functions service is purpose-built to implement Saga orchestration. It handles the complexity of tracking state, invoking services, and running compensating transactions automatically. This is the production-grade way to run Saga orchestration in the cloud. Each Lambda function in the workflow acts as a Saga participant, and Step Functions provides built-in error handling and retry logic.

Q: In the Agent TKI context, can agents execute in parallel? A: For a single distributed transaction, the sequence is typically sequential — the supervisor calls Sub-agent A, waits for its result, then calls Sub-agent B, and so on. Parallelism exists at the system level: thousands of end users may each be running independent distributed transactions simultaneously. Within one transaction, the steps are ordered because each step often depends on the previous step's output. If two steps are truly independent (neither consumes the other's output), they can be parallelized, but the Saga pattern's compensating-transaction semantics become more complex with parallel branches — the orchestrator must track which branches completed and which did not, and compensate accordingly.

The Saga pattern transfers directly from microservices to multi-agent AI systems. The vocabulary changes (services to agents), but the coordination logic is identical. AWS Step Functions provides managed Saga orchestration for production Agent TKI workflows.

8.6 Blackboard Pattern — Brief Overview

8.6.1 Concept and Contrast with Saga

Three patterns in the syllabus The Blackboard pattern is the third architectural pattern mentioned in the syllabus, alongside Saga orchestration and Saga choreography. It was not covered in detail during this session but is noted here for completeness. It will be covered in a future class.

The Blackboard pattern In the Blackboard pattern, multiple specialized agents (or knowledge sources) collaborate by reading from and writing to a shared data structure — the blackboard. A controller monitors the blackboard and decides which agent to invoke next based on the current state. Each agent contributes partial solutions or knowledge to the shared workspace, and the controller coordinates the overall problem-solving process.

The key idea: agents do not call each other directly. They communicate indirectly through the shared blackboard, and the controller decides who acts next based on what is currently on the board.

Analogy: a team working on a whiteboard Imagine a team of specialists gathered around a whiteboard. A data scientist writes their analysis results. A domain expert reads them and adds a domain interpretation. A visualization specialist reads both and creates a chart. No one directly tells the other what to do — they write on the board, and a project manager (the controller) decides who should act next based on what is on the board. This is the Blackboard pattern.

Contrast with Saga:

Dimension	Saga	Blackboard
Primary goal	Transactional integrity (all-or-nothing)	Collaborative problem-solving
Coordination	Sequential (forward transactions + compensating transactions)	State-driven (controller reads board, decides next agent)
Data flow	Each agent passes output to the next	All agents read/write a shared workspace
Failure handling	Compensating transactions undo committed work	Agents may overwrite or refine prior contributions
Use case	Distributed transactions (ordering, payment)	AI planning, speech recognition, multi-model reasoning

Exam note: The Blackboard pattern will not appear on the midterm exam (neither regular nor makeup) for this section. Students may review it independently; it will be covered in the next class session.

The Blackboard pattern uses a shared workspace and a controller for collaborative problem-solving, unlike Saga which uses sequential transactions for transactional integrity. Not on the midterm exam.

Exam Guidance Summary

The midterm exam covers lectures 1 through 7. The Saga pattern (orchestration and choreography) is in scope. The Blackboard pattern is not on the exam.

Question types (30 marks, flat — no scaling, no optional questions):

Agree/Disagree with justification — A statement is given. You must state whether you agree or disagree and provide a proper justification. This tests conceptual understanding, not rote memorization. The justification matters more than the binary choice.
Differentiate between X and Y — Compare and contrast two related concepts (e.g., orchestration vs. choreography). Write from your own understanding — reproducing PPT bullet points verbatim is not expected or rewarded. Explain the distinction in your own words. A table comparing dimensions is an effective format.
Scenario-based questions — A specific scenario is described (possibly with a diagram). You must apply concepts to that scenario. If the question says "take a data science experiment of your choice" (e.g., diabetes prediction), you must stay within that scenario throughout your answer. Do not drift into generic definitions. Instead, explain how the concept applies to the specific scenario. You may make reasonable assumptions, but all subsequent reasoning must be consistent with those assumptions.

What is NOT on the exam:

No coding questions. Code is only required in the assignment.
No Blackboard pattern questions.
No heartbeat pattern or feature-store pattern (these were in the original course handout but are not covered in lectures 1–7).

What IS in scope:

M1: Foundations of ML Systems Engineering
M2: Requirements Engineering for ML Systems
M3: Architecture and Design — including microservices patterns, architectural patterns, design patterns, and the Saga pattern (orchestration and choreography) as applied to Agent TKI
Agent TKI concepts (LLM as the agent core, memory components) covered in earlier sessions

Study reference: The PPTs from sessions 1 through 7 are the definitive study material. The course handout modules M1–M3 align with the covered content, but a few items (heartbeat pattern, feature-store pattern) listed in the handout are not covered and will not be asked. The Saga and Blackboard patterns were promoted from their original slot (sessions 15–16) to sessions 7–8 because they fit naturally within the architecture and design module.

General advice:

The exam is closed-book, but approach it with an open mind — questions test understanding and application, not memorization.
Write concisely. Do not add irrelevant general definitions to scenario-based answers. Stay within the scenario.
For agree/disagree questions, the justification matters more than the binary choice.
Expect roughly 5–6 questions combining the three types above. The exam is designed to be completed in about 1 hour 30 minutes to 1 hour 40 minutes.

Exam note: The Saga pattern is very important — it is one of the most significant architectural patterns for distributed systems and agent coordination. Expect at least one question on it, likely scenario-based or as a differentiation question (orchestration vs. choreography).

Assignment note: Assignment 1 deadline will be extended by one week due to midterms. Assignment 2 will have a strict, non-extendable deadline. For the assignment, implementing the Saga pattern (either orchestration or choreography) is strongly encouraged — it is the best way to internalize the concepts. Code must be submitted as a PDF containing the implementation and screenshots of the output.

Key Industry Applications

AWS Step Functions — purpose-built Saga orchestration service; provides visual workflow definition, automatic retry, error handling, and compensating transaction chaining for microservices and Agent TKI workflows
Amazon Bedrock — managed service for building and deploying AI agents; integrates with Step Functions for Saga-based agent coordination
AWS Lambda — serverless functions used as Saga participants in AWS examples; each Lambda can represent one agent's forward and compensating logic
Amazon DynamoDB — NoSQL database used as the per-service data store in AWS Saga examples; flexible schema suits agent-generated data
Apache Kafka / RabbitMQ — message brokers used in choreography-based Saga implementations; decouple publishers from subscribers via durable event logs
Swiggy / Zomato — food-delivery platforms whose order-to-delivery flow is a classic distributed transaction; Saga patterns underpin their microservices coordination
Ride-booking platforms — another canonical distributed-transaction domain; booking then driver assignment then payment then confirmation flows are Saga transactions
Agent TKI (Agentic AI frameworks) — multi-agent coordination that reuses Saga semantics; a supervisor agent orchestrates sub-agents, each with its own forward and compensating logic
NoSQL databases — the data store of choice for autonomous agent systems (autonomy levels 5–6) because their flexible schemas can accommodate unpredictable agent outputs
Pydantic models — at lower autonomy levels (3–4), agents can output bounded, structured JSON constrained by Pydantic schemas, allowing SQL databases as viable data stores

SEML Lecture 8 notes · Saga Pattern and Architectures for Agentic AI

Software Engineering for Machine Learning· postgraduate· 2026-07-26

Sections Breakdown

1The Saga Pattern — Distributed Transactions in Microservices

Introduction to the Saga pattern as a solution for distributed transaction problems in microservices, covering formal definition, compensating transactions, and orchestration vs. choreography.

2Practical Demo — Saga Orchestration for Research Paper Processing

Three-agent pipeline demo with deterministic orchestrator, success, early termination, and compensation scenarios.

3Practical Demo — Saga Choreography for Research Paper Processing

Event-driven agent coordination with compensation cascades through failure-event subscriptions.

4Compensating Transaction Implementation — A Code Walkthrough

Agent-level execute and compensate methods and orchestrator compensation loop code walkthrough.

5Saga Pattern in the Agent TKI World

Direct mapping from microservices to multi-agent AI systems and AWS Step Functions integration.

6Blackboard Pattern — Brief Overview

Shared workspace and controller pattern for collaborative problem-solving, contrasted with Saga.

7Exam Guidance Summary

Midterm exam scope, question types, and study advice for lectures 1 through 7.

8Key Industry Applications

Real-world implementations: AWS Step Functions, Lambda, DynamoDB, Kafka, food-delivery platforms.

Postgraduate students in Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Saga Pattern and Distributed Transactions

Must-know: Saga = sequence of local transactions with compensating transactions for undo. Orchestration centralizes control; choreography distributes it via events. Orchestration scales to many participants; choreography is simpler for 3-4 participants.

Top pitfall: Assuming a compensating transaction is always a simple inverse (e.g., unsending an SMS is impossible). Also: confusing choreography with ordinary event-driven architecture — choreography adds compensating-transaction semantics.

Self-check: In a 4-step Saga where step 3 fails, which compensating transactions run and in what order?

Connects to: Sections 8.2, 8.3, 8.4, 8.5

Saga Orchestration Demo

Must-know: Orchestration demo: Validation then Review then Summarize. Orchestrator runs compensation in reverse order when summary fails. Deterministic if-else orchestration is preferred over LLM-based orchestration for reliability.

Top pitfall: Using an LLM-based orchestrator for deterministic flows — risk of hallucinated step sequences. Every agent must have a defined compensate() method, even if it is a no-op.

Self-check: In the paper-processing demo, what happens when the Summary Agent fails? List the compensation sequence.

Connects to: Sections 8.1, 8.3, 8.4

Saga Choreography Demo

Must-know: Choreography: agents are both publishers and subscribers. No direct API calls. Compensation cascades through failure events (summary_compensated then review_compensated then validation_compensated). Message broker is a dumb pipe — no business logic.

Top pitfall: Confusing choreography with ordinary event-driven communication — choreography adds compensating-transaction semantics on top.

Self-check: In choreography, how does the Review Agent know to compensate when the Summary Agent fails?

Connects to: Sections 8.1, 8.2, 8.5

Compensating Transaction Implementation

Must-know: execute() = forward transaction (write output + update state). compensate() = inverse (delete output + remove state). Orchestrator iterates in reverse order. Compensating transaction is not rollback — rollback implies single ACID undo.

Top pitfall: Calling it rollback instead of compensating transaction. Compensating in forward order instead of reverse.

Self-check: Why does the orchestrator compensate in reverse order rather than forward order?

Connects to: Sections 8.1, 8.2

Saga Pattern in Agent TKI

Must-know: Saga pattern maps 1:1 from microservices to Agent TKI. Supervisor agent = orchestrator. AWS Step Functions = managed Saga orchestration service. Saga is one of two pillars of distributed system design (alongside CQRS).

Top pitfall: Thinking the pattern is different for AI agents — it is identical, only vocabulary changes.

Self-check: In the Agent TKI context, what is the equivalent of a microservice's local data store?

Connects to: Sections 8.1, 8.2, 8.3

Blackboard Pattern

Must-know: Blackboard = shared workspace + controller for collaborative problem-solving. NOT on midterm exam.

Top pitfall: Confusing Blackboard with Saga — Saga is for transactional integrity, Blackboard is for collaborative problem-solving.

Self-check: What is the role of the controller in the Blackboard pattern?

Connects to: Section 8.1

Exam Guidance

Must-know: 30 marks flat. Saga pattern is very important — expect at least one question. No Blackboard, no coding. Questions test understanding, not memorization.

Top pitfall: Reproducing PPT bullet points verbatim in differentiation questions. Drifting into generic definitions in scenario-based questions.

Self-check: What three question types appear on the midterm?

Connects to:

Key Industry Applications

Must-know: AWS Step Functions = managed Saga orchestration. Kafka/RabbitMQ = message brokers for choreography. Food-delivery platforms = canonical Saga use case.

Top pitfall: None specific — this section is for domain awareness, not exam testing.

Self-check: Name two AWS services used in Saga orchestration for Agent TKI.

Connects to: Sections 8.1, 8.5

Previous LectureLecture 07 · Design Patterns for ML Systems: Registry, Serving, and Agentic Orchestration

Next LectureLecture 09 · Coding Practices and Code Performance Analysis for ML

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Saga Pattern and Architectures for Agentic AI

8.1 The Saga Pattern — Distributed Transactions in Microservices

8.1.1 Compensating Transactions — Rolling Back in a Distributed World

8.1.2 Two Implementation Styles — Orchestration vs. Choreography

8.2 Practical Demo — Saga Orchestration for Research Paper Processing

8.2.1 Demo Overview and Agent Design

8.2.2 Execution Scenarios

8.3 Practical Demo — Saga Choreography for Research Paper Processing

8.3.1 Event-Driven Agent Coordination

8.3.2 Message Broker Role and Q&A

8.4 Compensating Transaction Implementation — A Code Walkthrough

8.4.1 Agent-Level Compensate Methods

8.5 Saga Pattern in the Agent TKI World

8.5.1 Transferring the Pattern to Multi-Agent Systems

8.5.2 AWS Step Functions and Q&A

8.6 Blackboard Pattern — Brief Overview

8.6.1 Concept and Contrast with Saga

Exam Guidance Summary

Key Industry Applications

SEML Lecture 8 notes · Saga Pattern and Architectures for Agentic AI

Sections Breakdown

Exam Revision Notes

Saga Pattern and Distributed Transactions

Saga Orchestration Demo

Saga Choreography Demo

Compensating Transaction Implementation

Saga Pattern in Agent TKI

Blackboard Pattern

Exam Guidance

Key Industry Applications

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat