Monolith, Microservices, Event-Driven Architecture, and Model Registry

6.1 RAJ Architecture with Multimodal Input

6.1.1 Multimodal Input Strategies

In the previous session, the professor introduced the RAJ architecture: load documents, split them into chunks, embed those chunks, store them in a vector database, retrieve the most relevant chunks at query time, and let an LLM generate a grounded answer. That pipeline works cleanly when the input is plain text — PDFs, Word files, CSVs. But real-world documents are rarely pure text. Lecture slide decks mix bullet points with diagrams. Medical reports embed X-rays alongside narrative descriptions. Research papers scatter figures throughout their body. The question becomes: how does the RAJ pattern adapt when the input is an image or a document that contains images?

The professor outlined two concrete strategies, and the key insight is that both strategies converge to the same downstream pipeline — only the input pre-processing stage changes.

Approach 1 — OCR then text pipeline

OCR (optical character recognition) is a technology that reads text from images — think of it as a scanner that converts a photograph of a page into editable text. Tools like Tesseract (open-source, maintained by Google) or cloud services like AWS Textract perform this extraction step.

The workflow is straightforward:

1. Extract text from each image using OCR.
2. Feed the extracted text into the standard RAJ pipeline: chunk, embed, store in ChromaDB, retrieve at query time.

This approach treats images as a text-extraction problem. Once the text is recovered, the rest of the RAJ architecture does not need to know the original source was an image. It is the simpler approach, and it works well when the images primarily contain text (typed documents, printed slides, handwritten notes with clear handwriting).

Approach 2 — Multimodal LLM directly

Modern LLMs are multimodal — they can process both text and images natively. Models like GPT-4o, Gemini, and Claude can look at an image and describe its content in natural language. Instead of running OCR, you pass the image directly to a multimodal LLM and ask it to produce a textual summary (typically 100–200 words).

The workflow becomes:

1. Send the image to a multimodal LLM with a prompt like "Describe the content of this image in 100–200 words."
2. Receive a textual summary that captures not just any text in the image, but also the visual content — shapes, relationships, labels, layouts.
3. Feed that summary into the standard RAJ pipeline: chunk, embed, store, retrieve.

The LLM understands the image content and produces a concise text representation. This summary then enters the vector store just like any other text chunk.

Why both approaches are viable today

Multimodal LLMs have matured significantly. A few years ago, only specialized vision models could interpret images, and they required fine-tuning. Today, general-purpose LLMs handle images out of the box with no additional setup. This makes Approach 2 practical for most use cases.

The critical architectural insight — the one the professor emphasized — is that the RAJ pattern itself does not change. Both approaches ultimately produce text that gets chunked, embedded, and stored in the vector database. The retrieval and generation stages are identical regardless of whether the input started as plain text, OCR-extracted text, or an LLM-generated image summary. This is the power of the RAJ design: the core pipeline is stable, and only the ingestion adapter changes.

Practical considerations when choosing between the two

In practice, many production systems use a hybrid approach: run OCR first for text-heavy pages, and fall back to the multimodal LLM for pages where OCR returns sparse or low-confidence results. This balances cost against information capture.

Real-world connection. Companies processing large document corpora — insurance claims with attached photos, legal contracts with embedded diagrams, academic papers with figures — increasingly use multimodal LLMs as their primary ingestion strategy. The cost premium over OCR is offset by the significantly richer information capture, especially when downstream accuracy matters.

6.2 Implementing RAJ with CQRS and Pipe-and-Filter

6.2.1 CQRS Write and Read Pipelines

The professor modeled the RAJ architecture using two established software design patterns simultaneously: the pipe-and-filter pattern and CQRS (Command Query Responsibility Segregation). Understanding why both apply — and how they complement each other — is the key architectural lesson of this section.

CQRS (Command Query Responsibility Segregation) is a design pattern that separates the write side (commands that change state) from the read side (queries that retrieve state) into distinct pipelines, models, and often separate code files. The word "segregate" means to set apart — CQRS literally means "separate the responsibility of writing from the responsibility of reading."

The textbook (T1 Ch08) identifies this as a core architectural challenge: "Decomposing the system and deciding how to divide the work is a key step." CQRS provides a principled way to decompose RAJ along the command/query boundary.

CQRS in RAJ — write pipeline vs query pipeline

On the write side (the ingestion pipeline), data flows through a sequence of filters connected by pipes — this is the pipe-and-filter pattern:

1. Documents are loaded from source files.
2. Documents are split into chunks.
3. Chunks are embedded into vectors.
4. Vectors are persisted into a vector store.

On the read side (the query pipeline), a different sequence operates:

1. The user enters a prompt.
2. The prompt is embedded using the same embedding model.
3. ChromaDB retrieves the most similar chunks.
4. The retrieved chunks and the user's query are sent to an LLM which generates a grounded response.

These two pipelines are implemented as separate files: ingest.py for writes and app.py (with a read module) for queries. They share only the vector store — the read side never modifies it, and the write side never queries it.

The write model (ingest.py) performs three sequential activities:

Each step is a filter in the pipe-and-filter pattern — it takes input, transforms it, and passes the result to the next step.

Step 1 — Document loading. Load PDFs (or PPTs, Word files) using appropriate document loaders. LangChain provides loaders for each format. The loader reads the file and produces a list of document objects — each object contains the text content plus metadata (source file name, page number).

Step 2 — Document splitting. Use a text splitter — the recursive character text splitter is the standard choice — to break documents into chunks. The demo uses a chunk size of 1,000 characters with a chunk overlap of 200 characters. Both parameters are configurable and represent a trade-off:

• Larger chunks (e.g., 2,000 characters) capture more context per chunk but reduce retrieval precision — the system might retrieve a chunk where only part is relevant.
• Smaller chunks (e.g., 500 characters) improve precision but may lose context that spans chunk boundaries.
• Overlap mitigates the boundary problem: the 200-character overlap ensures that information near a chunk boundary appears in both adjacent chunks, so it is not lost.

Step 3 — Building the vector store. Each chunk is embedded using OpenAI's text-embedding-3-small model and persisted in ChromaDB. The chunk becomes a document object, an embedding vector is computed (a list of numbers representing the semantic meaning of the text), and both are stored together. ChromaDB indexes these vectors for fast similarity search.

The read model (query pipeline)

The application (app.py) provides a user interface — Streamlit in the demo — where the user enters a prompt. The query pipeline then executes:

1. The user's prompt is embedded using the same text-embedding-3-small model (critical: the embedding model must be identical on both sides, or the vectors will live in different semantic spaces and similarity search will fail).
2. ChromaDB retrieves the most similar chunks using cosine similarity (or another distance metric).
3. Those chunks, along with the user's query, are sent to an LLM — GPT-4o mini via the OpenAI API — which produces a grounded, well-formatted answer.

The LLM's role is specifically to handle the language aspect: taking the raw retrieved data and reframing it in a natural, coherent response that matches the phrasing of the user's prompt. The LLM handles vocabulary, grammar, coherence, and summarization — everything regarding how the answer is expressed. The factual substance comes from the retrieved documents, not from the LLM's parametric knowledge.

Real-world connection. The CQRS pattern is used extensively in production ML systems. Netflix's recommendation pipeline, for example, separates the batch job that computes and stores user embeddings (write side) from the real-time service that retrieves recommendations (read side). The same pattern applies to any system where data ingestion and data querying have different performance, scaling, or reliability requirements.

6.3 RAJ Demo: Document Ingestion and Query Pipeline

6.3.1 Live Demo Walkthrough

The live demonstration processes five session PPTs (converted to PDF) with the following statistics:

The fact that 187 pages produce 183 chunks — roughly one chunk per page — tells us something about the average content density of the slides. Lecture slides tend to have short bullet points, so many pages produce fewer than 1,000 characters and are not split at all. Dense text pages (like reference material) would produce multiple chunks per page.

Ingestion phase (the write pipeline)

The ingest.py script executes the three-step CQRS write pipeline described in the previous section:

1. Load: Each PDF is read page by page using LangChain's PDF loader. The loader produces one document object per page with metadata including the source file name and page number.
2. Split: The recursive character text splitter breaks each page's content into chunks of 1,000 characters with 200-character overlap.
3. Embed and store: Each chunk is embedded using text-embedding-3-small (1,536-dimensional vectors) and persisted in ChromaDB.

Once ingestion is complete, the vector store exists on disk (or in memory). The ingestion script can be shut down — it has done its job. The Streamlit app loads the existing vector store without re-ingesting, which is the CQRS principle in action: the write side and read side are separate processes.

Boundary behavior — out-of-context queries

The professor tested a deliberately out-of-scope question to demonstrate how RAJ handles queries that fall outside the ingested documents.

Configuration and flexibility

The demo uses:

• GPT-4o mini for generation (the LLM that formulates answers).
• text-embedding-3-small for embeddings (the model that converts text to vectors).
• Both accessed via OpenAI API key — no models are downloaded locally.

Students are free to experiment with alternatives:

• Local models: Mistral 7B, Gemma 2B (which works well for small document sets when run locally).
• API-based models: Gemini, Claude, or any other LLM via API.
• Local embedding models: Sentence Transformers, BGE embeddings, or other open-source options.

The architectural pattern remains the same regardless of which specific models you use — only the API endpoints and model names change.

Real-world connection. This demo is a miniature version of what companies like Notion, Slack, and Google build for their internal knowledge-base search features. The pattern — ingest documents, embed, retrieve, generate with citations — is the standard architecture for enterprise search and question-answering systems. The key difference in production is scale (millions of documents instead of 187 pages) and operational concerns (monitoring, access control, incremental updates).

6.4 RAJ Student Q&A

6.4.1 Question and Answer Exchanges

The professor's demo triggered several student questions that surfaced important architectural concepts. These Q&A exchanges are grouped by confusion point — each addresses a distinct concern about how RAJ works in practice.

This question reveals a common assumption: that RAJ is stateless. In the basic demo, each query is treated independently. But production conversational RAJ systems do maintain history — typically by storing previous exchanges in the same vector store (ChromaDB) and retrieving them alongside document chunks. The system can then answer "What did we just discuss?" by retrieving the conversation history chunks.

This is the adapter pattern in action. LangChain provides a library of document loaders — one for each format. The loader's job is to convert any input format into a uniform list of document objects. Once that conversion happens, the rest of the pipeline (split, embed, store) does not know or care what the original format was.

This question and answer introduce a critical architectural evolution. Standard RAJ is a fixed pipeline — data flows in one direction. Agentic RAJ adds a decision-making layer that can dynamically adjust the pipeline's behavior based on the query. This is a step toward more autonomous AI systems.

Real-world connection. The distinction between retrieval-grounded generation and free-form generation is central to enterprise AI adoption. Companies using RAJ for internal knowledge bases need assurance that the system will not fabricate answers about company policies, financial data, or legal requirements. The "retrieve first, generate second" architecture provides this assurance by design.

6.5 Monolithic Architecture — Software Engineering Foundations

6.5.1 Definition and Food-to-Go Example

Monolithic architecture means the application is composed as one single piece. The user interface, business logic, data access layer, and any service layer are tightly wrapped into one codebase, deployed as one unit, and connected to a single shared database. The word "monolith" comes from Greek: mono (one) + lithos (stone) — a single, uncarved block.

The textbook (T1 Ch08) defines this precisely: "The system is composed of a single unit where internals are interwoven rather than separated. Internally there might be modules and libraries, but they are usually not intentionally arranged as services or in layers." The key word is interwoven — the modules exist, but they are not isolated from each other.

The Food-to-Go (Swiggy/Zomato) example. The professor used a food delivery application to make monolithic architecture concrete. The application involves three actors: the customer (consumer), the delivery agent (courier), and the restaurant. The system is organized into modules:

• Restaurant management
• Order management
• Delivery management
• Payment
• Notification
• Billing

All modules reside in the same codebase, written in the same language (Java in the example), and share one MySQL database. External cloud services (Twilio for messaging/SMS, AWS SES for email, Stripe or Razorpay for payment) are integrated through adapters, but the core application remains a single block.

Advantages of monolithic architecture:

• Shared memory and speed. The entire application runs as one process (e.g., on a single EC2 instance), so components call each other in-process, making inter-module communication fast. There is no network hop between the order module and the payment module — they share the same memory space.
• Straightforward end-to-end testing. A tester can trace one complete flow — login, create order, restaurant accepts, payment processed, delivery completed, notifications received — all within one deployable unit. Testing is easier because everything is co-located and there are no distributed system failure modes to worry about.
• Simple deployment. One artifact to build, one artifact to deploy. No orchestration, no service discovery, no load balancing between services.

Disadvantages:

• Technology lock-in. The entire application is written in one language stack (e.g., Java). Every developer must work in that stack. You cannot use Python for the ML recommendation engine and Java for the payment service — it is all one codebase.
• Scalability is the biggest problem. When load increases, the entire application must be scaled as one unit — you cannot scale only the payment module or only the order module independently. If Black Friday drives 100× more payment traffic but only 2× more restaurant browsing traffic, you still have to scale the entire application 100×. This wastes resources on the modules that did not need scaling.

• Difficult to understand and maintain. A large codebase accumulated over years becomes impenetrable for newcomers. The professor cited a real-world example: a legacy VB (Visual Basic) application with 660 database tables, where even a small change risked breaking unrelated functionality. When modules are interwoven, changing one part can have unpredictable ripple effects across the entire system.

The textbook (T1 Ch08) reinforces this with the Twitter case study: "Twitter was originally designed as a monolithic database-backed web application, written in Ruby on Rails by three friends. Once Twitter became popular, it became slow and hard to scale. Developers introduced caches throughout the application and bought many machines to keep up with the load, but they could not handle spikes in traffic." Twitter eventually had to completely redesign the system — a costly and rare undertaking.

The fundamental tension. Monoliths are simple to start but painful to scale, both technically and organizationally. The textbook frames this as an architectural trade-off: "When Twitter was first started, scalability was likely less important than releasing a prototype quickly to gain venture funding and users, so the monolithic Ruby application may have been appropriate at a time, just not future proof given how difficult to change architectural decisions are later."

Real-world connection. Most successful tech companies — Amazon, Netflix, Twitter, Uber — started as monoliths and migrated to microservices as they grew. The monolith-to-microservices migration is one of the most common architectural transitions in the industry. The key lesson is not "avoid monoliths" but "know when you have outgrown one."

6.6 Monolithic ML Pattern and Iris Demo

6.6.1 Monolithic ML Pattern

In the monolithic ML pattern, all ML components — preprocessing, model loading, prediction logic, and serving (UI or API) — are tightly coupled and deployed as a single unit. The model is a library embedded in the application, not an independent service.

The textbook (T1 Ch08) identifies this as one of the common system structures: "Machine-learning components may be interwoven in such systems, often using libraries. System development is initially simple and local without the need for networked communication and the complexities of distributed systems."

Advantages:

• Simple to develop and deploy. Everything is in one place. A data scientist can train a model, serialize it (save it to a file), load it in a FastAPI app, and ship it — all in a single codebase.
• Low operational overhead. One process to monitor, one container to deploy, one log stream to watch. No service discovery, no load balancing, no distributed tracing.
• Complete control over the project. Every component is accessible and modifiable in the same repository.
• Easier end-to-end testing. You can test the full flow — send input, get prediction — with a single test script against a single endpoint.
• Appropriate for: prototypes, small projects, hackathons, internal tools, or initial exploration where the application will not grow significantly.

Disadvantages:

• Cannot scale independently. If prediction load surges (100× more requests at peak hours), the entire application must scale as a monolith — you cannot scale only the inference service while keeping the preprocessing or UI unchanged. This is the same scalability problem as the general monolith, but applied to ML workloads.
• Model updates require full redeployment. Even a minor model version change (e.g., version 5.1 to 5.2 with slightly improved accuracy) forces redeployment of the complete application — the API, the UI, the preprocessing logic, everything. In a microservices architecture, you would redeploy only the model service.
• Technology lock-in. In practice, most ML applications use Python, so language diversity is less of an issue than in general software. But flexibility to adopt new technologies within components is still restricted — you cannot easily swap the serving framework from FastAPI to something else without touching the entire codebase.
• Single point of failure. Any error anywhere in the application — a bug in preprocessing, a null pointer in the API handler, a model loading failure — can bring the entire system down. There is no isolation between components.

Iris classification demo — the monolithic ML pattern in code

The professor demonstrated the monolithic ML pattern using the classic Iris dataset — four input features (sepal length, sepal width, petal length, petal width) predicting three flower classes: setosa (0), versicolor (1), and virginica (2).

A pre-trained logistic regression model is serialized as a pickle file (logistic_regression_model.pkl). Pickle is Python's built-in serialization format — it converts a Python object (like a trained model) into a byte stream that can be saved to disk and loaded later. The application, built with FastAPI and served via Uvicorn, does the following:

1. Loads the pickle model at startup.
2. Exposes a single POST /predict endpoint.
3. Accepts a JSON list of four feature values.
4. Calls model.predict(features) and returns the predicted class (0, 1, or 2).

{"features": [5.1, 3.5, 1.4, 0.2]}

The textbook (T2 Ch11) explains this pattern: "If this becomes a feature in your company's product, your model may need to receive input data and return predictions many times a second. You can build an API that waits for that data and returns the prediction." FastAPI with Uvicorn is the de facto standard for this in Python — it handles JSON serialization, input validation (via Pydantic), and auto-generates interactive documentation.

Everything — model loading, prediction logic, input handling, output formatting — lives in a single code file and a single process. This is monolithic in the purest sense: one unit for model serving, with no separation between model management and inference.

Real-world connection. FastAPI + Uvicorn is used throughout the industry for rapid ML model prototyping. Companies like Netflix, Uber, and Microsoft use this pattern for internal tools and early-stage products before migrating to more distributed architectures as the product matures.

6.7 Microservices Architecture — Foundations and Industry Impact

6.7.1 Principles and Industry Adoption

Microservices architecture decomposes an application into a collection of independently deployable services, each following the Single Responsibility Principle (SRP). The principle, from Robert C. Martin, states: gather together things that change for the same reason; separate things that change for different reasons. In microservices terms: if two components scale differently, deploy differently, or are maintained by different teams, they should be separate services.

The textbook (T1 Ch08) defines this precisely: "A system is organized into multiple self-contained services (processes) that call other services through remote procedure calls. The services are not necessarily organized into layers, and typically each service represents a cohesive piece of functionality and is responsible for its own data storage. This design allows independent deployment, versioning, and scaling of services and flexible routing of requests at the network level."

Revisiting the Food-to-Go example under microservices. The same three actors — customer, courier, restaurant — interact not with a single monolithic block but through an API gateway. The gateway is a single entry point that routes each request to the appropriate microservice:

Each service owns its own database — this is a critical design rule. Services do not share databases, because shared databases create hidden coupling (changing the schema in one service breaks queries in another). Each service communicates with external providers through its own adapters.

One service can invoke another service:

• Synchronously via REST or gRPC (the caller waits for the response).
• Asynchronously through message queues like Kafka (the caller sends a message and continues without waiting).

The API gateway handles routing and aggregation, but each service is developed, tested, deployed, and scaled independently.

Industry-scale evidence of microservices benefits:

The textbook (T1 Ch08) uses Twitter's migration as a case study: after redesigning from a monolith to microservices, "the new system was more complex (inherent in distributed systems) and more costly to develop, but this was deemed a necessary trade-off for achieving the primary four quality goals" — latency, reliability, maintainability, and modifiability.

Key advantages:

• Polyglot development. Each microservice can use the most appropriate language and framework. Instagram, for example, is built with Python, React Native, and JavaScript. One service might use Java with Spring Boot (robust, enterprise-grade), another Python (rapid prototyping, ML integration), and another C (performance-critical path).
• Polyglot persistence. Each microservice can choose its own database type — relational (MySQL, PostgreSQL), NoSQL (MongoDB, Cassandra), or in-memory (Redis) — based on its specific data access patterns. The order service needs ACID transactions (relational); the notification service needs fast key-value lookups (Redis).
• Independent scalability. This is the most important advantage. The professor's example: if the recommendation service on YouTube goes down, users can still play videos, write comments, and download content because those services are independent. Similarly, if video streaming traffic spikes 100× during a live event, only the streaming service scales — the comment and recommendation services remain at their current capacity.
• Faster development velocity. Teams can work on different services in parallel without stepping on each other's code. No merge conflicts, no "who broke the build" blame games, no waiting for the other team to finish their feature before you can deploy yours.
• High reliability. The failure of one service does not cascade into a total system outage (provided dependencies are well-managed). This is called fault isolation — a bug in the notification service does not prevent users from placing orders.

Synchronous communication in microservices. Services typically communicate synchronously through:

• REST APIs — the most common, using HTTP with JSON. Simple, well-understood, universally supported.
• gRPC — using Protobuf as the binary data-exchange format over HTTP/2. Faster and more resource-efficient than REST, but requires both sides to implement the same Protobuf contract.
• GraphQL — enables a single request returning data from multiple backend services. Popular in frontend-heavy applications.

Asynchronous communication uses message brokers like Kafka, RabbitMQ, or cloud-managed services (Amazon SQS, Azure Service Bus, Google Pub/Sub). Section 6.10 covers event-driven architecture in detail.

Real-world connection. The microservices pattern is the dominant architecture for large-scale web applications. Netflix (hundreds of microservices), Uber (thousands of microservices), and Amazon (tens of thousands of microservices) all use this pattern. The trade-off is operational complexity: microservices require sophisticated tooling for service discovery, load balancing, distributed tracing, and failure handling — tooling that monoliths do not need.

6.8 Synchronous Communication: REST, GraphQL, and gRPC

6.8.1 Comparing API Protocols

In a microservices architecture, services communicate over a network. The protocol they use determines how requests are formatted, how responses are structured, and how fast the communication is. The professor compared three options: REST, GraphQL, and gRPC.

REST (Representational State Transfer)

REST is the most widely used approach: approximately 90% of ML applications use REST. It was defined by Roy Fielding in 2000 as a software architectural style for distributed systems.

REST operates over HTTP with JSON as the data-exchange format. The textbook (T2 Ch11) explains: "RESTful APIs are the most common... A RESTful API uses HTTP methods to make a request, then returns a response as a JSON file." The four standard HTTP verbs cover most API needs:

REST calls are individual: each call retrieves or submits a specific resource. AWS Bedrock, OpenAI's API, and most cloud ML services expose REST APIs.

GraphQL

Developed by Facebook (now Meta), GraphQL addresses REST's limitation when a client needs data from multiple resources.

query {
  user(id: 123) {
    name
    posts {
      title
      comments { text author { name } }
      likes { count }
    }
  }
}

The request hits any number of backend data sources (DynamoDB table, S3 bucket, relational database, another REST API) and collates everything into one response. This "single request, all-inclusive response" pattern makes GraphQL especially popular in social media applications where data is deeply nested and clients need flexible queries.

There is no inherent limit on response size; pagination and lazy loading can be implemented at the presentation layer — GraphQL supports both.

gRPC (Google Remote Procedure Call)

gRPC extends the traditional RPC (Remote Procedure Call) concept: the caller treats a remote service as if it were a local function call. Instead of thinking in terms of HTTP resources (GET /users, POST /orders), the developer thinks in terms of function calls: getUser(123), createOrder(orderData).

The client defines a contract (a .proto file) specifying the service methods and their input/output message formats. Both client and server must adhere to this contract. The contract looks like:

service PredictionService {
  rpc Predict(PredictionRequest) returns (PredictionResponse);
}

message PredictionRequest {
  repeated float features = 1;
}

message PredictionResponse {
  int32 prediction = 1;
  float confidence = 2;
}

gRPC uses Protobuf (Protocol Buffers) — a binary serialization format — instead of JSON, and operates over HTTP/2, which supports bidirectional streaming.

Real-world connection. Google uses gRPC extensively for internal service communication (it was invented there). Netflix uses a mix of REST (for external APIs) and gRPC (for internal service-to-service calls). Facebook/Meta uses GraphQL for its mobile apps (which need to fetch deeply nested data in one request). For ML model serving, REST via FastAPI remains the standard choice for most teams.

6.9 Microservices ML Pattern and Demo

6.9.1 ML-Specific Microservices and Implementation

Microservices ML Pattern. The core principle of microservices extends cleanly to ML systems: each microservice focuses on one specific ML functionality. The textbook (T1 Ch08) identifies this as a core architectural challenge: "Decomposing the system and deciding how to divide the work is a key step." In ML systems, the natural decomposition follows the ML lifecycle:

Each of these can be an independently deployable microservice communicating with others via synchronous (REST, gRPC) or asynchronous (message queue) mechanisms.

Independent scaling example. Suppose the application has a model service (version 5) and a prediction service. If the number of users grows from 10 to 1,000,000, only the prediction service needs to scale horizontally — spin up more instances behind a load balancer. The model service remains at its current scale until the model itself needs updating (from version 5 to 5.1). This targeted scaling is impossible in a monolith, where everything scales as one unit.

Microservices ML demo — the Iris application refactored

The professor refactored the monolithic Iris application (section 6.6) into three microservices:

1. API Gateway (port 8000): Receives user input, routes requests to the appropriate service. It does only routing — no business logic. The gateway is the single entry point; users never call downstream services directly.

2. Model Service (port 8001): Loads the logistic regression pickle model, accepts feature lists via POST, runs model.predict(), and returns the prediction (0, 1, or 2). This service owns the model — it is the only service that knows about the model file, the feature order, or the prediction logic.

3. Logging Service (port 8002): Receives both the input features and the model's output prediction, and logs them. In the demo, this appends to a list; in production, this would write to a database (PostgreSQL, Elasticsearch, or a cloud logging service).

Deployment order matters. The model service and logging service must be running before the API gateway starts, because the gateway depends on both downstream services. If the gateway starts first and receives a request, it will fail because the model service is not yet available.

This is a common operational concern in microservices: service dependencies. In production, tools like Docker Compose, Kubernetes, or service meshes handle startup ordering and health checks. The demo manages this manually by starting services in the right order.

Real-world connection. Production ML systems at companies like Uber (Michelangelo), Netflix, and Spotify follow this pattern: a model registry stores trained models, a model service loads and serves them, a feature service provides real-time features, and an API gateway routes requests. The Iris demo is a simplified version of this architecture.

6.10 Event-Driven Architecture

6.10.1 Publish-Subscribe and Message Brokers

Event-driven architecture decouples producers from consumers through a message broker. A producer publishes an event; any number of consumers subscribe to and process that event asynchronously. Producers and consumers are completely agnostic of each other — they know only the event topic or queue.

The textbook (T1 Ch08) defines this pattern: "Individual system components listen to messages broadcasted by other components, typically through some message bus. Since the component publishing a message does not need to know who consumes it, this architecture strongly decouples components in a system and makes it easy to add new components."

What is an event? An event is a fact about something that happened in the system. Examples:

• A user logs in, logs out, places an order, cancels an order.
• A new product is launched.
• A model detects data drift.
• A training job completes.

Each event carries relevant data — for an order-placed event, the items and quantities. For a model-drift-detected event, the drift metric and the affected model version. Events can trigger mass notifications (emails to a million users, SMS, WhatsApp messages), and asynchronous processing avoids the latency of waiting for each downstream action.

Publish-subscribe model. The professor explained the pub-sub model with these properties:

• A set of producers sends events to a message platform.
• A set of consumers subscribes to event topics and receives events as they arrive.
• The relationship is flexible: one producer to one consumer, one to many, many to one, or many to many.
• New consumers can be added without modifying producers, and vice versa. This is the key extensibility benefit.
• The message platform (broker) guarantees reliable delivery, message ordering, and persistence until all consumers have processed each event.

Message brokers. The broker sits between publishers and subscribers, ensuring three guarantees:

• Reliable delivery — no events are lost. The broker persists events until all subscribers have processed them.
• Ordering — a morning 25°C reading arrives before the afternoon 32°C reading. Events are delivered in the order they were published.
• Persistence — events are held until consumed by all subscribers; after that, they can be archived for replay or audit.

Event-driven architecture in ML. Logging is the canonical ML use case for asynchronous communication, but there are many others:

• Model monitoring → retraining trigger: A monitoring service detects data drift (input distribution changed) and publishes a "drift-detected" event. The training service subscribes and automatically starts retraining.
• Pipeline orchestration: Data ingestion service publishes "ingestion-complete" event → preprocessing service subscribes and starts cleaning. Preprocessing publishes "features-ready" → training service starts.
• Batch inference notification: A batch prediction job completes and publishes a "batch-complete" event → downstream services (dashboards, databases, notification systems) consume independently.

The architect's design decision. The professor highlighted an important design refinement that a student identified during the demo.

In the event-driven version of the demo, the API gateway published log events to the message broker. But this violates the gateway's design principle: an API gateway should only route, not perform business logic like logging.

This is a recurring theme in software architecture: the demo optimizes for teaching clarity, while production systems optimize for correctness. Recognizing this gap — and knowing which design is actually correct — is a key architectural skill.

Real-world connection. Event-driven architecture is the backbone of real-time ML systems. LinkedIn uses Kafka to process over 7 trillion messages per day, including ML feature pipelines. Uber uses event-driven architecture for real-time pricing, driver matching, and fraud detection. The pattern is essential whenever ML models need to react to events in real-time rather than processing batches on a schedule.

6.11 Model Registry Pattern and Version Control

6.11.1 Model Registry and Version Control Systems

The registry pattern — well-established in software engineering — provides a central hub where artifacts (code, binaries, models) are stored and versioned. A package registry (like npm, PyPI) stores code packages. A container registry (like Docker Hub) stores container images. In ML systems, the model registry pattern extends this concept: a centralized repository for tracking, versioning, managing, and governing ML models throughout their lifecycle.

Core capabilities of a model registry:

Popular model registry tools:

The professor noted that DVC will be covered in a dedicated webinar, and MLflow model registry will be demonstrated in the next class.

Version control evolution — three generations. The professor traced the history of version control to explain why ML needs specialized tools:

Why ML needs multiple tracking dimensions. This is the key architectural insight of this section.

Unlike pure software, ML systems involve multiple artifacts that change independently:

Git alone tracks code well but struggles with large datasets (gigabytes of training data) and model binaries (hundreds of megabytes). Storing a 500MB model file in Git is technically possible but bloats the repository and slows every operation. This is why specialized tools like DVC, MLflow, and Weights & Biases have emerged — they complement Git by handling the data and model dimensions of versioning.

Real-world connection. MLflow is the most widely adopted open-source model registry, used by companies like Databricks, Microsoft, and thousands of ML teams. DVC is popular in teams that want Git-like workflows for data and models. Weights & Biases dominates experiment tracking in research and industry labs. Amazon SageMaker Model Registry is the natural choice for teams already on AWS. The choice depends on your stack, team size, and whether you prefer open-source or managed services.

Exam Guidance Summary

Key Industry Applications

This lecture referenced numerous companies and tools. Below is a consolidated reference of where each appears in the architectural landscape.

Microservices and deployment at scale:

• Amazon: Approximately 23,000 microservices deployments per day; e-commerce and AWS services use microservices and event-driven architectures.
• Google: Approximately 5,500 microservices deployments per day; gRPC originated at Google.
• Flipkart (2014): Big Billion Day failure attributed to monolithic architecture that could not scale under peak load — the canonical example of monolithic scalability limits.
• Hotstar (2019): Scaled to 25 million concurrent users during an India vs New Zealand cricket match, enabled by microservices architecture.
• YouTube: Uses microservices — recommendation service can fail independently without affecting video playback, comments, or downloads.
• Instagram: Built with Python, React Native, and JavaScript; polyglot microservices architecture.
• Swiggy/Zomato: Food delivery platforms — the professor's archetypal example for comparing monolithic and microservices architectures.

RAJ ecosystem:

• OpenAI: GPT-4o mini and text-embedding-3-small used in the RAJ demo; multimodal LLMs enable image-to-text pipelines.
• LangChain: Modular framework for building RAJ pipelines with pluggable document loaders, text splitters, embedding models, and vector stores.
• ChromaDB: Vector database for storing and querying embeddings in the RAJ architecture.
• Streamlit / Gradio: Lightweight UI frameworks for ML application prototyping and demo interfaces.

ML serving:

• FastAPI + Uvicorn: De facto standard for ML serving in Python; used throughout all architecture demonstrations.

Event-driven infrastructure:

• Apache Kafka: Dominant message broker for event-driven architectures in both software and ML systems.
• RabbitMQ: Widely used open-source message broker using the AMQP protocol.
• IoT sensors + event-driven: Environmental monitoring systems where sensors publish readings as events consumed by dashboards, warehouses, and alerting systems.

Model management:

• MLflow: Open-source model registry and experiment tracking tool.
• DVC: Data and model version control tool extending Git-like workflows to ML artifacts.
• Weights & Biases: Experiment tracking and model management platform.
• Amazon SageMaker: Cloud-native ML platform with built-in model registry.