Kafka Architecture

7.1 Message Delivery Semantics: The Three-Question Decision Framework

7.1.1 The Three Delivery Semantics in Review

The session opened with a recap of message delivery semantics, the concept carried over from the previous session: exactly once, at least once, and at most once. These are the three ways a streaming system can behave when it moves messages between components. A message is either delivered exactly one time, delivered one or more times, or delivered at most one time (which means it is allowed to drop the message). This was flagged as a very, very important concept — not a topic to skim, because it drives how every streaming use case gets designed.

The three promises, stated precisely:

• At most once. A message may be lost, but it will never be read again by a consumer. The system does not retry and does not redeliver. Losing a message is allowed; repeating one is not.
• At least once. A message will never be lost, but it may be read again. The system retries deliveries, so the same message can reach a consumer two or more times. Repeating is allowed; losing is not.
• Exactly once. A message is never lost and is read by a consumer once and only once. This is the strongest promise, and it is the most expensive to deliver.

The three semantics are not a menu of "nice features". They are answers to the question of what your business can and cannot live with. That is why the practical payoff of the recap matters: for any use case you can run a small set of questions, and the answers tell you which delivery semantics fits. The class walked through the three questions one by one.

7.1.2 The Three Questions That Decide the Semantics

Question one is about impact. What will be the impact if communication between the components breaks down? This is the business impact question — if the stream stalls or messages get lost, what does that cost the business in the real world? For a fraud-detection service, a stalled stream means fraudulent purchases pass unnoticed while they happen — the cost is direct and immediate. For a recommendation engine, a stalled stream means less-personalized product pages — the cost is softer and indirect.

Question two is about tolerance. How much data loss can the business tolerate? Not "would you like to lose data", but how much worth of data the business can afford to lose. Some businesses can shrug off a dropped reading; others cannot. A bank that loses a transaction record may lose a customer's trust and face legal trouble; an analytics dashboard that loses one temperature reading out of a thousand is barely affected. Notice the question is asked in terms of worth, not volume: ten minutes of lost stock-market ticks can matter more than ten hours of lost sensor logs.

Question three is about storage. What is the requirement for long-term storage of historical data? The scope of storage decides how much you must keep around, which changes what the system has to guarantee. If a business needs to replay a month of events for reporting, the queue must keep messages durable for weeks — that is a very different system from one that deletes each message as soon as a consumer reads it.

So the framework is: business impact, data-loss tolerance, and storage scope. Every use case goes under these three and gets answered before you pick a name for the delivery semantics.

7.1.3 Using the Framework: Bank Application and Recommendation System

Two example domains were revisited as the running examples that motivated the framework: a bank application and a recommendation system. (The full worked-through versions of both belong to the previous session; this class returned to them only as reminders.) The point of the recap is the method: you do not start from the answer. You start from the three questions, and the use case's answers align you to one of the three semantics.

Sketching the alignment only briefly: for a bank application, a communication breakdown is expensive (a missed payment instruction can cost money and trust), data loss is not tolerable (every transaction is financially meaningful), and historical records are needed for statements and audits — all three answers point toward exactly-once. For a recommendation system, a stalled stream costs some sales, an occasional lost browsing event barely matters, and historical data is used to refine the model rather than to keep legal records — the answers point toward at-least-once, and sometimes even at-most-once is enough.

7.1.4 Exam Note: How to Answer a Delivery Semantics Question

Why does the framework lead, rather than the definition? Because the definitions say what each semantics does, but only the three questions say which one a given use case needs. An answer that names exactly-once without walking the three questions has skipped the reasoning the question is testing.

7.2 Kafka in the Generalized Streaming Architecture

7.2.1 Kafka as the Data Flow Layer

Kafka is a messaging system. In the generalized streaming architecture, data moves from a collection tier to a processing layer, and between them sits the data flow layer. Kafka is what you keep in place of that data flow layer — that is its main utility. Its job is to match the impedance in terms of speed: events arrive at the collection tier at one speed, and the analysis tier can only consume at another speed. The message queue in the middle absorbs that difference.

Impedance is a word borrowed from electronics. Two circuits connected directly transfer signal well only when their impedances match; a mismatch causes reflections and wasted power. The streaming version is the same: the collection tier and the analysis tier have different natural rates, and the data flow layer is the matching element between them. The word keeps its meaning here: match the rates, and data flows smoothly; leave them mismatched, and the system wastes effort or drops data.

The generalized picture runs collection tier → data flow layer (Kafka) → analysis layer. The processing engine used alongside it in this course is Spark: Kafka performs the data ingestion, the messages are then sent to Spark, processing happens in Spark, and the results go onward to a sink. Kafka architecture itself continues as a topic even after the mid-semester exam.

7.2.2 The Publish–Subscribe Model and Kafka Connect

Kafka works on a publish–subscribe model. Publishers push messages into a message queue, and downstream client applications consume them on the other side. On one end you have a producer API; on the other end, a consumer API. In the Kafka Connect framework these map to source connectors on the producer side that bring data in (source connect), and sink connectors that push processed data out (sink connect) — different types of interfaces on each side.

The producer API works with concrete classes — concrete meaning set up against specific, defined producer topics. Producers can be added on the fly and removed on the fly; you can scale them dynamically. Consumers work the same way: many consumer API instances, added dynamically, each identified by a key ID, because the key ID is what tells you which producer or consumer you are dealing with. Messages are produced for further analysis, and they arrive in batches — batches of, say, 100 events, or 20 events, and so on. Batching is a throughput trick: one network round trip that carries 100 events moves more data per second than 100 separate trips ever could.

One scope note: this course does not look at orchestrating Spark and PySpark components. Platforms like Databricks make that configuration easy, and it is treated as a separate matter.

7.2.3 Student Questions: Topics Only, and Where Spark Fits

That answer is worth slowing down on, because "queue" is the loaded word in it. In traditional messaging products like RabbitMQ or ActiveMQ, a queue is a first-class object with its own rules, and a message disappears from the queue once a consumer takes it. Kafka deliberately has no such object: the unit of organization is the topic, and everything — grouping, buffering, reading — happens through topics and their parts. The queue-like behavior people expect comes from the topic machinery itself, not from a separate queue facility.

The abstract view is the one to carry into the exam: the Kafka architecture is defined by the producer–consumer relationship, not by any particular processing engine that happens to sit on either side. The course's wiring — Kafka into Spark into a sink — is one instance of the general picture, not the definition of it.

The marketplace picture maps cleanly: producers publish products (messages) into one central place (the broker), customers (consumers) walk in, and each customer selects what they need (subscribes to topics) rather than taking everything on the shelf. The analogy has a limit worth noting: a market has finite shelves, while a Kafka topic keeps accumulating until retention removes old segments.

7.2.4 Industry Applications: Kafka in Production

Real-world: Adidas uses Kafka as a data streaming platform. Its purpose is enabling the teams to implement real-time event processing for monitoring, analytics, and reporting.

Real-world: a Shopper Stop purchase is a real event. When a purchase happens, a message is generated that carries which bank card you used, the volume of the transaction, and the kind of purchase. Different customers come in and generate such messages continuously, and depending upon the business objective the system generates appropriate insights from those instant details.

Real-world: Agoda uses Kafka for analytical streaming purposes, and an event-cloud platform was mentioned for monitoring messages. Coursera is another user: Kafka powers education at scale, serving as the data pipeline for real-time learning analytics and dashboards.

These use cases share a shape: a real-world event (a purchase, a click, a course interaction) becomes a message, and the business value comes from acting on the message while it is still fresh. That freshness is exactly what the data flow layer buys — the analysis tier no longer waits for a nightly batch to learn what happened today.

7.2.5 A Distributed, High-Throughput System

Apache Kafka is a distributed publish–subscribe model — which means it runs as a cluster of machines. It is designed as a high-throughput distributed system, suitable for large-scale applications. Its importance lies in scalability: you can add producers and consumers, where producers are nothing but the various data systems through which data actually comes into the processing engine (Spark, in this course's setup). The log-based design lets a production cluster handle millions of messages per second: the work is spread across machines, and each machine only sees its own slice of the load.

Kafka takes care of many non-functional requirements — reliability, scalability, maintainability, and so on. Within the cluster there is a notion of leaders and followers; the leader is also a node. A bunch of machines are connected together and all work toward one common goal. In replication terms, each partition of a topic has one leader, and the leader serves all reads and writes for that partition; follower nodes hold copies of the partition, so that if the leader fails the cluster can keep going with another node.

The basic running model: a broker is the Kafka server. In the Java distribution you run a server start script (the server.bat file), and executing it is how you configure Kafka. A set of producers publish to topics, and consumers subscribe to those topics and consume them. In a single-machine setup one broker sits at the center; in a production cluster, brokers are the nodes of the cluster.

7.3 Topics and Partitions

7.3.1 What a Topic Is: The Inbox and Subject-Line Analogy

A topic is a group of messages of the same type — a kind of entity, a subject. The best way to feel it is through the inbox. Think of your Outlook inbox at the office: do you respond to every message you receive? Nobody does; some messages are not even read. On what basis do you decide? The subject line. The subject line tells you what a message is about, and that decides whether you pay attention to it or not.

A Kafka topic is the subject line for messages. Incoming messages are segregated into named streams by topic, and that segregation is what lets the right consumers pick them up. The topic determines which messages a consumer will receive.

The mapping in simple words: a message's topic is what tells a consumer "this is for you" or "this is not for you". A consumer subscribes to the topics it cares about and ignores the rest — exactly like reading only the emails whose subject lines matter to you. The analogy's limit: an inbox subject line is written by a human and read by a human, while a topic label in Kafka is a technical contract between producer and consumer — both sides agree on the topic name and the message shape before any data moves.

7.3.2 How Messages Reach Consumers

Messages come into the system as a stream. They are written to topics, and the topics are read by the broker. When a producer sends messages, it cannot write directly to consumers — the messages go to a topic first, and it is the broker that looks at the topics (a topic, or several topics, each internally divided). Because the messages are sent based on some schema, they can be segregated by topic as they arrive. Consumers then consume the messages they are interested in through their subscribe method, based on the topics that are part of their subscription. The topics a subscriber holds are processed in small portions, and the analysis runs on those portions.

Under the hood, a topic is a log — an append-only sequence of records on disk. Producers do not overwrite old messages; they append new ones to the end. Consumers read the log from front to back. This one design choice explains the queue behavior from the previous section: messages wait inside the log until a consumer is ready, and a new consumer can even start from the beginning of the log and read history — something a traditional queue that deletes consumed messages cannot do.

7.3.3 Partitions: The Data-Frame Analogy

Each topic is made up of many partitions — that rule is emphasized as important. A partition is a logical group of messages. The analogy: a data frame. You take a data frame and divide it into subsets of data frames, and each subset is then modified independently. Kafka partitions a topic the same way: the topic's messages are split into subsets, and each partition can be processed independently, which is what gives Kafka its parallelism.

The partitions live across the nodes of the cluster — it is a distributed network, and each node maintains its share of partitions. So Kafka supports partition-level processing of the same family that Spark performs, all facilitated through Kafka's own topic and partition machinery.

Two properties follow from this layout, and both matter for exams and for design:

• Parallelism is bounded by partition count. Within one partition, messages keep their order; the broker assigns every message a sequence number called an offset (a counter that starts at 0 and grows by 1 per message). Parallel processing comes from assigning different partitions to different nodes and different consumers — not from splitting one partition's stream among several workers. A topic with 4 partitions can be consumed by at most 4 consumers working in parallel (one per partition); a topic with 40 partitions can spread the load much wider.
• Ordering is guaranteed within a partition, not across the topic. If messages A, B, C land in the same partition, consumers see A then B then C. If they land in different partitions, no global order exists. So when order matters, the producer must route related messages (say, all events for one customer) to the same partition.

Where the data-frame analogy stretches: a data frame you split and modify in parallel is usually reassembled afterward; Kafka partitions are deliberately never merged back — the whole point is that independent consumers each chew on their own subset forever.

7.4 Ingestion Rate vs Processing Rate (Ri and Rp)

7.4.1 The Rate-Matching Problem

Back to the architecture picture: the collection tier emits events at some speed, and the analysis tier processes events at some speed. Two rates govern the design. $R_i$ (R-sub-i) is the ingestion rate — the rate at which messages are incoming to the streaming layer. $R_p$ (R-sub-p) is the processing rate — the rate at which messages are being processed. The message queue in the data flow layer exists to match the impedance between these two speeds.

Picture the two tiers as pipes of different diameter. The collection pipe feeds events into a tank (the queue), and the analysis pipe drains the tank. If the feeding pipe is wider, the tank fills; if the draining pipe is wider, the tank stays empty. The water level at any moment is the backlog — the number of messages waiting. The two rates decide whether that level rises, falls, or stays flat, and the queue exists so the tank is there to absorb the difference instead of letting the pipe burst.

Whenever you compare the two rates, first make sure the units are matched. You can only divide or compare them when both are in the same unit — for example, both in MB per second. Comparing 16 GB per minute against 10 MB per second without converting is not a comparison at all: it is two different numbers about two different things.

7.4.2 Worked Example: Rp = 0.8 × Ri

7.4.3 Worked Example: Ri = 30 × Rp

Setup: $R_i=16$ GB per minute and $R_p=10$ MB per second. One unit is in GB, the other in MB; one is per minute, the other per second — nothing matches yet. Before any division, the unit conversion must happen.

The rule to remember: this course converts with the decimal convention, 1 GB = 1000 MB, not the binary convention of 1024 MB per GB. The binary convention exists (it comes from powers of 2 in memory addressing), but exam problems in this course are set with 1000, because the arithmetic stays clean.

7.4.4 Interpreting Ri = 30 × Rp: What Your Architecture Must Do

The question's trap is the word "streaming". Both layers belong to the streaming system, so "the streaming layer" is ambiguous; the rate comparison removes the ambiguity. The ingestion rate $R_i$ is 30 times the processing rate $R_p$, so the ingestion side wins the race by a factor of 30 — and winning the race is exactly the problem.

So what do you do? First option: scale up the streaming layer — instead of a one-node cluster, use a 20-node or 30-node cluster, so parallel processing happens across the nodes. In other words, speed up the processing time. Second option: keep a buffer storage to hold data before processing. The catch with a buffer: after one minute you get that many new messages again, so buffering alone never finishes the job. The backlog is not an accident; it is the arithmetic of $R_i=30R_p$ repeated every minute. Put concrete numbers on it: writing continuously at 150 MB/s, a 6 TB disk fills in about 11 hours (6,000,000 MB ÷ 150 MB/s ≈ 40,000 seconds). So a buffer buys time measured in hours or days, not forever — it is a breathing room, not a solution.

The delivery semantics also decides how much you must keep around. With exactly-once semantics you need to do something extra — you can make it a queue and handle each message once. With at-most-once semantics you do not need every message persisted, stored and processed; you can take small samples and process those, because at-most-once allows you to drop messages, which fits operational monitoring. With exactly-once you cannot blindly drop messages; some kind of rework or reconciliation has to be done.

7.4.5 Exam Notes on Rate Numericals

7.5 Design Decisions for the Data Flow Layer

7.5.1 Choosing a Strategy from the Rates

The rates decide how you process. If Ri is of the order of Rp, you can write messages directly to the processing layer — the queue is barely needed. If Ri is much greater than Rp, you go for batch processing, horizontal scaling, and buffer or queue storage. So depending on these rates, you basically decide how you want to process them. The boundary case to keep straight: it is the fast ingestion side (Ri much larger than Rp) that forces the buffered or batched path, because the processor cannot keep up with what arrives. If the rates are comparable, the system can afford to hand each message straight to the analysis tier.

Message delivery semantics also plays a role. At-least-once semantics means a lot of messages must be sent (retries included), which changes how the rates can be tuned — the delivery promise constrains how Ri and Rp may be fine-tuned, because the effective volume a system must absorb includes every retransmission, not just the first copy of each message.

7.5.2 Horizontal Scaling and the Kafka Cluster

Horizontal scaling means: instead of a single machine, you create a Kafka cluster. A cluster has multiple brokers, and each broker is one node in the cluster. Adding brokers is the standard way to scale the data flow layer — you keep adding more brokers, and parallel processing happens across the nodes. The partitions of the topics spread over the broker nodes, so each new broker takes on a share of the partition-level work, and the throughput of the whole layer grows with the number of machines.

A practical sizing rule from production practice: for a development or test environment one broker is usually enough, while a production system runs at least three to five brokers so that replicas of a partition live on different machines and a single node failure does not take the layer down.

7.5.3 Queue and Broker Together

The two answers come from two different pressures. Brokers address the processing side of the mismatch: more nodes, more parallel processing, faster $R_p$. The queue addresses the arrival side: it holds the events that arrive while the processors catch up, which is what the "buffer 30 messages per message processed" rule from the previous section calls for.

So the two tuning knobs — horizontal scalability and buffering — combine rather than compete. The queue sits in front of the brokers and feeds them; the brokers parallelize the processing of what the queue hands over. A common pipeline shape in this course's architecture is exactly this: incoming events land in the queue, the queue forwards them to the broker cluster, and the brokers feed the analysis layer.

7.5.4 Delivery Semantics Interplay

Which strategy goes with which scenario was deferred to the next session, but the logic is already visible. At-most-once semantics lets you drop messages, so sampling small portions is legitimate — that is the operational monitoring class of use cases. At-least-once means the system must keep delivering, with retransmission, so the ingestion volume handled is higher. Exactly-once means every message matters: you cannot drop, and something must be done about duplicates — the queue-and-handle pattern, where each message is worked exactly once and any repeat delivery is detected and reconciled.

7.6 Setting Up and Running Kafka

7.6.1 Download and Folder Structure

Downloading Kafka (the Confluent Kafka distribution was used for the demonstration) puts a folder on your local machine. Inside it you open the bin directory: it contains so many batch files or shell scripts. The scripts are Unix or Linux shell scripts; when you work on a Windows system you double-click into the Windows folder, which holds the equivalent batch (BAT) files. Same scripts, same order of steps, different file extension — the Windows side wraps the same commands in .bat files so they run without a Unix shell.

7.6.2 The Four Startup Steps

Running a Kafka application requires four steps, in order:

These steps were shown only as a preview; the actual walkthrough was deferred to the next session because the scripts take time to execute.

7.6.3 Student Questions: Live Demo Next Session

The preview is worth keeping straight until the demo: the order is Zookeeper first, broker second, then the two client configurations. Every Kafka application — including the one shown in the live demo — follows this same sequence.

7.7 Zookeeper Recap: Node Types

7.7.1 Persistent, Ephemeral, Sequential

A quick recap from the Zookeeper session: Zookeeper keeps data in nodes called z-nodes, and there are three types. Persistent nodes stay until they are explicitly removed. Ephemeral nodes exist only as long as the session that created them is alive. Sequential nodes get a sequence number appended, so ordering among them is preserved.

These three node types — persistent, ephemeral, sequential — are the building blocks discussed earlier, and they matter again here because Kafka uses Zookeeper for cluster coordination: it is the first thing you start when you launch Kafka. The connection to the Kafka startup steps is direct: Zookeeper tracks which brokers are live (ephemeral nodes that vanish on failure), where partitions live, and which replicas are in sync — the shared state that step 1 of the four startup steps provides.

Exam Guidance Summary

• Message delivery semantics is a very important concept. A use-case question on it expects the three-question framework written out: business impact if communication breaks down, how much data loss the business can tolerate, and long-term storage requirements. Discuss the use case along all three before naming the delivery semantics.
• Rate numericals on Ri and Rp are important. Expect given ingestion and processing rates in different units; match the units (1000 MB per GB, both converted to MB per second), compute the ratio, and interpret what it means for the architecture.
• The topics and partitions model is examinable: what a topic is, how partitions give parallelism, and the inbox and data-frame analogies are the mental models the examiner expects you to reproduce.
• Syllabus status: the material up to and including this session — the Kafka architecture — is part of the mid-semester scope; everything covered today is examinable.
• A list of mid-semester topics to focus on will be shared.
• A sample question-and-answer discussion is planned for the next session, so you can see how answers are expected to be structured — useful for both the delivery-semantics questions and the architecture questions.

Key Industry Applications

• Real-world: Adidas uses Kafka as its data streaming platform, enabling teams to implement real-time event processing for monitoring, analytics, and reporting.
• Real-world: a Shopper Stop purchase generates a real-time message carrying which bank card was used, the transaction volume, and the kind of purchase; different customers generate such messages continuously, and the system derives appropriate insights from those instant details.
• Real-world: Agoda uses Kafka for analytical streaming purposes.
• Real-world: Coursera uses Kafka to power education at scale — it serves as the data pipeline for real-time learning analytics and dashboards.
• Real-world: event-cloud platforms use Kafka-style streaming for monitoring messages.
• Real-world: Azure Event Hub is the alternative mentioned when a single-subscriber queue model is not enough — multiple topics or an event hub carry the data instead.
• Real-world: the producer–consumer picture is the same one you see in a marketplace — producers publish products, and subscribers (customers) select and filter what they want.