Mid-Semester Problem Solving: Architecture, Message Design, and Kafka Partition Estimation

8.1 How to Approach a Scenario-Based Architecture Question

Picture the exam question: "Design a data processing architecture for the following system…" followed by a paragraph of business description. Most students react by drawing boxes immediately. The instructor's advice is the opposite — before you draw anything, before you name a single component, make one decision first: is this a batch problem, a stream problem, or both? Everything else in the answer hangs off that single choice.

8.1.1 The First Decision: Batch or Stream

Every scenario question in this area comes down to one starting decision: is this a batch processing use case, a stream processing use case, or both? Before you draw anything, before you name any component, decide that. The instructor's rule for reading a problem is direct: read the scenario, look for the activities, and for each activity ask which pipeline it belongs to. You do not invent use cases. You find the ones that are already written into the problem statement.

Two quick definitions to anchor the decision. A batch processing use case (a job that runs on a full collection of data at once) is one where the data is collected first and processed later, on a schedule — nightly, hourly, or daily. A stream processing use case (a pipeline that handles each message as it arrives) is one where the answer must be produced while the events are still happening — the moment a message is generated. Some scenarios are purely one or the other; many are both, with different activities on each side. Your job is to label every activity in the text with its regime before choosing an architecture.

To find them, underline every activity mentioned in the problem text. Only those activities get mapped to components. If a requirement is genuinely unclear, that is not a licence to guess silently — you may state an assumption out loud: "this requirement is not clear, but I am assuming that this requirement is also present." Assumptions based on the domain are acceptable. But the default is: the problem statement is the contract, and every underlined activity must land somewhere in your architecture.

8.1.2 Lambda versus Kappa

Once you know whether the use case is batch, stream, or both, you pick the architecture family. Lambda architecture carries both a batch provision and a stream provision — batch layer, speed layer, and a serving layer that merges them. Kappa architecture is useful only for stream processing; there is no batch layer in it. So the reasoning chain is: decide the processing regimes → if both batch and stream are needed, Lambda; if only stream, Kappa; if only batch, Lambda with the streaming layer removed (the instructor's phrasing for the education case later: "Lambda architecture without the streaming layer").

If you choose Lambda, you then enumerate its components and map the problem's scenarios onto individual components. That mapping — not the generic description of what a collection tier does — is what earns the marks.

8.1.3 Reading the Problem: Underline Activities, Map to Components

When asked to explain the components of a data processing architecture, the instructor's advice is concrete:

• Keep a generalized architecture in mind (collection tier → data flow layer → analysis/processing tier → serving/delivery).
• For each component, write only what happens with respect to this use case. Do not write the generic story. The evaluator already knows what a data flow layer is responsible for; you are not being tested on that. You are being tested on what this specific data flow layer does for this specific problem.
• Keep each component as its own subheading and write under it. If the question says "draw", draw; otherwise headings and prose are fine.
• Once the data processing is done, think about what happens after — the delivery tier. That part of the flow is easy to forget and is part of the same question.

The four-tier skeleton is worth keeping fixed in memory, because it is the template every scenario maps onto: the collection tier is where events enter the system; the data flow layer moves and holds messages between tiers (this is where a message queue or a Kafka-style broker lives); the analysis/processing tier does the computation on the data; the serving/delivery tier pushes the results out — to screens, customers, or downstream systems. When you are asked to "explain the components," walk these four in order and say, for this scenario only, what each one does.

8.1.4 How Marks Are Distributed and How to Write Answers

For the sample question the breakdown was made explicit: explaining the various components of the architecture is worth six marks; evaluating the architecture on non-functional requirements is worth three marks; message management (delivery semantics) is worth three marks. The instructor's advice about six-mark parts: first look at what is asked, then answer it properly and completely — points are not counted, substance is.

8.1.5 Student Questions and Answers

Real-world connection: this method is not exam-specific. In industry, system-design interviews and architecture reviews follow the same discipline — an architect is given a business description, must separate the workloads that need immediate answers from the ones that can wait, and must justify every component against a stated use case. An architecture that "does streaming because streaming is modern" fails review; one that can point at the specific activity each tier serves passes.

8.2 Worked Case Study: Smart Device Banking System

This is the first full run of the recipe from Section 8.1 on a real exam question. Watch how a paragraph of banking business text turns, line by line, into a Lambda architecture with named components — and how every answer point traces back to an underlined activity in the problem statement.

8.2.1 The Problem Statement

The sample question ran as follows. Consider a smart device system pertaining to the banking domain. The core activities under consideration are customer service, home loan, personal loan, unit link insurance plans, and similar products, and the bank wants to track and process these transactions. At the same time the bank is expected to provide safety measures around credit card and debit card transactions — tracking them and processing the data. The closing line of the problem says the bank likes to use insights from card-related transactions for the loan department as well.

The instructor walked the class through the last line explicitly, because it is the hinge of the whole problem. "Based on your spending history, they may also offer you some loan" — that is what it means. The main emphasis is on credit card and debit card transaction alerts and the analysis that follows them.

8.2.2 Mapping Use Cases: Batch and Stream

The decision tree starts with the transaction alerts. Card transactions happen in real time, so the alerts must be produced as the transactions arrive — that is stream processing territory: process the message the moment it is generated. Fraud detection sits in the same bucket: to ensure safety you need fraud detection in real time, at the moment the message is generated, not later.

The loan side is batch. The bank wants insights from historical transactions — how much customers are spending, what they are buying — and uses those insights to broadcast offers from time to time. Historical analytics, promotions, spending analysis: all batch. The activities around customer service, home loan, personal loan, and unit link insurance plans are tracked and analyzed for insight generation, and that tracking can also be placed in the batch regime.

So the answer is: the use case needs both batch processing and stream processing. You should be able to say exactly which scenarios go to batch and which go to stream — and the instructor stressed that naming the split correctly is the first thing the question is checking. On that foundation the architecture is Lambda, which carries both provisions.

8.2.3 Batch Views and the Database-View Question

A question that produced real confusion: what kind of view do we generate from the batch? The instructor's example: a view in terms of the number of customers who have opted for a particular loan, or a view under demographics. That raised a follow-up — is this view the same as a database view? The answer is yes, it is, and the reasoning matters. You store the batch layer data in a database; the view is generated from that stored data, so the batch view is literally a database view. It is a snapshot of the analysis you generated, kept in the form of views. One more detail the instructor added: the views you generate come from the historical data in the batch layer and also from the speed layer.

Real-world: this is exactly how a bank's offer engine works — batch views of loan uptake and customer demographics feed campaigns, while the real-time view (transaction volume in the last two or three minutes) feeds the alerting side.

8.2.4 Replacing the Data Flow Layer with Publish-Subscribe

A common exam twist is to ask you to model the generalized architecture using a specific mechanism — for example, replace the data flow layer with a publish-subscribe model (pub/sub for short). The instructor walked through the mapping:

• The publisher-subscriber model has a producer, a broker, and a consumer. The broker maintains a message queue internally: producers put messages into the queue, and the queue delivers them to consumers.
• The producer is nothing but whatever comes through the collection tier. So the producer is part of the collection tier, not a separate box.
• The broker and its message queue reside in the data flow layer.
• The consumer is the analysis tier.

Then you explain how it works and what it buys you. Because the model decouples producers from consumers, you can add different types of producers on one side and different types of analytics platforms on the other side, and the broker maps them. That flexibility is the reason the design is chosen, and it should appear in your answer.

The decoupling benefit is worth spelling out in an answer: because the broker is the only thing the two sides talk to, the bank can add a new event source (a new card channel, a new loan product) without touching the analytics platforms, and can add a new analytics platform without touching the sources. In industry this is exactly why message queuing tiers exist — they absorb growth on either side without re-plumbing the pipeline.

8.2.5 After Processing: The Delivery Tier

The instructor's reminder: once the data processing is done, think about what happens next — the delivery tier. In the banking problem that is where alerts reach the customer, offers reach customers, and results of analytics are broadcast. It is part of the same architecture question and students routinely stop at the analysis tier.

8.2.6 Student Questions and Answers

The two questions about the scenario split were asked while the class was still reading the problem, and the instructor answered them inline:

The pub/sub mapping produced its own exchange:

Real-world connection: modern banking alerting runs exactly on this split. Card transaction systems such as those built on Kafka streams fire fraud alerts in seconds per transaction, while the marketing analytics warehouse recomputes offer and demographic segments daily — the two halves of one Lambda-style pipeline, one serving the customer-facing alert, the other the campaign engine.

8.3 Non-Functional Requirements Evaluation

The architecture is drawn; now the question turns: "evaluate the proposed architecture on non-functional requirements." This part carries three marks and is the most template-shaped of the whole paper — there are exactly three requirements to discuss, they were taught in class, and each gets a subheading and a use-case-specific story.

8.3.1 The Three Requirements

When a question says "evaluate the proposed architecture on non-functional requirements in the big data systems," the three to write about are the three taught in class: low latency, high availability, and horizontal scalability. Put each one as a subheading and write, for each, what you expect the architecture to provide in this use case.

8.3.2 Horizontal Scalability

The answer template the instructor modeled: considering the volume of customers the bank has, you may want a cluster with multiple nodes. Multiple brokers, and an analysis tier that is itself a cluster consisting of many consumers. The scale-out story is told in terms of the workload: if the bank is huge and has many customers, multiple brokers and multiple consumers handle the volume. You do not decide the topology first and then justify it; you look at the volume, and the volume recommends the cluster.

The direction of the argument matters in the answer. The wrong version starts with the design ("I will use a cluster of ten brokers") and then hunts for a justification. The right version starts with the workload ("a large bank processes a large number of card transactions per minute") and lets the volume dictate the topology ("therefore multiple brokers share the message flow, and the analysis tier is a cluster of consumers, each handling a share of the stream"). One sentence naming the volume driver, one sentence naming the cluster it implies — that is the template.

8.3.3 Low Latency: The Two Latencies

The class pushed back on a naive reading of low latency — "customer should be responded back with an alert, so latency matters," and separately "low latency is not critical because loan data is stored anyway." The instructor corrected both directions with the same point: the question is not whether you need low latency or not; it is what latency means in this use case and what you set up to meet it. Latency shows up in two places:

1. In the batch layer — how fast you can store something and retrieve it. That store/retrieve latency matters because the serving of insights depends on it.
2. In the streaming path — the speed at which Kafka is serving the analysis tier, the speed at which messages are ingested into the streaming layer, and the speed at which the streaming layer processes them.

Those speeds should be compared and analyzed, and the cluster is set up accordingly. You cannot say "latency is not required" by looking at the use case alone, because it depends on the number of customers and the number of loan takers. What you can say is: considering the volume, you may want to set up the cluster — that is the recommendation.

8.3.4 High Availability

High availability is the third pillar — the architecture should keep serving even when components fail. In this use case that translates to having redundancy across the tiers: more than one broker so a broker failure does not stop the message flow, and a consumer cluster so the analysis tier keeps running. The professor presented availability and scalability as siblings: the volume argument that drives cluster sizing is the same argument that buys you failover.

The sibling relationship is the key insight to reuse in the answer. You do not write two disconnected paragraphs — you write one story twice: the volume is too large for a single broker, so we run several brokers, and because several brokers exist, losing one does not stop the message flow. The cluster sized for load is the same cluster that provides redundancy. In the banking case: a second (or third) broker means a single broker failure degrades nothing, and a multi-consumer analysis tier means one failed consumer leaves the rest processing.

8.3.5 Student Questions and Answers

Real-world connection: the same three requirements appear in every production data platform. Cloud providers market exactly these properties — a message queue's replication factor answers "availability," its partition count answers "scalability," and its end-to-end delivery times answer "latency." A bank evaluating Kafka for card alerts reads the vendor documentation looking for these three columns before any feature list.

8.4 Message Delivery Semantics

The second question of the paper's pair is about message design — typically message delivery semantics, with justifications. This section holds the class's sharpest debate of the session: exactly-once versus at-least-once in banking, and the rule that finally resolves it — one semantics per architecture.

8.4.1 The Three Justification Questions

Whenever a question asks about message delivery semantics, the instructor's framework is three questions, and every answer must put all three down and justify each:

1. What is the business impact if I lose a message?
2. How much worth of data can the system tolerate?
3. What is the requirement for long-term storage?

The semantics choice — at-most-once, at-least-once, exactly-once — is a consequence of answering these three for the specific problem. The marks come from the justification, not from naming a semantic.

The three questions are the bridge from business to semantics: the first decides whether loss is tolerable, the second decides how much duplicate/retry machinery is worth paying for, and the third decides whether the system must keep messages for replay or can discard them. Answer all three for the given problem and the semantic chooses itself.

8.4.2 Exactly-Once versus At-Least-Once for Banking

The class opened with a confident guess: for banking, delivery must be exactly-once. The instructor agreed with the instinct but forced the reasoning out: what kind of transaction are you doing? For credit card transactions, exactly-once is the natural choice — a duplicated or lost charge is a real business event. But the full architecture is not only the card stream; it also carries messages pertaining to batch layer semantics. For the batch layer that kind of guarantee is not required; for the streaming layer you need the guarantee.

Then the alternative surfaced: at-least-once delivery, with the analysis tier handling replicated messages — deduplicating and ignoring duplicates. The instructor accepted that as a legitimate answer: it is also possible, provided you say why and you specify how the analysis tier handles duplicates, because the analysis tier is part of your design too. Both exactly-once and at-least-once-with-dedup are acceptable if justified.

The cost angle matters: exactly-once needs to be ensured from both ends — producer and consumer — and from a cost perspective that is a challenge. A student also pointed out that the use case gives no numbers — how many messages per second or per minute is not stated — which is a legitimate argument for the simpler at-least-once choice. When the problem does not quantify load, you can argue from that absence.

8.4.3 One Semantics per Architecture

The sharpest point of the debate: you may propose only one message delivery semantics for an architecture. Whether you have a batch layer and a stream layer, you cannot have one semantics for batch and another for stream. A student tried to argue that at-least-once is architecturally weak because delegating deduplication to the processing layer violates the separation of duties — the bifurcation of layers. The instructor agreed the point was valid, and generalized it: one message delivery semantics for the whole architecture, and if you delegate the dedup work to the processing layer, then how the processing layer does that work is part of your answer, because you are the one implementing that layer.

The logic behind "one semantics per architecture" is worth stating in full. The message pipeline — collection, data flow layer, processing — is one path; a message does not know whether it is a "batch message" or a "stream message" at the point it is produced, and the data flow layer cannot apply two different sets of rules to the same flow without doubling its machinery. So the guarantee is chosen for the whole architecture, and the stronger requirement (here, the card stream) sets the floor. If the processing layer is asked to clean up duplicates, the cleanup mechanism is your design decision and must be explained as part of the architecture — you cannot hand the problem to an unspecified layer.

8.4.4 Student Questions and Answers

Real-world connection: the same trade-offs govern production messaging choices. Fraud detection and financial transaction pipelines demand strong guarantees — losing a single fraud signal can have an "undesirable effect," so operators accept the cost of exactly-once-style coordination. Web analytics pipelines, by contrast, tolerate a dropped event here and there and run the cheapest guarantee. The three questions are literally how practitioners pick: the business impact column decides.

8.5 Worked Case Study: Online Education System

The second sample problem flips the first: the class read the banking scenario and split it into stream plus batch; here they read an education platform and must realize that nothing needs real-time processing. The whole case turns on one observation — certificates are not events — and one rate judgment: the tallies run once a day, so the processing is batched.

8.5.1 Identifying the Events

The second sample problem is an online education system, and the class's first struggle was identifying what the events are. The instructor's example: suppose you are tracking the progress of a learner. A student completes a module, or a lab session — that is an event. Certificates are not events; they will not be generated as an event because they are the outcome of accumulated progress. So the event stream is progress events: module completions, lab session completions. You collate all of them and see how much the learner has progressed.

The class found this genuinely tricky, and the instructor's response was that it is not tricky — it is straightforward once you notice the scheduling. This is about scheduled events, not per-second events. You do not track every minute or every second; you need to track every day.

8.5.2 Daily Tracking Means Scheduled Batches

Here is the reasoning that resolves the case: a two-month course has an expected daily pace — how much is expected to be completed each day, and how much has been completed. That daily progress is the event stream. But because you process daily, each day is a batch. When you are tracking every day, it is batch processing, full stop. The rate question settles it: you have plenty of time to process these messages, so nothing about the design is driven by real-time pressure.

The decision rule being applied is the same one from Section 8.1 — read the activities, then ask the rate question: how fast must the answer exist after the event? Here the answer to "how much has this learner progressed today" is needed once per day, not once per transaction. A daily need means a daily job: collect the day's module and lab completions, run the rollup, update the progress view. Nothing in the pipeline is waiting per-second, so no per-second machinery is justified.

8.5.3 The Architecture: Lambda Without the Streaming Layer

Because the streaming requirements are minimal, Kappa is ruled out — you cannot use it as a stream-only design when there is barely a stream. The architecture is Lambda architecture without the streaming layer. The instructor's exact formulation: "Lambda architecture without the streaming layer would be the architecture." Batch ingestion, batch views of learner progress, and a serving side for certificates and progress reports.

8.5.4 At-Most-Once Delivery

For the message delivery semantics, the answer is at-most-once, and the student who said it first got a "fantastic." The justification is built from the three questions: if a message is lost, the business impact is tiny — one progress event missing from a daily rollup barely matters; the data tolerance is high; and long-term storage needs are modest. The instructor's supporting point: the hard semantics — at-least-once, exactly-once — exist because systems cannot cope with message volume. Here the messages arrive slowly and you have a lot of time to process them, so you do not need retries or deduplication machinery at all. Losing a message is acceptable, so at-most-once is correct.

8.5.5 Student Questions and Answers

Real-world connection: the pattern of "events, not outcomes" and "daily means batch" is everywhere. Learning management systems roll up learner progress nightly and issue certificates from accumulated totals; payroll systems treat each clock-in as an event but settle pay in a weekly batch; analytics platforms ingest website clicks in real time but compute the daily dashboard overnight. Naming the event stream correctly is the same skill as naming the real-time boundary — get that right and the architecture mostly draws itself.

8.6 Kafka Topics, Partitions, and Offsets

Before the partition-count formula of Section 8.7 can make sense, three terms must be absolutely clear — topic, partition, offset. Each one is simple alone; the confusion comes from who does what (the broker does nearly everything) and from picturing them as unrelated ideas when they are actually three views of one structure: a topic is the label, a partition is the file, an offset is the position in that file.

8.6.1 Topic

Three terms must be familiar before any partition math: topic, partition, offset. A topic is a kind of unique name for a data stream. The instructor's analogy is email: in an office you do not respond to every email — you ignore some, and the basis for deciding is the subject line. A topic is nothing but a subject line. Events come in, and you group them into different logical groups; each logical group is a topic.

In the banking example: if several events are connected with the amount of transactions, you can say that a transaction-volume-based event is a topic. If you want to analyze whether a message is a fraudulent transaction or not, you create a topic called "validate transaction." A producer might send different types of data; the topics group the events according to those types. Topics have unique names. The same message can land in multiple topics — a transaction can be looked at as "volume of purchase" and as "valid or fraudulent" at the same time. You can do that.

The labeling detail is easy to get wrong: the producer does not put the label. The broker puts the label, and according to that label the producer's messages are routed to the matching topic. Producers write messages to topics; consumers say what type of messages they are interested in; the broker maps them accordingly.

8.6.2 Partition

A topic will accumulate a large volume of messages, and a single broker may not be able to handle them. In that case you create partitions. A partition is a small independent portion of a topic. Concretely: each partition is an individual machine in the cluster — some messages are written to partition P1, some to P2, and so on. The broker takes care of which message goes into which partition; producers and consumers query the broker and it is all done by the broker.

Two design facts from the discussion: messages fill partitions in order — unless one partition is fully filled, the broker will not start putting messages into the next one. And the number of partitions is a design decision you specify explicitly. There is no runtime provision for the broker to decide the partition count; there may be a default, but you must mention the number you want.

8.6.3 Offset

Every partition has offset numbers. The instructor's picture: every partition is like an array, and the indexing of that array is the offset ID. When a new message arrives in a partition, it gets appended at the end of the existing offset IDs. Consumers read messages based on the offset ID: the consumer subscribes to a topic, the broker takes it to the partition, and from the partition it goes to the offset ID — that is how the message is fetched. The offset ID is nothing but the indexing in an array.

The consumer is not aware of offsets at all. That bookkeeping is handled internally by the broker: the broker tracks how far the consumer has read and updates the offset accordingly. A student asked whether the broker itself stores each consumer's last offset, and the answer refined the picture: that state information is not necessarily maintained by the broker alone — supporting systems exist for it, for example a resource manager like YARN.

Real-world: this topic-partition-offset model is exactly what lets Kafka scale a single logical stream across many machines while still giving consumers an ordered, resumable read position per partition. Because the offset is recorded, a consumer that crashes can restart from its saved position instead of losing or re-reading work — the mechanism underneath the delivery semantics of Section 8.4.

8.6.4 Who Coordinates: The Broker

Putting the flow together: the producer sends messages to topics; the broker labels and routes messages into partitions, appends at the end of the partition's offsets, and tracks each consumer's position; the consumer subscribes through the broker and fetches from partition offsets. The broker is the coordinator for all three jobs — routing, ordering, and position tracking — which is why the broker, not the producer or consumer, holds the design decisions in a Kafka deployment.

8.6.5 Student Questions and Answers

The partition discussion produced two questions about who controls partitions:

And the offset questions, in order:

Real-world connection: LinkedIn — the company that built Kafka — routes billions of messages a day through topic-partition structures exactly like this: hundreds of services producing and consuming through shared topics, each consumer group reading independent positions from shared partitions, and partition counts tuned to throughput targets. The topic-partition-offset model is not an exam curiosity; it is the load-bearing structure of the most widely deployed message platform in industry.

8.7 Estimating the Number of Partitions

With topics, partitions, and offsets in place, partition count becomes the session's first pure-arithmetic question — and the numerical shape of the exam. Three letters carry the whole thing: $T_t$, $T_p$, $T_c$. Everything else is unit conversion and division.

8.7.1 Mathematical Formulation

With topics, partitions, and offsets in place, the number of partitions is a formula-based question. The instructor's setup, given verbatim: "Tt is total system throughput. Tp means maximum throughput of a producer writing to a partition. TC is of a consumer reading from a partition." The number of partitions is

$$N_{\text{partitions}}=\max \left(\frac{T_t}{T_p},\frac{T_t}{T_c}\right)$$

where $T_t$ is the total system throughput (what the whole pipeline must move), $T_p$ is the maximum throughput of a single producer writing to a partition, and $T_c$ is the maximum throughput of a single consumer reading from a partition. The two ratios also have names: $T_t/T_p$ is the number of producers needed by the system, and $T_t/T_c$ is the number of consumers needed by the system. Because partitions must be able to absorb the producers' output and keep the consumers fed, you take the larger of the two ratios.

Unit hygiene is the whole game in these problems: system throughput is usually given per minute (GB per minute), while producer and consumer throughput are given per second (MB per second), so you must convert one side before dividing. The instructor accepts $1\text{GB}=10^3\text{MB}$ ("more precise is 1024, it doesn't matter"), so the numbers stay clean.

8.7.2 Worked Example: 10 GB per Minute System Throughput

Problem: system throughput $T_t=10$ GB per minute, and the maximum throughput of a producer writing to a partition is $T_p=10$ MB per second. Estimate the number of partitions — and the consumer throughput it implies.

8.7.3 Worked Example: 24 GB per Minute with 100 Producers

Problem: a streaming system with system throughput $T_t=24$ GB per minute. Throughput from the producers is 50 MB per second in total, spread over 100 producers. Find the expected number of partitions.

8.7.4 Student Questions and Answers

Real-world connection: partition-count planning is a standard capacity exercise in production Kafka deployments — operators estimate expected message rate and per-partition throughput, then size the topic. Too few partitions caps parallelism (one consumer per partition, so a topic cannot be read faster than the number of partitions allows); too many adds file-handle and bookkeeping cost. The exam formula is the same trade-off, stripped to its arithmetic.

8.8 Storage Requirements for the Partitioned System

The partition count from Section 8.7 does not just size the parallelism — it sizes the disks. Once $N$ is known, storage falls out of a three-factor product: how many partitions, how many messages each holds, and how big each message is.

8.8.1 Mathematical Formulation

Once the partition count is known, the storage question follows. You know the number of partitions $N$. Assume each partition can store $n$ messages, and each message is on the order of $m$ kilobytes. Then the storage requirement is

$$\text{Storage}=N\times n\times m$$

The professor's verbal description: "this is nothing but 800 times n times m." To use it you need two facts: how many messages each partition holds, and how big each message is. Message sizes are assumed to be small — the instructor explicitly says each message is on the order of some small $m$ KB, because stream messages are lightweight, not megabytes. The $n$ per partition is the offset ID idea in action: a partition is the array, and $n$ is how many entries it can store.

8.8.2 Worked Example: 800 Partitions

In the 24 GB per minute problem the partition count came out to $N=800$. So the storage requirement is

$$\text{Storage}=800\times n\times m\text{KB}$$

with the two parameters left symbolic because the problem does not fix them: $n$, how many messages one partition can store, and $m$ KB, the size of a single message. The exam variant flips the missing pieces — sometimes the question gives you message size and retention and asks you to size the cluster's disks.

A student raised a sharp question during this discussion: if the system can process 400 MB per second and the producers only generate 50 MB per second total — the processing throughput is eight times the production rate — why store anything at all? The answer: storage is not about catching up; it is the partitions themselves. The messages must live in the partitions regardless of how fast they are consumed.

8.8.3 Student Questions and Answers

Real-world connection: this multiplication is what capacity planning looks like in production. Kafka operators estimate messages per second and average message size, multiply by the retention window to get per-partition disk need, multiply by the partition count — then order that many gigabytes of cluster storage. The "why store at all" question also answers itself in industry: consumers are often offline or batched (hourly jobs, re-runs, backfills), and the partitions must hold the data while those consumers catch up.

8.9 Security in the Generalized Architecture

A security question framed at the level of the generalized architecture is independent of the problem statement — the instructor says it has nothing to do with the use case; it is general. That is the liberating part of this topic: the same answer template works for banking, education, or any other scenario, because it is about the tiers, not the business.

8.9.1 What Security Means Here

A security question framed at the level of the generalized architecture is independent of the problem statement — the instructor says it has nothing to do with the use case; it is general. Security here means identification and authorization: how secure the connections are between the tiers. The concrete checklist: the link between the collection tier and the data flow layer is connected via a secured connection only after authorization, and the analysis tier likewise connects to the data flow layer only after authorization. Protocols like TLS or SSL are exactly the vocabulary expected — a student asked "security means we have to tell about TLS or SSL or something of that sort, right?" and the answer was yes.

The check should also cover data at rest — messages stored in the data flow layer or in storage should be protected when written, not only while moving between tiers. In the exam answer, the two or three sentences that matter are: each tier-to-tier link is authorized, then secured (TLS/SSL), and stored data is protected as well.

8.9.2 The Touch Points and Fault Tolerance

The second half of security is the touch points between layers. Walk the architecture from left to right: collection tier → data flow layer (where the Kafka broker sits) → processing tier. For each link, ask what happens if the link breaks. What happens if the link between the collection tier and the data flow layer is broken? What happens if the link between the data flow layer and the processing layer is broken? You list three or four aspects related to fault tolerance — the aspects that keep the pipeline alive when a tier or a link fails. That is what a security question is really asking at the generalized level: not firewall trivia, but protected connections and resilient touch points.

The pattern to state in the answer: for every seam in the architecture, name the failure, name the resilience mechanism that absorbs it, and show that the flow resumes — that is the fault-tolerance story the question wants.

8.9.3 Student Questions and Answers

Real-world connection: production data platforms enforce exactly this pattern — Kafka clusters run TLS for in-flight encryption, authenticate clients with certificates, authorize produce/consume per topic, and replicate partitions across brokers so a single broker or disk failure does not stop the flow. When a company's security review asks "what happens if this link breaks?", the answer is the same touch-point walk taught here.

8.10 Message Writing Mechanisms: Ingestion Rate versus Processing Rate

The last concept of the paper — and the instructor's own exam intel says it is in the midterm syllabus — gives you rates and asks what kind of queue or writing scheme the data flow layer needs. It is a two-number problem: ingestion rate versus processing rate, one ratio, and a decision.

8.10.1 The Decision Rule: Compare the Rates

The message writing mechanism question gives you rates and asks what kind of queue or writing scheme your data flow layer needs. The model: collection tier → data flow layer → processing tier, with ingestion rate $r_i$ (rate of injection) and processing rate $r_p$ (rate of processing). The first move is always the same: compute $r_i/r_p$. That ratio decides whether the data flow layer is fine as it is, needs a bigger queue, or forces you to scale the processing layer.

8.10.2 Mathematical Formulation

Let $r_i$ be the rate at which messages are ingested into the data flow layer and $r_p$ the rate at which the processing layer consumes them. Compute

$$\frac{r_i}{r_p}$$

• If $r_i/r_p\le 1$: ingestion can be absorbed as it arrives, and the data flow layer can be a simple in-memory message queue with no special requirements.
• If $r_i/r_p>1$: messages accumulate. The backlog grows at $r_i-r_p$ per unit time, and over a refresh window of duration $\Delta T$ the queue must hold

$$\text{Backlog stored}=(r_i-r_p)\times \Delta T$$

messages. Then choose: scale the processing layer horizontally so its capacity reaches $r_i$, or switch the writing mechanism to batches — the choice depends on how big the backlog is and whether processing capacity is the real constraint.

The refresh rate deserves a definition: it is the rate at which the data flow layer is cleaned up, or the rate at which the system starts the process again — you run the application for some time (say ten minutes), stop, and start again. Over a 30-minute refresh window you accumulate all incoming messages, so the queue must be sized for the whole window, not for one second.

8.10.3 Example: Ratio Below One

First example: $r_i/r_p=0.99$. That means $r_i=0.99r_p$ — ingestion is 99 percent of the processing rate, so processing is on par with ingestion and can handle the load. In this case the data flow tier can be dropped to a simple in-memory message queue with no special requirements.

8.10.4 Example: Ratio of Twelve

Second example: $r_i=120$ MB per hour, $r_p=10$ MB per hour. The ratio is 12: by the time one message is processed, twelve messages have been ingested — eleven messages are waiting to be stored. If the rates are per hour, eleven messages an hour is nothing; a simple message queue handles it. But notice what happens if the same ratio runs per minute or per second: eleven messages per minute means the data flow layer fills up quickly. You cannot just use an in-memory data store — direct writing will not work — and the fix is to scale up the processing layer, not the queue.

8.10.5 Worked Problem: 10 MB/s against 1 MB/s with a 30-Minute Refresh

The main worked problem: a streaming architecture with ingestion rate $r_i=10$ MB per second, processing speed $r_p=1$ MB per second, and a refresh rate of every 30 minutes. Suggest a suitable message writing mechanism.

8.10.6 Horizontal Scaling versus Batching

The instructor then showed the second lever: batching. If the backlog estimate had come out small — he used 162 messages as the hypothetical — you would not scale anything; you would change the writing mechanism to batch processing, sending messages in batches of ten instead of event-by-event.

But there is a condition that rules batching out. If the processing capacity itself is not there — you start with one single producer, one single processor — then the notion of sending batches is completely ruled out, because there is no capacity waiting for the batch. Batching is a way to amortize existing processing capacity; it cannot create capacity that does not exist. That is the trap to avoid in the exam answer: choose horizontal scaling when the ratio is large and capacity is the constraint, choose batching only when capacity exists and the backlog is small.

8.10.7 Student Questions and Answers

Real-world connection: this is the same arithmetic used to size real streaming deployments. When a team deploys Kafka to move events into Spark or Flink, they compare ingest throughput against processing throughput, set the refresh/checkpoint window, and decide between scaling the processor cluster and batching writes. Systems that skip this step find out the hard way — a queue that fills until write latency spikes, exactly the latency the worked problem wanted to avoid.

Exam Guidance Summary

• Session shape: a problem-solving session built around a sample question paper. Expect one scenario-based architecture design question and one message design question, in the same style.
• Architecture question marks: explaining the various components of the architecture carries 6 marks; non-functional requirements evaluation carries 3 marks; message management (delivery semantics) carries 3 marks.
• How to write the architecture answer: decide batch vs stream first, then Lambda vs Kappa. Underline the activities in the problem statement and map only those to components. Write each component as a subheading with use-case-specific content, never the generic story. If a requirement is unclear, state an assumption. If the question says draw, draw.
• Non-functional requirements: the three to write about are low latency, high availability, and horizontal scalability — the three covered in class. For each, write what it means for the given use case (volume-driven cluster sizing, the two latencies in batch layer and streaming path, redundancy for availability).
• Message delivery semantics: answer the three questions — business impact of losing a message, data tolerance, long-term storage requirement — and justify the choice from them. One semantics per architecture, never one for batch and one for stream. Exactly-once and at-least-once with deduplication are both acceptable in the banking case if justified; the education case is at-most-once because messages are slow and loss is harmless.
• Numerical problems: number of partitions $N=\max (T_t/T_p,T_t/T_c)$, with units converted (GB per minute to MB per second, accepting $1\text{GB}=10^3\text{MB}$); storage $=N\times n\times m$; message writing mechanism via the $r_i/r_p$ ratio with backlog $=(r_i-r_p)\times \Delta T$, choosing horizontal scaling of the processing layer (cluster of nodes, e.g. 10 nodes for a 10:1 ratio, 1000 nodes for the 30-minute case) or batching (groups of ten) only when processing capacity exists.
• Exam answer shape: each part of the paper rewards the reasoning, not the label — the mapping of activities to components, the justification behind the delivery semantic, and the unit-clean arithmetic behind each number. Write the reasoning in full and the marks follow the substance.
• What to focus on: message delivery semantics problems and architecture design — that is sufficient preparation. The shared presentations and the notes are enough; the earlier sessions that repeated the same scalable-data-systems content will be consolidated into one presentation placed on the course portal.
• Logistics: the mid-semester exam is an evening session next week; there is no class next week. The quiz is published and has no makeup component — complete it in time.

Key Industry Applications

• Banking alerts and fraud detection: credit card and debit card transaction alerts are processed in real time as messages are generated; fraud detection must be real time, at the moment the message is generated — the stream side of a Lambda architecture.
• Bank offer engine: insights from card-related spending history drive loan offers — batch analytics over historical transactions, delivered as batch views (database views over stored batch data).
• Kafka partitioning at scale: a single logical topic is split into partitions, each an independent machine in the cluster, with offset-based append and read — this is how one logical stream scales across machines in production.
• Broker as coordinator: the broker routes messages to topics and partitions, appends at the end of offsets, and tracks consumer read positions; offset state may live in supporting systems such as a YARN resource manager.
• Cluster sizing from volume: multiple brokers and consumer clusters are recommended from the volume of customers — the same numbers drive horizontal scalability and availability decisions.
• Secure tier connectivity: connections between collection tier, data flow layer, and processing tier are secured (TLS/SSL) and established only after authorization, with the tier-to-tier links considered as fault tolerance touch points.

The through-line across all six applications: the same vocabulary tested in the exam — batch versus stream, Lambda components, partitions and offsets, delivery semantics, rate-based sizing, and secured links — is the vocabulary used to build and operate production big data systems today.