Stream Processing: Architecture and Processing Strategies

4.1 Batch vs Stream Processing: The Four Dimensions

4.1.1 Data Scope, Data Size, Performance, and Analysis

We return to the comparison between batch processing versus stream processing that began in the previous session, and we revisit the differences between them against four dimensions. The advice is direct: write these down in your own words, because they matter from an examination standpoint.

The four dimensions are data scope, data size, performance, and analysis. Before reading the details, notice the shape of the argument: every dimension is really one question that batch and stream answer differently.

• Data scope decides which portion of the data is considered for processing. In batch processing the scope is the entire accumulated dataset; in stream processing it is the live events arriving right now. The textbook term for these two situations is bounded versus unbounded data: a batch input is bounded — it has a known, finite size, so the job knows when it has read everything — while a stream is unbounded and never "complete" in any meaningful way.
• Data size is about the size of the data in terms of memory. Batch systems handle large accumulated volumes; streaming systems work with smaller per-moment volumes that never fully materialize. The stream processor never holds "all the data" — only the events currently in front of it.
• Performance is essentially the time scale of the work: seconds and minutes for streaming, hours for batch windows. One way to turn batch into something closer to streaming is to shrink the chunk: instead of processing one day's worth of data every day, split it into fixed chunks of hours or minutes (we will see this micro-batch idea again in Section 4.5.4).
• Analysis refers to the nature of the analysis you can run: comprehensive, hindsight-style analysis on a full batch versus lightweight, in-moment analysis on each event. Batch can answer "what happened overall?"; stream answers "what is happening right now?"

The four dimensions, side by side:

The pair to remember is the one in the first and last rows: the bigger the scope you look at, the more complete the analysis you can run — and the longer you have to wait. That trade-off drives everything else in this session.

4.1.2 Why Batch Gives Accurate Results

The follow-up point matters: know not just what batch does, but why. Because the entire dataset is present at processing time, every event contributes to the outcome, so the result reflects the complete picture. A streaming system, by contrast, decides from a partial view, so its conclusions are provisional.

An everyday picture helps. Think of grading an exam. A teacher who waits until every answer sheet is submitted can report the class average with full confidence — every sheet contributes. A teacher who announces an average after the first five sheets have come in is giving a provisional number that will move as the rest arrive. Batch is the first teacher; stream is the second. The analogy breaks in one direction: a stream system is not graded "wrong" for being provisional — timeliness is its whole reason for existing, and in many use cases (an alert, a live dashboard) an early approximate answer is worth more than a late exact one.

Real-world connection. The contrast is visible in everyday systems. Weather forecasting runs batch jobs over accumulated observations to build and refresh prediction models, while the live temperature you see on a weather site is stream-processed from current readings. Banks run batch reconciliations overnight for exact end-of-day totals, while fraud alerts on a credit-card swipe are stream decisions made from the transactions seen so far. In both cases the same raw data feeds both styles — which is exactly why the architecture in Section 4.4 places both kinds of processing in one pipeline.

4.2 The Rate Mismatch Problem

4.2.1 Problem Setup: 10 MB/s Ingestion, 6 MB/s Processing

The lecture moves from comparison to calculation with a worked problem. Consider a processing system where the data ingestion rate is 10 MB per second while the processing speed is 6 MB per second. You must recommend a processing strategy, knowing that the data analysis is performed hourly. Take a minute and try the problem before reading the walkthrough — problem solving is the whole point of this exercise, because the same style of problem reappears later.

Visualize the system before doing arithmetic. Data injection happens from somewhere unknown — this is the injection layer, where data is collected. Somewhere else, processing happens — this is the processing layer. The two operate at different speeds. The ingestion rate $R_i=10$ MB/s is the data speed; the processing rate $R_p=6$ MB/s is the processing speed.

The first calculation is the ratio of the two:

$$\frac{R_p}{R_i}=\frac{6}{10}=0.6$$

So the processing speed is 60 percent of the ingestion rate. Put in messages: if 100 messages arrive in one second, the system can cater to only 60 of them; 40 are left out. In real life the processing layer is slower than the ingestion layer, so this leftover is not a special case — it is the bounded, expected situation. The question is what to do with the 40 that you cannot handle.

Visual intuition. Picture a bathtub. The tap pours in at 10 L/s (the ingestion rate), the drain lets out 6 L/s (the processing rate). The water level rises at the difference, 4 L/s, no matter how long you wait. The bathtub is the future data flow layer: its size decides how long the system can survive the imbalance before the water overflows. The professor's version of the same picture is the hose that swells when its end is plugged — the pipe fills and eventually bursts unless something gives. Both images say the same thing: an unabsorbed rate gap always shows up somewhere.

4.2.2 Worked Example: Hourly Backlog and Sizing

If every incoming message is processed and sent downstream as it arrives, and the processing speed is slower than the arrival speed, some messages will be lost. You must prevent that loss. The solution is an auxiliary storage where the leftover messages are parked until the processor can take them. This is the seed of the data flow layer, which we formalize in Section 4.4.

The answer structure is the same in both versions: capacity of the data flow layer = leftover volume over one refresh window. Depending on the scenario you might not need to store every message — sometimes losing some messages is acceptable — and we discuss exactly that in Section 4.3. But when the requirement says full retention, the answer is 14.4 GB.

4.2.3 The Data Flow Layer

The data flow layer is introduced to handle the mismatch between the processing speed and the data ingestion speed. You cannot afford to keep a situation where messages silently vanish. The layer sits between injection and processing, accepting messages as they come and feeding them to the processor at the processor's pace. It acts as the buffer that absorbs the difference between the two rates.

This exchange is worth sitting with. The first instinct is to assume a continuous transfer, but nothing in the problem forces that. The architect's job is to see the refresh cadence and exploit it: the system only needs to withstand one hour of imbalance, because the hourly refresh acts as a stopper. During that hour, injection runs continuously and processing runs continuously at 6 MB per second. When the source stops, the processor keeps running until the backlog is gone, and only then does injection restart.

The deeper lesson is architectural method: a single numerical problem motivates every component we add. It is not that somebody tells you the components exist; you derive why each one must exist from the numbers. Common sense is the most important tool here — architectures get proposed from common sense, not from reading books.

4.2.4 Worked Example: Clearing Time

The next number to compute is how long the system must sit without new input. The hourly backlog is 14,400 MB, and processing runs at 6 MB per second, so the clearing time $t$ is:

$$t=\frac{C}{R_p}=\frac{14,400}{6}=2,400\text{ seconds}$$

Divide by 60 to get minutes:

$$2,400÷60=40\text{ minutes}$$

The clearing time is part of the strategy, not an afterthought — if you cannot afford 40 minutes of idle injection, hourly refresh is the wrong design, and you need the continuous alternative below.

4.2.5 Continuous Streaming: Horizontal Scaling and Clusters

If the use case demands that injection never stops, there is a concept called continuous streaming: you should not stop. The only way to run continuously is to make the processing layer fast enough — it should be much, much faster than the ingestion rate. Suppose the processing speed is double the ingestion speed. Then nothing is left over, and the processor is idle for some time; for safety you can still keep a small data flow layer in between so that small imbalances do not stall anything.

How do you double the processing speed? You can have two processors. This is horizontal scaling, a concept from the first session: add more machines instead of making one machine faster. For continuous streaming you propose a cluster with two nodes, where one node takes half the load and the other node takes half the load, and a central mechanism does the merging of the results. This is where the cluster enters stream processing.

There is a cost to the cluster: dividing the data among processors adds latency, and merging results adds latency too. Someone has to coordinate all of this work, and that coordination is called orchestration. We look at orchestration in detail later; for now, the point is that a multi-layer processing system needs it, and the reason for every layer traces back to a rate.

4.2.6 Replication and Fault Tolerance

Once you have multiple nodes, you want the system to survive a node failure. You maintain a copy of the data — that is replication. If one node fails, the other node preserves the state information of whatever has been done so far, so no work is lost. This is fault tolerance: the cluster keeps functioning even when a member fails, because the state lives in more than one place.

Real-world connection. The mismatch this section quantifies is the daily reality of message brokers. LinkedIn's Kafka, built to carry the site's event traffic, uses a log-structured buffer so that producers can run at full speed even when consumers lag, with retention policies (for example, keeping 12 hours or the last 50 GB of events) acting as the "refresh window" sizing decision from this section. In flight-radar and ride-hailing systems, bursty ingestion with a fixed-capacity processor is exactly the 10-versus-6 situation: the queue size is computed from the peak mismatch, and the cluster is sized from the sustained rate — the same two numbers we just computed.

4.3 Sampling-Based Processing

4.3.1 Problem Setup: Sampling 10 Percent Per Minute

The next problem modifies the first one. In a streaming system, messages are to be processed as samples. The data ingestion rate $R_i$ is 10 MB per second and the processing rate $R_p$ is 6 MB per second, but now the sampling of the messages is done at the rate of 10 percent per minute: of all the messages produced in one minute, only 10 percent are taken for processing. Discuss the processing strategy, assuming a refresh rate of one hour.

The single modification from the previous problem: you cut down the 10 percent fraction into the compute. Everything else stays the same. Before reading on, try the calculation yourself — the professor explicitly said this material is important from an examination standpoint, because problems in this style will be used again.

4.3.2 Worked Example: Per-Minute Processing

One student's approach: schedule the processing every minute, and let the sampling happen every minute too. In one minute, 600 MB accumulates, and the processor takes 10 percent of it — 60 MB — which takes a few seconds to process. That is the right shape of answer, but the professor pushes it to the message level, and there the numbers come out differently.

The message size is not given, so we assume a size — say one message is 1 MB — to make the fraction concrete. Then 10 messages arrive per second and 600 per minute.

The processor runs every minute, and its runtime is only 10 seconds. Either calculation — 60 MB of 600 MB at 6 MB/s, or 60 messages of 600 at 6 messages/s — leaves the processor overwhelmingly ahead of demand.

4.3.3 The Leftover Problem Disappears

This is the key correction. The earlier calculation needed 14.4 GB of auxiliary storage because 40 percent of the incoming volume had to be retained. Sampling changes the arithmetic entirely: we only take 10 percent of the messages, so we never build a backlog at all. Common sense confirms it — we can process 60 percent of the traffic but only need 10 percent, so we are far ahead of the requirement.

The other 90 percent of the data is neither delayed nor stored. It is ignored, and the professor's conclusion is blunt: those messages are lost. This is a legitimate engineering decision — it only works where loss is affordable, which is exactly the situation the next scenario describes.

4.3.4 Weather Monitoring: A Sampling Use Case

This is why sampling is defensible in practice: data dynamics in the real world do not change that quickly. Two general principles follow:

• Depending on the use case, you can afford to lose some portion of the messages.
• It is not mandatory that every message generated and injected into the injection layer be processed.

The answer to "how much can we afford to lose?" is always "it depends on the use case." When no specific use case is given, you state your assumptions and make inferences like the ones just made.

4.3.5 Continuous Sampling Without a Refresh

One more student proposal rounds out the design space: keep the processor continuous. That means taking 1 MB per second and discarding 9 MB per second, continuously:

$$10\%\times 10\text{ MB/s}=1\text{ MB/s}$$

4.3.6 Starvation and Empirical Tuning

There is a name for the idle-processor situation: starvation — the process is ready, but messages are not coming. The fix is to go back and increase the sample size, from 10 percent to 20 percent, and see whether the analytics output changes. These studies are empirical in nature: empirical means you inspect and adopt. When you start you do not know the right mix, so you run experiments and fine-tune the strategy until it behaves. Sampling rates are not handed down; they are tuned.

Real-world connection. Sampling is standard practice wherever the full stream is unaffordable. Meteorological observation networks sample sensor feeds at intervals precisely because atmospheric change is slow; website analytics pipelines drop or down-sample raw click logs before heavy analysis; and distributed counters use sample-based estimation when exact counts are too expensive to compute in motion. The formal machinery behind this — reservoir sampling, which keeps a fixed-size random sample so that every element of the stream has the same chance of being included — is the textbook version of the professor's 10 percent, and it is what the "statistical approximation" chapters of the reference books build on. Where exactness is required instead, the opposite end of the design space — exactly-once delivery — is the topic of Section 4.4.7.

4.4 The Layered Streaming Architecture

The two problems above established why a data flow tier exists: its purpose is exactly to match the rates — to absorb the difference between the processing rate and the ingestion rate. With that motivation in hand, we now walk through the layers of a streaming architecture from left to right: collection, data flow, analysis, delivery, and the data stores attached beneath them.

Visual intuition — the pipeline at a glance. Draw five boxes in a row, left to right: collection → data flow → analysis → delivery, with a store drawn as a box hanging under the analysis tier. Messages enter at the left, are admitted by collection, held and paced by the data flow queue, computed on by the analysis tier, and finally handed to clients on the right; the store box below holds insights for slow clients and for machine learning. Every box exists because of one of the rate mismatches we computed in Sections 4.2 and 4.3 — that is the whole design rule of this architecture.

4.4.1 Collection Tier

The collection system uses different application program interfaces — TCP protocol, HTTP — and accepts different file formats such as JSON files and Avro files. Avro is the structured data format developed for this exact purpose — a binary format with an embedded schema that different systems can read without sharing code; JSON is the plain-text alternative that is easy to read and debug. Both are portable formats whose job is to let many different producers and consumers exchange messages without caring about each other's internals. The collection tier's job stops at admission: once a message is inside, it is handed over to the next tier.

4.4.2 Data Flow Tier

From the collection tier, messages are fed into the data flow tier. It acts as a message queue: messages are appended into the queue and retrieved in first-in, first-out fashion. The separation between the collection tier and the processing tier exists because the rates at which these systems work are different — exactly the imbalance quantified in Section 4.2. If one system cannot cope with the other, the intermediate layer takes responsibility for accepting messages from the collection tier and providing them to the processing tier.

You must specify how much capacity the intermediate layer requires, and that number comes from the computation performed earlier: capacity is the leftover rate times the refresh window:

$$C=(R_i-R_p)\times t$$

where $C$ is the required capacity of the data flow layer, $R_i$ is the ingestion rate, $R_p$ is the processing rate, and $t$ is the length of the refresh window over which we must survive the imbalance. The professor stated this rule numerically in Section 4.2 (14.4 GB from $4$ MB/s $\times 3600$ s); this symbolic form is the general statement of the same rule, and it checks dimensionally: $(\text{MB/s}-\text{MB/s})\times \text{s}=\text{MB}$.

The buffering-layer framing is correct: the size of the buffer depends on the processing speed and the injection speed.

4.4.3 Analysis Tier and the Data Locality Principle

The third tier is the analysis tier, where processing happens. It works on the data locality principle: your application moves towards the data.

Processing can be done with different frameworks: Storm, Spark, Kafka, and others. But one framework cannot do everything in isolation, and you best stick to one of these things for the core pipeline. The same holds for databases: you can use a variety of them, but the architecture does not depend on a single one. The analysis tier performs filtering, aggregation, and insight generation on the messages as they move.

4.4.4 Delivery Tier

The delivery tier is nothing but the downstream. Once the data is processed, who captures it? Streaming clients — client applications sitting on the right-hand side of the entire flow. Those clients can be built with WebSockets, JavaScript, HTML, and similar technologies, or they can be documents being rendered somewhere. Each streaming client is itself an application with its own pace.

Real-world: the delivery app example is the mental model — a restaurant like a Swiggy delivery partner receives the order and delivers it without knowing anything else about you. The analytics tier renders messages to the downstream without worrying about who is consuming them or how. Because the tiers are decoupled this way, somebody must orchestrate the whole flow, and that orchestration role is what connects delivery back to the coordination we met in Section 4.2.

4.4.5 Storage: In-Memory Store and Long-Term Storage

Storage appears in two places. First, an in-memory data store right after the analysis tier: insights are placed there so that clients can pick them up. Second, optionally, a long-term storage layer. It exists because not everyone takes the data the moment it is produced — client applications have their own speed of collecting data. If clients are not collecting, you need somewhere to store. The in-memory store exists for the same reason as the data flow layer: to beat the mismatch between the consumption layer and the processing layer. If the rate of consumption and the rate of processing are out of step, even while both sides are actively running, the store absorbs the lag.

So there are three rates in the architecture: the rate of ingestion, the rate of processing, and the rate of consumption. Each mismatch between neighbors is absorbed by a layer sitting between them.

The reason the long-term store is not a relational database is often misunderstood. It is not the real-time expectation that rules out SQL or Oracle — it is the variety of the data. The data you are dealing with are messages, and you cannot expect any structure from them, so you do not limit yourself to a schema-driven database. MongoDB and Cassandra are the natural fits: they store artifacts of whatever shape, produced by filtering and aggregation. A relational table demands fixed columns in advance; a streamed message is whatever shape it is, so the store must accept documents of any structure — which is exactly what these stores do.

That is the second reason persistence exists — the enrichment of the stream with machine-learned knowledge: the stored insights become the training set for a model that is then put back into the live path.

4.4.6 Serialization in the Streaming Pipeline

Serialization appears when the in-memory store holds objects. Because we come from a Java background, the immediate intuition is that serialization means the data is being manipulated or persisted for transaction management. The correction: we do not manipulate the data at all — it is written once and read, never altered. Serialization exists for portability and speed when multiple systems in the downstream consume the data, and that is why JSON and adjacent formats became popular. Speed is part of it too: a portable, fast serialization format keeps the in-motion pipeline moving.

4.4.7 Message Delivery Semantics

One more concept belongs to the data flow tier: message delivery semantics — how you want to deliver the messages to the processing layer. Do you want every message delivered, or are samples enough?

The guarantee commonly cited is at least once semantics: every message that should be delivered is delivered at least once, so the processing layer never silently misses an event it was supposed to see. The cost is visible in the name — "at least once" allows duplicates, and the consumer must be written to absorb them (an idempotent operation: performing it twice has the same effect as performing it once). The sampling discussion of Section 4.3 is the flip side: when the use case tolerates loss, you relax the semantics deliberately and drop messages on purpose.

Real-world connection. This five-tier shape is the actual skeleton of real streaming stacks. Twitter-scale pipelines run a Kafka broker as the data flow tier between ingestion and Storm/Spark processors, and delivery to browsers happens over WebSockets and server-sent events; the Meetup RSVP pipeline in the course's primary reference is exactly this architecture with JSON events, a messaging tier, an analysis tier, and a publish-subscribe delivery channel. The same pipeline shape also explains why storage choices split: MongoDB and Cassandra for the schema-less long-term store, Redis-class in-memory stores for the fast insight cache — the two stores named in this section map to those two roles.

4.5 Lambda and Kappa Architectures

4.5.1 Lambda Architecture

Why do these architectures exist? They provide the provision for either stream processing or batch processing, or both. Batch processing means you do not process events as they arrive; you accumulate the messages and process them as a single batch. Stream processing means messages are processed as they come in. Lambda architecture carries both arms: in the upper arm a batch layer and a serving layer; in the lower arm a speed layer. You have provision for batch processing and stream processing at the same time.

Visual intuition. Draw a Y-shaped flow. The incoming stream reaches a fork: the upper branch feeds a batch layer box (with a serving layer box after it) that produces indexed views, the lower branch feeds a speed layer box that produces real-time views, and both arms converge on the user's query, which receives the merged answer. The upper arm moves slowly — hours — and the lower arm moves instantly; the query waits for neither, because each arm answers the part of the question it owns.

The delegation logic follows the work being done. When the problem involves machine-learning-type activities, some use cases require the scheduling of events at periodic intervals — analyze the data every three hours or every four hours — and those fall to batch processing. Simultaneously, you may need to enrich each event and predict something from it in real time, and those fall to the streaming layer. When the use case demands both aspects, you propose Lambda architecture and delegate messages to the speed layer and to the batch layer appropriately.

4.5.2 Why Both Layers Receive the Data

New data goes to both layers. The batch layer collects until a sizable number of messages has formed, then processes the whole batch; the speed layer filters and aggregates in real time and sends each message downstream immediately. Why both, and not one of them? Because batch processing yields more accurate insights. A machine learning model cannot be developed from the speed layer alone — you need to accumulate messages to a certain size, a batch level, and use the entire batch for model development.

The weather example makes it concrete. Seeing the current temperature of a locality is streaming — you always get the current reading. Predicting tomorrow's temperature or the day after is analytics: the data accumulates on the batch layer, you run analytics on top of it, and you get the prediction. Persistence is the difference: the batch layer stores, the speed layer serves whoever needs it now. And it is not only persistence — the batch layer handles the scheduling of events. If a use case requires scheduling, such as processing jobs every three or four hours, that periodic work belongs to the batch side.

Sending the same messages to both layers raises the replication question. The tasks performed in batch and stream may be similar, but the accuracy of the information differs and the details of the inferences differ: batch processing gives a comprehensive view, while stream processing just enriches the event. The two layers are not copies of each other; they answer different questions from the same raw feed.

The serving layer, in the upper arm, creates multiple batch views from the previous batch, and searching speed comes through an indexing mechanism on those views.

4.5.3 Kappa Architecture and Its Limitation

There are scenarios where you do not need the batch layer at all — that is called Kappa architecture. Kappa is nothing but Lambda with the entire batch layer arm removed: it keeps the serving layer and the speed layer, and stream processing alone drives the pipeline. Its advantage is simplicity: one pipeline, no dual bookkeeping. Its limitation follows directly: because it has no batch layer, you cannot get the comprehensive, accurate insights that batch produces; you only get real-time enrichment of events. Every architecture choice is a trade of these pros and cons, and you should be able to state both sides for each architecture.

The one-line rule: Lambda when the use case needs both accurate accumulated insights and real-time enrichment; Kappa when real-time enrichment alone is enough.

4.5.4 Evolving Between Architectures

The question of whether the choice is fixed comes up twice, so the answer is worth restating. It is better to decide from the use case itself whether you need a batch layer — that decision should come from the problem, not from convenience. But the diagrams look static and the reality is not: architectures evolve as a function of time, regularly; nothing is written on stone. If a batch requirement appears later, you extend the Kappa architecture into a Lambda architecture — extensibility is a design principle for exactly this reason.

The frameworks help with the transition. Spark streaming, for example, is by default based on micro batch: it is neither stream nor batch until you say otherwise, but you can specify the size of the micro batch in the configuration files, and depending on the size, the framework switches toward batch or toward stream. So the platforms themselves provide a mix of both, selected by configuration — nothing comes free, and you specify your configuration to get the behavior you need.

Real-world connection. Lambda architecture in its canonical form is exactly the design used by companies that serve both live and historical views of the same events: LinkedIn-style analytics with Kafka streams feeding both a speed layer (live dashboards) and nightly batch jobs (reporting), and weather services whose current-temperature feeds run through a speed layer while their forecast models are rebuilt from accumulated data on the batch side. The books' reference design even documents the failure-tolerance argument: because the serving layer is rebuilt from the batch views, a mistake in the speed layer is automatically corrected when the next batch arrives — the same "batch corrects stream" idea that closes this section.

4.6 Stream Data Properties and Real-Time Systems

4.6.1 Stream Data Properties

What are the properties of stream data? It is loosely structured: you do not have schema information defined for it. It arrives from various sources, and it can be structured, semi-structured, or unstructured, and so on.

A concept that matters when filtering and analyzing such data is cardinality — the number of unique values a field takes. When you want to filter down and show analyses, the distribution of unique values shapes the work, and that is why the properties of the data drive the storage and processing choices made in Section 4.4.

4.6.2 Real-Time Systems: Frequency of Events

One of the major challenges of real-time systems is computing the frequency of occurrence of events. As events stream in, how do you find accurately how many of this particular type have come and how many of that particular type have come? With a batch you can always recount; in motion, you have no provision to compute those numbers accurately while the stream keeps moving. Counting and histogramming in-flight data, without pausing the flow, is precisely the hard part that streaming analytics exists to solve.

Visual intuition. Picture a live histogram on a dashboard — event types on the horizontal axis, running counts on the vertical axis — with the bars growing as events stream in. In a batch world the histogram is drawn once from a fixed table, and you can redraw it any time. In a stream world the bars grow in front of you and the data that raised them is already gone; the histogram itself is the only copy of the counts. The challenge of Section 4.6.2 is that the bars must stay accurate while the data under them is unreadable a second after it arrives.

4.6.3 High Availability, Low Latency, and Scalability

The main things when it comes to real-time systems are high availability, low latency, and horizontal scalability. Horizontal scalability means adding more and more servers, or more memory, in parallel — you create a cluster with a set of nodes, and that cluster is your horizontal scaling, the same mechanism used for continuous streaming in Section 4.2.

Real-world connection. The frequency-counting challenge is daily work in observability platforms (counting error types per second across a microservice fleet), network monitoring (counting packets by protocol at 40 Gbps — a rate at which exact per-event counters are impossible), and social platforms (tracking engagement event counts per second). Cardinality estimation through HyperLogLog is built into analytics databases and distributed monitoring stacks, and the high-availability–latency–scalability trio is the standard checklist used to evaluate streaming platforms in industry comparisons — the same trio the next session will examine in detail.

Exam Guidance Summary

This lecture's examinable material, consolidated:

• Differentiation question: if asked to differentiate between batch processing and stream processing, answer along the four dimensions — data scope, data size, performance, and analysis — or no marks are given. Write the comparison in your own words.
• Rate problems: problems like the ingestion/processing mismatch will be used again. You must know the full computation: the ratio (6/10 = 0.6), the per-hour leftover (14,400 MB = 14.4 GB), the clearing time (40 minutes), and the sampling variant (10 percent per minute → 10 seconds of processing per minute). Exam note: "if you know how to solve, you will be able to do; if you don't know how to solve, you will not." Practise the chain with different numbers — change the ingestion rate, the processing rate, or the refresh window and recompute ratio, backlog, capacity, and clearing time end to end.
• Architecture design question: a scenario may be given and you may be asked to develop or design a suitable architecture. Keep the processing components in mind — collection, data flow, analysis, delivery, storage — and the rate logic that sizes them, then choose between Lambda (batch + speed layers) and Kappa (speed layer only) based on the use case. State the pros and cons of the architecture you choose.
• Assumptions: when no specific use case is given, state your inferences and assumptions, as done in the sampling problem — the message size assumption (1 MB), the sampling fraction, and the refresh window must be stated before the arithmetic begins.
• Sampling and loss: remember that depending on the use case you can afford to lose some messages, and not every injected message must be processed.
• Next-session material: high availability, low latency, and horizontal scalability are continued next time, and the next session opens by questioning the material from this one — so treat the rate problems, the four dimensions, and the five-tier architecture as review material for the coming session.

Key Industry Applications

The named tools and use cases from this lecture, collected in one place:

• Real-world: Storm, Spark, and Kafka are the named streaming frameworks; one framework cannot do everything in isolation, so a pipeline sticks to one core engine. Storm processes events one at a time for the lowest latency; Spark cuts the stream into micro-batches; Kafka is the log-based broker that holds and replays the stream.
• Real-world: Spark streaming works on micro batches by default, and the batch size is a configuration choice that moves the pipeline between batch-like and stream-like behavior — the mechanism by which a single platform can serve either side of the Lambda/Kappa choice.
• Real-world: MongoDB and Cassandra are named as the long-term storage engines for stream insights, preferred over schema-driven databases because message data has no reliable structure — the variety argument of Section 4.4 in concrete form.
• Real-world: JSON and similar portable formats became popular for serializing messages between downstream systems — portability and speed, not data manipulation, is the reason.
• Real-world: streaming clients are built with WebSockets, JavaScript, and HTML, and messages enter through TCP or HTTP APIs in formats such as JSON and Avro — the WebSockets choice gives a full-duplex, fault-tolerant delivery channel to client applications.
• Real-world: weather and meteorological monitoring is the motivating use case for sampling — atmospheric parameters change on the scale of minutes or hours, so sampling 0.1 percent per second still captures the behavior, and the same slow-dynamics logic justifies sampling in industrial sensor and telemetry pipelines.
• Real-world: delivery apps illustrate the decoupled delivery tier — the restaurant receives the order and delivers without knowing the customer — and the same decoupling is why the delivery tier can be built and scaled independently of the analysis tier.