Real-Time Systems and Stream Processing

We start from the data model comparison done earlier: traditional systems and streaming applications model data differently, and that difference matters because it decides everything downstream — how fast we respond, how we classify a system, and whether we can afford to lose a message. The next set of ideas builds directly on that: what makes a system "real time", why response time is one of the most important aspects of stream processing, and how batch processing differs from stream processing. These are not textbook abstractions; each category below was motivated in class by asking "what could you actually lose if a message is delayed or dropped?" — a credit card swipe, a cricket siren, a train signal, a chemical chamber reading.

3.1 Real-Time Systems: Hard, Soft, and Near

3.1.1 The Two Things That Decide Real-Time Performance

When we call this "real-time processing", two things are critical:

1. Data ingestion speed — the rate at which data arrives from the various sources. Sources (Twitter, sensors, card swipes) feed data in, and a mechanism must capture that data and send it downstream for processing.
2. Processing speed — the rate at which the processing engine handles those messages once they arrive.

Think of a funnel: the top (ingestion) pours messages in, the neck (processing) lets them out. If the neck is narrower than the top, the funnel backs up and latency grows until either the sources slow down or messages are dropped. A real-time system is a funnel sized so that the neck keeps pace with the top — or, if it cannot, the system has decided in advance how much spill it can tolerate.

Scope of the model: these two rates are the throughput view of latency. They describe steady-state behaviour — how fast messages flow, on average, through the pipeline. A system with a huge burst every few minutes may still be fine at the average rate but fail its deadline in the burst; that burst behaviour is exactly where the hard/soft/near classification (next subsections) bites.

3.1.2 Hard Real-Time Systems

Real-world note: hard real-time systems are almost always embedded — pacemakers, anti-lock brakes, engine control units, surgical robots. They are designed with strict worst-case guarantees (bounded processing time, bounded network delay), which is why the reference book lists them as a category that is well studied but outside the scope of a streaming-data course. When a use case is hard real-time, you leave stream processing entirely (see section 3.4).

3.1.3 Soft Real-Time Systems

Real-world: credit card transactions. Swipe a card and you immediately get a message confirming the transaction with the details and amount, plus a "call customer care if you did not make this purchase" prompt. The classification of this example generated a genuine debate in class — a debate worth preserving, because it is the first time the course's classification rule is exercised under pressure:

The resolution: even though the immediate notification feels instantaneous, the end-to-end business process (settlement) tolerates up to 24 hours of delay, so the whole system is soft real-time. The immediate swipe feedback is one part of the system; the classification follows the tolerance of the entire process, not the snappiest step. This is the moment that established the course rule: classify by the whole process's tolerance, not by the part that feels fast.

3.1.4 Near Real-Time Systems

The mental distinction to carry forward: traceability systems tolerate minutes; control systems do not. If a position update were being used to steer the vehicle, the same data would demand a different class (see sections 3.5 and 3.6).

3.1.5 The Latency-Tolerance Table

The textbook's classification table (table 1.1 of the reference) matches this exactly in the latency ranges and tolerance wording, and adds its own example choices: hard — pacemaker, anti-lock brakes; soft — airline reservation system, online stock quotes; near — video chat, home automation. Notice that the textbook itself calls airline reservation and online stock quotes soft real-time — the same call the professor makes in example 7 below. Where the professor's table says "milliseconds to microseconds" for hard, the textbook writes "microseconds to milliseconds": the same range, just read in the opposite direction.

3.1.6 Classifying Use Cases: Worked Examples

Each example below was worked through live in class, and the classification logic (not just the answer) is the lesson. Follow the reasoning line by line: latency scale → life risk → tolerance → class.

Comparison — the seven cases at a glance:

When to pick which: traceability with minutes of slack → near; any money loss on minutes-scale delay → soft; any life risk → hard.

3.1.7 Student Questions and Answers

The correction carried in this exchange is the moment the classification stopped being "life risk only" and became "loss type + latency". Hard is reserved for zero-tolerance loss; once money loss enters, the system is pushed out of hard into soft even when no one is at risk.

The two axes — tolerance and latency — are the entire decision apparatus. Life risk is not a third dimension; it is how the tolerance column is read (zero tolerance of delay because the consequence is life). Everything else is a consequence of these two axes.

3.1.8 The Classification Decision Process

Putting the worked examples together, the systematic process is:

1. What is the latency? What time scale are we looking at — milliseconds, seconds, or minutes?
2. Is there life risk involved? Zero tolerance (life threat) → hard.
3. If not hard, what is the tolerance for delay — high or low? Minutes-scale tolerance → near; seconds-scale → soft.

Recap and bridge. A system is hard, soft, or near real-time based on two axes only: the latency scale and the tolerance for delay, with life risk as the hard/not-hard gate. Keep the seven worked examples and the table in mind — the next section asks what stream processing is for in the first place (filtering and aggregation), which is the other half of the lecture's core picture.

Real-world connection. The classification is not classroom bookkeeping; it is how industry buys and builds systems. Autonomous-vehicle stacks budget fixed latencies per stage (perception, planning, actuation) and demand hard guarantees. Payment networks (Visa, Mastercard) run authorization in milliseconds but settle over days — soft real-time in exactly the professor's sense. Transit agencies track fleets with minutes-level freshness — near real-time. And industrial control systems (chemical plants, power grids) size their tolerance by the hazard of the substance in the pipe, exactly as example 5 shows. Whenever an engineer is asked "what happens if a message is late?", they are being asked to run this classification.

3.2 The Purpose of Stream Data Processing: Filtering and Aggregation

3.2.1 The Goal of Data Processing

When a sensor on a chemical chamber streams temperature and pressure messages, what are we actually trying to do? The class's first answer: capture and process each and every event as much as possible. But the deeper point — and the professor's framing for the whole course — is that stream data processing performs only two operations:

1. Filtering
2. Aggregation

3.2.2 Aggregation

Why the window matters. The five minutes are part of the aggregation: the summary is over exactly the events in that window, no more and no less. Change the window and you change the answer — a mean over the last five minutes is not the mean over the last hour, and a live dashboard hides that difference if the window is not labelled. The window is the aggregation's boundary condition, not an implementation detail.

3.2.3 Filtering

The filter and the count together are one operation: the filter selects a subset of the stream, and the count summarizes it. The lecture audio garbled the pressure unit (it sounded like "hertz", which is a frequency unit; pressure is measured in bar or kilopascal), but the unit does not matter for the idea — the condition "pressure about 700" is a threshold filter, and any unit plugs into the same predicate.

Where the filter lives. In practice the filter can sit at different tiers of the pipeline (at the source, in the analysis tier, or at the client), and filtering can be static (fixed at design time, like "only pressure readings") or dynamic (chosen at run time by the consumer, like a dashboard user picking a band of pressures). The lecture's point is simpler and more fundamental: whichever tier runs it, a filter is one of the only two things a stream processor is allowed to do.

3.2.4 Stream Processing Is Not a Control System

That is why filtering and aggregation are the only two operations: they are all you need to produce insight, and they are all you are allowed to do when you are not a control loop. A control system needs a third operation (actuate), which is why the moment a system starts steering — applying brakes, switching tracks — it leaves the streaming family and becomes a (possibly hard) real-time system (sections 3.4 and 3.5).

Recap and bridge. Stream processing selects (filter) and summarizes (aggregate) events into insight, and nothing more — because it generates insight for humans to act on, not actions of its own. Next, section 3.3 asks how the computational model of a streaming system differs from the input-process-output model you already know.

3.3 The Computational Model of Streaming Systems

3.3.1 Input, Process, Output

Every computational model you learned in programming rests on three building blocks: input, process, output. Streaming systems still need all three. What differs is the character of each:

• the input is heterogeneous (3.3.2),
• the process runs over events that arrive continuously rather than sitting in files,
• the output is a set of enriched events consumed by diverse clients (3.3.3).

3.3.2 Heterogeneous Input

Heterogeneous means "made of unlike parts": a temperature reading (a number), a tweet (text), a swipe (an ID plus a timestamp), a form submission (fields), a GPS fix (latitude and longitude). The ingestion layer receives all of these through the same funnel and must accept them without knowing their shape in advance — it is the producer, not the consumer, who decides what arrives. Contrast this with a classic database application, where the application defines its schema and refuses anything that does not fit; a streaming system has no such luxury, because the producers outnumber the system and answer to no one.

3.3.3 Enriched Events and Diverse Clients

Difference number two is on the output side. Once events are processed, the insights we share are called enriched events — raw events with computed insight attached.

These enriched events go to different clients in different formats. Some streaming clients expect a dashboard rendered on a mobile device; others on a laptop; others a data feed for further analysis. So: different types and formats at the ingestion layer, and different types and formats of requirement downstream. There is diversity in both directions — the diversity of the incoming data types and the diversity of the clientele (the nature of clients and the format they want).

Recap and bridge. The computational model keeps input-process-output, but the input is heterogeneous (many uncontrolled sources), the output is enriched events (raw events plus computed insight) served to diverse clients that consume at their own pace. Next, section 3.4 places this model in the real-time classification from 3.1: what kind of real-time system is a streaming system?

3.4 Streaming Systems versus Real-Time Systems

3.4.1 Stream Processing Operates in Non-Hard Real Time

Streaming data systems are expected to work in a non-hard real-time manner. The reference book states this as the very definition of a streaming data system: a non-hard real-time computation whose clients consume the data when they need it, not when it is produced. This triggered a clarifying question in class:

The picture: we take stream processing, "double-click" it, and inside we find a couple of options — soft real-time and near real-time. The difference between soft and near is blurred; the difference between hard and the rest is a wide opening. Hard real-time systems are a separate category from stream processing.

3.4.2 Can a Hard Real-Time System Be a Streaming System?

The reasoning, made explicit: hard real-time is defined by zero tolerance and potential loss of life, and its decisions are made from the enriched event itself (3.5.3). A streaming system, by contrast, generates insight for a human to act on — a human waiting is exactly what a hard real-time system cannot afford. "There is no streaming system which would wait on something to happen with potential loss of life" is the professor's one-line proof: if life depends on the message, no analytics layer may sit in the path.

3.5 Complex Event Processing

3.5.1 The CCTV Worked Example

The motivation came from a traffic-tracking scenario. We take snapshots of images at regular intervals — CCTV footage. Suppose the footage shows two people riding a bicycle crossing a location at 10 o'clock, and the two-wheeler moved out at about 100 kilometers per hour. In the next lane, after about five minutes, a police jeep with a siren crosses the same location — the CCTV footage of that arrives after five minutes.

The reference book describes CEP in the same terms: an approach, developed in the 1990s, for analyzing event streams by searching for certain patterns of events — the way a regular expression searches for patterns of characters in a string, CEP searches for patterns of events in a stream. The rules describing the pattern are kept long-term, the events flow past them continuously, and when a match fires, the engine emits a complex event carrying the details of the pattern it detected.

Real-world connection. CEP is the technique behind correlation-style analytics — separate, apparently unrelated events in a stream are correlated to infer something none of them says alone. Industrial deployments: credit-card fraud desks (a purchase in city A followed minutes later by one in city B), bank anti-money-laundering systems (two linked transactions within a short window), surveillance and safety systems (the CCTV case), and trading systems that detect order patterns. Named CEP engines in the field include Esper, IBM InfoSphere Streams, Apama, and TIBCO StreamBase.

3.5.2 CEP Runs in Seconds to Milliseconds

What matters most for CEP is the time scale you operate in. In the CCTV example the time scale is seconds and milliseconds — not minutes, because by the time minutes pass, the people will have run away. A time scale of seconds/milliseconds with no life risk puts CEP in the soft real-time category.

3.5.3 Student Questions and Answers

The self-driving car case was used to draw the line between insight and action:

Recap and bridge. CEP is correlation across time — combining events into inferences none of them carries alone — and it runs at a seconds-scale, soft real-time pace, because its value dies with delay. The action test from this section now carries into section 3.6, where the train tracking case study applies it to decide soft versus near.

3.6 The Train Tracking Case Study

3.6.1 The Train Event

The running example: track the geographic location of a train for decision making. What is the event? The event carries:

• timestamp — when the reading was taken
• longitude — the east–west coordinate of the train
• latitude — the north–south coordinate of the train
• which station it crossed recently
• the speed of the train

That event is the message flowing into the processing system.

3.6.2 Soft or Near? The Minutes Question

Classification process applied: the latency is not milliseconds — there is some bandwidth (time slack) — so it comes under the second category. But is it soft or near?

The reasoning chain: the event can tolerate minutes (a stale position is not dangerous by itself), but the decision cannot — a routing or signaling decision must reach the train while there is still time for the train's inertia to respond. A train is a big machine: braking and changing direction take seconds of physics, so the decision must arrive well before the deadline. And the network is not one train: Rajdhani, super fast, and local trains share the track, so the system must be sized to the slowest-reacting, fastest-moving worst case (the "least common denominator" of all the trains). The direct signal path — control to the train itself — forbids minutes.

The life-risk objection fails on the action test from 3.5.3: collision safety is handled by the existing railway infrastructure (signaling, block sections, interlocking) — not by the stream processing system. The stream system enriches and monitors train messages for a human controller; it does not directly actuate the brakes. Compare the self-driving car, where the enriched event is the brake command: there the analysis feeds the next movement directly, and the system must be hard. Here, you do not give instant input based on the processing of the event.

So the classification: the train tracking system is soft real-time — a student first said near real-time; the resolution is that the reaction time of the train plus the direct signal path pulls it back from minutes.

3.6.3 Driverless Trains

A further nuance on the same example:

Real-world connection. Metro systems running driverless operation — the CBTC-based metros mentioned in the discussion, communications-based train control, where trains continuously communicate their position to a control center and receive movement authority over radio instead of relying on fixed signals alone — operate closer to hard real-time than classic driver-controlled rail, because their enriched events feed automatic train protection directly. The same technology family, from classic block signaling to CBTC, is exactly the "existing infrastructure" that keeps the monitored-train case soft in 3.6.2.

Recap and bridge. The train case study shows the full classification machine working on one example: event fields (3.6.1), the minutes question (3.6.2), and the driverless flip (3.6.3) — ending in soft real-time for monitored trains and hard real-time for autonomous ones. Section 3.7 now leaves real-time classification behind and compares the batch world: what changes when you stop processing events as they arrive and collect them instead?

3.7 Batch Processing

3.7.1 What Batch Processing Is

3.7.2 The Six-Hour Batch Walkthrough

3.7.3 The Bucket Analogy: Time-Based and Count-Based Windows

The water-bucket analogy explains the two ways to decide what goes into a batch. Imagine putting a bucket into a stream of water:

• Water-level way (count-based): collect water until it reaches a certain level inside the bucket — the batch fills up to a fixed number of events.
• Stopwatch way (time-based): start the stopwatch when the bucket enters the water, and remove it when the stopwatch says the time is up — the batch collects everything that arrives in a fixed time span.

Count-based windows shine when event rates are steady (every batch has a predictable size); time-based windows shine when the metric is "per hour" regardless of volume. The catch with a pure count window is that a quiet period fills it slowly — the batch period becomes variable, and a very quiet period may delay insights past their usefulness.

3.7.4 Batch Is Always Scheduled

The schedule is the contract: the collection window (how much data), the firing time (when processing runs), and the output generation are all fixed in advance. Nothing about batch is reactive; the engine runs because the clock says so, not because data arrived.

3.7.5 Student Questions and Answers

The point of the correction: "scheduled" does not mean "monthly". The schedule is whatever the business needs — daily for daily-wage workers, monthly for salaried staff. What makes payroll batch is that it runs on a fixed schedule over accumulated events, not that the period is long.

Two triggers, one conclusion: time-based (the stopwatch way) and count-based (the water-level way) both produce batches; the professor's "special window" is the count trigger. This is a favorite exam trap: students answer "batch is time-based" and lose the count-based special case.

Recap and bridge. Batch collects events into groups (time- or count-triggered), stores them, and processes them on a fixed schedule — deeper analysis, later answers. Section 3.8 turns the batch-versus-stream contrast into a precise four-way comparison.

3.8 Batch versus Stream Processing: Four Differentiators

The comparison was summarized in four parameters. Stream processing: continuous input, data processed in small time scale, no storage required (messages are not accumulated — data is processed as and when the system is available). Frameworks like Storm and Spark are basically micro-batch processing in this spectrum. Examples of stream processing: ATM bank transactions; processing happens in fixed timelines with a data collection window and a data eviction window.

3.8.1 Data Scope

Data scope is how much of the data you process: are you processing the entire dataset, or a sample of it? If you process most of the data, that is batch processing. Stream processing deals with a small fraction of the data — the fraction considered in the current analysis is very small, because you do not have the capacity to process everything.

3.8.2 Data Size

Data size is the window size or batch size — within the batch or window, how many events do you have? It is related to but different from data scope: scope is about how much of the main stream you collect (the sample size); data size is how many events sit inside the current window or batch.

3.8.3 Time Scale and Depth of Analysis

The performance dimension is the time scale / latency: batch runs on a schedule (hours, daily), stream processes in small time units (seconds-to-milliseconds per event). And the depth of analysis differs: batch processing does complex analysis, and the analysis is more comprehensive — why? Because you have a lot of data. Batch processing is definitely more accurate and more comprehensive than the insights generated by stream processing, because stream processing only does enrichment of events; it cannot do detailed analysis since it has no mechanism to store all the data.

Comparison — batch vs stream on the four axes:

When to pick which: need deep analysis of everything with answers on a schedule → batch; need just-in-time insight with small analysis over what passes → stream.

3.8.4 Student Questions and Answers

The correction preserved here is subtle and easy to flub: the rolling window does eventually scan every message (the lamp does visit the whole stream), so "stream misses data" is the wrong objection. What is true is that each individual analysis covers only the tiny fraction inside the current window — the lamp's circle, not the river. The scope of the current analysis is small; the stream itself is not fully processed in any single pass.

Recap and bridge. Batch and stream differ on data scope, data size, time scale, and depth of analysis — with storage as the enabling difference underneath (batch stores and accumulates, stream processes as it arrives). Section 3.9 names the industrial frameworks that implement this spectrum, and section 3.10 closes the loop by asking why you would choose stream processing at all.

3.9 Processing Frameworks

3.9.1 Storm, Spark, and Kafka

A quick orientation on what each does, since the assignment asks you to explore them:

• Apache Storm — a stream processing framework that handles data tuple at a time (tuple = one message, one event), designed for low-latency real-time processing. Its mental model is a topology: spouts emit events, bolts transform or aggregate them, and events flow through the topology continuously.
• Apache Spark / Spark Streaming — treats the stream as a series of small batches. Spark Streaming divides the incoming stream into micro-batches over short intervals (down to sub-second) and runs a batch job on each — which is why the lecture places Spark on the micro-batch end of the batch-stream spectrum.
• Kafka — the distributed log / messaging platform at the front of the pipeline: producers write events to a durable log, consumers read from it at their own pace. Kafka is the decoupling layer that lets producers and consumers of section 3.3.3 operate on their own time scales.

The industry pattern that emerges: Kafka holds the stream, Storm or Spark Streaming processes it, and downstream systems consume the enriched results. The reference texts confirm this division — the analytics frameworks named there (Apache Storm, Spark Streaming, Samza, Kafka Streams, Flink) sit on top of exactly this kind of log, and the lecture's list of Storm, Spark, and Kafka is the standard trio for this pipeline shape.

3.9.2 The Exploration Assignment

As part of assignments, you will be given platforms like these and asked to explore them and generate insights. You will look at what the platform does and evaluate it — on what basis you evaluate it will be explained, but you need to explore the different data architectures based on the rubric given.

Recap and bridge. Storm, Spark, and Kafka are the industrial trio: Kafka moves the stream, Storm and Spark Streaming process it (with micro-batching), and downstream consumers receive enriched events. Section 3.10 steps back from the how to the why: four reasons a designer chooses stream processing over batch.

3.10 Why Stream Processing: Four Reasons

3.10.1 The Nature of Data: A Never-Ending Stream

Reason one: the nature of the data itself. Events arrive as a never-ending stream. With batch processing, some events fall into one batch and other events into another batch — so you may lose collective intelligence: by combining events across batches you could get a more accurate insight than any single batch gives.

3.10.2 Time Series Data

Reason two: time series data. If your input data is time series data, stream processing is appropriate.

What is time series data? The stock market is the classic example: for a particular stock, the price is a function of every second, millisecond, and minute. In mathematical terms:

$$p=f(t)$$

where $p$ is the stock price and $t$ is time measured on the scale of milliseconds, seconds, or minutes. The professor described the relationship in words ("the price of the stock is a function of every second, millisecond and minute"), and $p=f(t)$ is the standard way to write exactly that: price is a value that changes continuously over time, and that continuous dependence is what makes it time series data. The function $f$ maps each instant $t$ to a price $p$; there is a price at every $t$, and consecutive prices are related — the price at $t+1$ is never far from the price at $t$.

3.10.3 Storage and Infrastructure Limits

Reason three: hardware and other limitations to store events. If you have lesser infrastructure, and only approximate insights are enough for the enrichment of these events, then stream processing is the right choice — you process what you can, when it arrives, instead of storing everything.

3.10.4 The Data Source

Reason four: the source of the data. If the data is generated by IoT devices or other monitoring systems, you probably go for stream processing.

Sensors, smart meters, vehicle telemetry, wearables, and industrial monitors are built to emit continuously and cheaply — a Raspberry-Pi-class gateway emits readings by the second, not by the batch. Two properties push IoT streams toward stream processing: the sheer event rate (a fleet of devices outproduces any scheduled collection's usefulness window) and the monitoring use case itself (the insight is "is something wrong right now?", which is inherently per-event, not per-month). When the source is a monitoring system, the question it answers is a streaming question.

Comparison — the four reasons at a glance:

When to pick which: any one of the four being true is a working argument for stream processing; all four true together is the definition of a streaming-native workload.

Recap and bridge. Four reasons justify streaming: the never-ending nature of the data, time series data ($p=f(t)$), storage and infrastructure limits, and IoT/monitoring sources. Section 3.11 closes the lecture by listing the concrete sources and use cases of streaming data in industry.

3.11 Streaming Data: Sources and Use Cases

3.11.1 Data Sources

3.11.2 Use Cases

The use cases of streaming data named in class:

• Financial institutions — the typical example of time series data (the stock market case from 3.10.2, run at milliseconds-to-seconds scale).
• Sensors in transportation vehicles — vehicle telemetry streams (the smart-car temperature monitoring from 3.1.6).
• Identification of defects — detecting anomalies from streaming sensor readings (a monitoring-system source, 3.10.4: a reading outside its normal band flags a failing part before it breaks).
• Twitter messaging and sentiment analysis — the trending-events use case from 3.1.1, where tweet volume and mood are aggregated into the news ticker.
• Flight tracking — position streams from aircraft (near real-time traceability, like the bus tracker of 3.1.4).
• ATMs and bank transactions — stream processing in fixed timelines, with a data collection window and a data eviction window (the ATM example of 3.8).
• Geolocation tracking of trains — the case study from 3.6, where the enriched event is the train's timestamp, coordinates, station, and speed.

Real-world connection. This is why banks, transit operators, and industrial plants run streaming platforms — the data is born continuously, and insight loses value by the minute. A fraud model fed by ATM and card streams must score each transaction before the settlement window (soft real-time, 3.1.3); a metro control room must see every train's enriched event within seconds (3.6); a chemical plant must filter and aggregate chamber readings to catch the first anomaly (3.2). The stream is not an exotic technology choice — it is what the data's own nature demands, which is the lecture's closing point and the lead-in to the architectures that come next.

Exam Guidance Summary

• Expect a use case classification question: given a scenario, classify the system as hard real-time, soft real-time, or near real-time. The exam tests whether you understand the basis of separation, so answer with the two axes — latency scale and tolerance for delay — plus the life-risk check. Answer format: name the latency scale, name the tolerance, then the class.
• Always refer to the latency-tolerance table (3.1.5) when classifying; the guidance was explicit that otherwise you will substitute your own reasoning. Reproduce the table from memory: hard = milliseconds–microseconds with zero tolerance; soft = milliseconds–seconds; near = seconds–minutes.
• Key distinctions to be able to reproduce: hard = milliseconds–microseconds, zero tolerance; soft = milliseconds–seconds; near = seconds–minutes. Money loss pushes a system out of hard into soft; life risk pushes it into hard. The soft/near boundary is blurry; the hard boundary is not.
• Be able to state the rule: decision based on the enriched event → hard real-time; analytics-only enrichment → not hard real-time (self-driving car vs train tracking; driverless train vs driver-controlled train).
• Know the two operations of stream processing: filtering and aggregation — and the point that stream processing is not a control system (it generates insight; a human takes action).
• Know that streaming systems are non-hard real-time: stream processing contains soft and near real-time, and hard real-time is a separate family (no streaming in hard real-time systems).
• Know the four differentiators between batch and stream processing (data scope, data size, time scale/latency, depth of analysis) and the two batch triggers (time-based and count-based — the "special window").
• Know the four reasons for choosing stream processing (nature of data, time series, storage/infrastructure limits, data source) and the four sources of streaming data (smart devices, IoT devices, mobile phones, and more).
• Reference book: Real-time Analytics by Byron Ellis — one book is enough; it is the course textbook for this material. Its real-time classification table matches the lecture's latency-tolerance table.
• Roadmap: architectures and streaming data come next.

Key Industry Applications

• Twitter trend feeds and sentiment analysis — near real-time text analytics powering the trending tickers on news channels (3.1.1).
• Credit card transaction alerts — instant notification with 24-hour settlement, a soft real-time system (3.1.3).
• Self-driving vehicles — hard real-time image-message processing where a lost message can cause an accident (3.1.2, 3.5.3).
• Live vehicle tracking (buses, trains) on maps — near real-time GPS traceability (3.1.4).
• Stock market feeds — soft real-time (milliseconds to seconds), where minutes-scale delay means lost money (3.1.6, 3.10.2).
• Chemical plant chamber monitoring — filtering and aggregation over temperature/pressure messages; classification depends on the hazard of the chemical (3.2).
• Smart car temperature monitoring — soft/near real-time with human intervention time (3.1.6).
• CCTV-based complex event processing — correlating frames (fast two-wheeler, police siren minutes later) to infer robberies in progress (3.5).
• Railway signaling and tracking — soft real-time enrichment of train events; driverless (CBTC-based) metro operation approaches hard real-time (3.6).
• Payroll processing — the canonical scheduled batch job, daily or monthly (3.7).
• Storm, Spark, Kafka — the industrial micro-batch/streaming platforms used for the frameworks assignment (3.9).
• IoT device and monitoring-system streams — the data-source reason for choosing stream processing, driving fraud detection, defect identification, and flight tracking (3.10.4, 3.11).