—

Stream Processing and Analytics

Spark Structured Streaming: Word Counts, Output Modes, and Windows

Published: 2026-08-07

Level: postgraduate

Audience: Postgraduate students learning stream processing and analytics

This session brings Spark streaming to life. Everything before was vocabulary and setup — the families of stream operations, the two working environments, and the idea that a stream is processed in small chunks. This session runs the classic streaming word count: a netcat listener feeds lines of text over a TCP socket, and a Spark application turns each line into a micro-batch, splits it into words, and counts them live, table by table.

The session then answers three questions that every streaming application must face. What should the output look like after each batch? Spark offers three answers — complete, update, and append — and the demo shows all three, including one that refuses to run without a watermark. What is the difference between when an event happens and when it reaches the system? The two timestamps — event time and stream time — and the gap between them, called time skew, turn out to be quiz material. How do you compute over a stream that never ends? The answer is windows, and the session covers the two core types — tumbling and sliding — with the trigger and eviction policies that define how each window fills and empties.

By the end, the word-count tables from the demo, the three output modes, the two timelines, and the two window types all connect into one picture: how a finite-memory engine processes an unbounded stream in slices, and why every result has a scope attached to it.

11.1 Spark Structured Streaming and the Streaming Word Count

The hook. What does a real streaming application look like from the inside — not a diagram, but a running program? This section builds one: a listener process on a terminal window, a Spark application in an IDE, and text messages flowing from one to the other over a network port. It is the smallest complete streaming pipeline there is, and it shows every moving part in action.

11.1.1 Where We Start: Transformations, Actions, and the Two Environments

The session picks up from where the last one left off. The earlier session introduced Spark streaming and covered the two families of operations you can run on a stream:

Transformations — operations that build a new stream or DataFrame from an existing one but compute nothing by themselves. They describe what to do; they do not do it.
Actions — operations that actually trigger computation and return results. The work happens only when an action fires.

Intuition — a recipe book. Think of transformations as the recipe and actions as the cooking. You can read a recipe and plan the whole meal without a single pot being touched — that is a chain of transformations. The moment you turn on the stove, the cooking starts — that is the action. A Spark program can pile up a long chain of transformations and still do no work; the work starts at the action, and every transformation in the chain runs at that point.

Two working environments were set up during installation. One is Jupyter Notebook, a browser-based cell-by-cell environment where you run small pieces of code in order and see each result immediately. The other is PyCharm, a full Python IDE where you write a script and run it as a program. Both work for Spark streaming; the live demo in this session runs in PyCharm.

One practical point from the setup phase: the streaming commands shown below do not need a notebook environment at all. A plain command prompt is enough for the data-generation side, and the notebook stack (such as Anaconda) is not required for that part. The two sides of the demo are deliberately low-tech: a terminal window on one side, an IDE on the other.

The example the session runs is the classic streaming word count. It is the "hello world" of streaming: a stream of text messages arrives, and the application counts how often each word appears. It matters beyond the classroom too — this is the first streaming application people are asked about in job interviews, because it exercises almost every core streaming idea: a data source, micro-batches, state, and output modes.

11.1.2 The SparkSession Singleton

Every Spark application starts the same way: by opening a SparkSession. A SparkSession is the entry point that ties together configuration, the cluster, and your code. Before you can read a stream, register a DataFrame, or run any computation, the session must exist.

The key rule: a SparkSession is a singleton. Singleton means one object for the whole application — exactly one session per use case. You do not create several Spark sessions for one use case; you create one session and run all the operations for that particular use case through it. The pattern in the demo application looks like this:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("StreamingWordCount") \
    .config("spark.sql.streaming.stopActiveRunOnRestart", "true") \
    .getOrCreate()

The application code sets a few things on the session: an application name (used to identify the job in logs and the cluster UI) and a configuration for shutdown — the demo requests a graceful shutdown so that the running query can stop cleanly rather than being killed mid-state. Logging is optional: if you want logs you can create a logger object; otherwise you can skip it.

Intuition — one front door. Think of the SparkSession as the single front door of a building. All visitors (operations) enter through that one door, and the building's systems (config, cluster connection, catalog) are all reachable from it. Two doors would mean two buildings — two separate Spark contexts with separate state, which is why the rule is "one use case, one session."

11.1.3 The Data Source: Netcat and a TCP Socket

The word count needs a stream of text as input, and the data source used here is a TCP socket. The setup has two sides. On one side, a small application called netcat (invoked as ncat) generates the stream of text messages; on the other side, Spark reads those messages from a port on the network. This is the "listen" model: the netcat process listens on a port, waits for a connection, and then forwards whatever you type to whoever is connected — in this case, your Spark application.

To generate the stream you first install netcat. The netcat application can be downloaded from nmap.org, which publishes installers for different machines — Windows, Mac, and others. On Windows you get an executable installer (the version used here is nmap 7.9.4 setup.exe). Once installed, you can run the message-generating command on the fly from a command prompt. You could also bake the netcat command into a configuration file, but the demo prefers running on the fly: open a command prompt in the folder where the streaming application lives, and type:

ncat -lk 9999

Here -l puts netcat into listen mode, -k keeps the connection open after the client disconnects, and 9999 is the port number the messages are sent to. The text is sent over the TCP/IP protocol to port 9999, and the Spark application, which has already been told about that port, receives the lines. Because the messages are generated and consumed on the same machine, the host is localhost.

Worked example — the message path, step by step.

Start the listener: run ncat -lk 9999 in a command prompt. The process now sits waiting on port 9999 — no messages exist yet.
Start the Spark application in PyCharm. The application opens its SparkSession and attaches a streaming read to the socket: stream format socket, host localhost, port 9999.
Type a message in the netcat window, for example Hello Spark, and press Enter.
The text travels over TCP/IP to port 9999 on the same machine (localhost).
The Spark application receives the line as one unit of stream data, splits it into words (Hello, Spark), counts them, and prints the result table.

Result: every Enter press in the netcat window becomes one round of processing in the Spark application. The message path is: netcat window → TCP socket → Spark application → count table. Sense-check: the direction is easy to verify — type anything in the netcat window and the Spark window prints a table within a moment; type nothing and no table appears.

Real-world: netcat, distributed from nmap.org, is the standard quick tool for generating test network streams — you will see it in many streaming tutorials. The word-count-over-socket setup is also a faithful miniature of how real streaming systems ingest data: messages pass through a data flow layer before reaching the processing engine. One caution from the demo: a socket source is fine for development and testing, but a production streaming application should not rely on a raw socket feed — Spark itself warns about this when you run it.

Pitfalls.

Forgetting the listener must run first. If the Spark application starts while netcat is not listening, it connects to nothing. The listener must be active on the port before the application reads.
Mixing up -l and -k. -l alone closes the connection after the first client disconnects; -k keeps it alive for the next batch. The demo needs both, in that order: -lk.
Using a socket source in production. A raw socket feed has no durability, no replay, and no ordering guarantees. It is a teaching and testing tool — real pipelines read from Kafka, Kinesis, or files.
Treating the port as an application property. The port is a number agreed between the two sides — netcat must listen on the same port (9999) that the Spark read uses. A mismatch silently produces no data.

11.1.4 What the Application Does End to End

Putting the pieces together: the netcat listener runs in one window; the Spark word count application runs in another. With the listener active but no messages typed yet, the application sits idle and prints counts of zero. The moment you type a line and press Enter, that line becomes a unit of stream data — a micro-batch — that Spark picks up, splits into words, counts, and prints.

Recap + bridge. The streaming word count is a complete miniature pipeline: netcat on a TCP socket produces the stream, a singleton SparkSession owns the application, and each typed line becomes a micro-batch that the engine transforms and counts. Handoff: section 11.2 runs the demo batch by batch and watches the counts — and the surprises — roll in.

11.2 The Streaming Word Count Demo, Batch by Batch

The hook. Type a sentence, press Enter, and a table appears. Type another sentence, and the table changes — but not the way you might guess: some words climb by one, some vanish entirely, and one word stubbornly refuses to count when it looks identical to a word counted before. This section works through those tables one by one, because the way counts move between batches is the way streaming state actually works.

11.2.1 Micro-Batches and How the Counts Roll In

A micro-batch is the small chunk of stream data that Spark processes at each trigger. The stream is not one giant computation — it is a sequence of tiny computations, one per micro-batch. In the demo, one typed line equals one micro-batch.

Here is the first micro-batch. In the netcat window you type:

Spark is good programming language

Press Enter. The application counts the words in this micro-batch: Spark, is, good, programming, language — five words, and each appears once, so each gets a count of 1.

Word	Count
Spark	1
is	1
good	1
programming	1
language	1

Now the second micro-batch. You type:

Spark is supports Java, Scala, Python

The moment you press Enter, a new batch table appears:

Word	Count
Spark	2
is	2
supports	1
Java,	1
Scala	1
Python	1

What the second table tells you. Two observations jump out. First, Spark shows a count of 2 even though it appears once in this micro-batch — because Spark was used in the first batch as well, and its stored count was carried forward and incremented. Second, look at what is missing: the words programming and language, both in the first batch, do not appear in this output at all.

Why? Because the batch update happens with respect to the words of the current batch. The output concentrates on the words from the current micro-batch only; it does not redisplay words from previous batches. But wherever a word appears in a previous batch and in the current one, that word's count is incremented — the accumulated value is updated even though the word itself was last seen earlier. So the rule has two halves:

The words shown are always the current batch's words.
The counts shown are the running totals across all batches so far.

This behavior is called update mode, and the name captures exactly what is happening: the update happens per the current batch, but the information from previous batches is updated too. We will come back to output modes formally in the next section.

Exam note: this update-mode demonstration is not just a demo detail — the session made a point of calling the output modes very important from an examination standpoint. Watch the tables here until the behavior makes sense before moving on.

11.2.2 Worked Example: Four Micro-Batches of Streaming Word Count

The demo continued through four micro-batches, and each one is worth reconstructing in full because the counts show the state behavior of the stream.

Worked example — micro-batches 3 and 4.

Micro-batch 3. You type:

C++ is good programming language

The next count table shows: C++ count 1, is count 3, good count 2, programming count 2, language count 2.

Word	Count
C++	1
is	3
good	2
programming	2
language	2

The interesting moves: programming and language appeared earlier as well, so their counts become 2 — they went from 1 to 2, not from nothing. good likewise becomes 2. is has now appeared in all three batches, so it reaches 3. C++ is new, so it enters with 1.

Micro-batch 4. Before typing, the class was asked to predict the output for:

Python is better here than Java

The prediction: Python will be count 2 (it appeared in the second batch as well), is will move to 4, here will be 1, than will be 1, and Java should be 2 — because the second batch contained the word Java, or so it seemed.

Word	Count
Python	2
is	4
better	1
here	1
than	1
Java	1

The table confirmed the predictions except for one: Java became 1, not 2. Why? Because in the second micro-batch the sentence was typed as "Java," — with a comma attached. The word-splitter breaks text on whitespace, so the token "Java," with the comma glued on is a different word from "Java" without it. The comma got counted as part of the token, not as punctuation separating it. When the sentence was typed again with a plain Java, no comma, the count went to 2 as expected.

Sense-check: every count in the four tables obeys the same rule — current batch's words shown, running totals incremented. The only way to check the Java mystery is to look at the raw token, not the word you meant: "Java," and Java are two different tokens, so 1 is the correct count for the fourth batch.

Pitfall — the whitespace-tokenization gotcha. This is the classic whitespace-tokenization gotcha: what looks like one word to a human can be two tokens to the counter if any punctuation is attached. In a real streaming system this is exactly why preprocessing pipelines normalize tokens — lowercasing, stripping punctuation — before counting. A message stream "Hello!" and "Hello" should describe the same word; the counter will not know that unless the pipeline cleans the tokens first.

11.2.3 Student Questions and Answers

Q: Why did Java show a count of one instead of two? I expected the Java from the second batch to be counted again.

A: Because the second batch was typed as "Java," with a comma attached. The splitter breaks on whitespace, so the comma-attached "Java," is a different token from plain Java. Only the plain word Java, typed without punctuation, is what the counter saw this time — so its count starts from one. Remove the comma and it becomes two.

Q: In the second batch, why do we not see programming and language anymore, and yet Spark shows a higher count?

A: Because update mode only displays the words from the current micro-batch. Words from earlier batches are not redisplayed. Their stored counts, however, keep accumulating — whenever the same word appears again, its count is incremented. That is why Spark, which is in the first batch as well as the second, shows two, while programming and language, which are not in the second batch, are simply absent from the output.

Q: So the displayed table always belongs to the current batch only?

A: Yes — the words are from the current batch, displayed with respect to the current micro-batch. The counts themselves are the running totals, because wherever a word appeared in a previous batch, that count is also updated.

Recap + bridge. A micro-batch is one typed line, and update mode shows the current batch's words with running totals — which is why Spark climbs to 2 while programming and language vanish, and why "Java," from batch 2 did not become Java in batch 4. Handoff: section 11.3 formalizes the output modes — the three ways the engine writes its result after every batch.

11.3 Output Modes: Complete, Update, and Append

The hook. After every micro-batch, the engine must decide what to write: everything it has ever counted, only the rows that changed, or only brand-new rows? Those three choices are the output modes, and the session demonstrated all three — including the one that refused to run. The session called this topic very important from an examination standpoint.

11.3.1 Complete Mode

Spark supports three output modes — the ways the streaming query writes its result after each micro-batch. The first is complete mode.

In complete mode, the result DataFrame is completely overwritten after every micro-batch. A DataFrame — the same idea seen in the earlier session — is the in-memory dataset that holds the result. Complete mode takes that entire result set and rewrites it from scratch each time: you get the whole output, the full result set across all micro-batches, on every trigger. Nothing is kept hidden; everything seen so far is redisplayed. That is exactly why it is called complete: the result set is overwritten in full, so the output always reflects the entire history of the stream.

Formalize — complete mode in one sentence. The whole result table is rebuilt and written out after every micro-batch: output after batch $k$ contains the full cumulative state of all $k$ batches, because the result set is overwritten in full each time.

The word-count engine keeps state between batches — a running total per word — and the three output modes are three different policies for which part of that state gets written when a batch finishes. Complete mode writes all of it; the other two policies are formalized in sections 11.3.3 and 11.3.4.

11.3.2 Worked Example: The Complete-Mode Rerun

To see complete mode, the application was stopped with Ctrl+C, the checkpoint directory was removed, and the same application was restarted with the output mode changed from update to complete. (The checkpoint directory is where the application stores its streaming state so it can resume; when you rerun a streaming query you must remove the old checkpoint directory or the run resumes old state instead of starting fresh.)

Worked example — the complete-mode rerun.

The same messages were typed again — "Spark is good programming language", then "Spark is supports Java, Scala, Python". This time the output is different: instead of showing only the current batch's words, the complete-mode table shows every word that has ever been counted, with its running total. The first batch still shows the five words at count 1, but by the second micro-batch the output is the full cumulative table:

Word	Count
Spark	2
is	2
supports	1
Java,	1
Scala	1
Python	1
programming	1
language	1

Note what changed versus update mode: programming and language are back in the output even though they did not appear in the current micro-batch, because complete mode gives the complete output across all micro-batches — the earlier history is included, and it is rewritten on every batch.

Sense-check: every word ever typed appears exactly once in the table, with the number of times it has appeared. Five words from batch 1, five from batch 2, two overlaps (Spark, is) — so $5 + 5 - 2 = 8$ rows. The table has exactly 8 rows, and the two overlapping words are the only counts above 1. The state and the display now agree perfectly.

Pitfall — rerunning a streaming query without clearing the checkpoint. If you restart a streaming application and the checkpoint directory still holds the old run's state, the query resumes from where it left off — old counts persist, and the "fresh" demo output never starts clean. Remove the checkpoint directory (or point the application at a new one) before every rerun.

11.3.3 Update Mode

The second output mode is update mode, which we already met live in the word count demo. In update mode you see only those records that are either new — first appearing in the current micro-batch — or records whose old value is updated — words that were seen before and now have a higher count. So where there are common words across batches, those are updated; entirely new words appear fresh; and everything else stays out of the output. The words (or records) displayed are the ones from the current batch, with respect to the current micro-batch, and their displayed counts carry the accumulated totals.

To say it compactly: update mode writes only the changed rows — new records and updated counts — while complete mode rewrites the entire result set.

Comparison — complete vs update on the same two batches.

	Complete mode	Update mode
After batch 1	5 rows: every word, count 1	5 rows: every word, count 1
After batch 2	8 rows: every word ever seen, full history	5 rows: only batch 2's words (Spark 2, is 2, supports 1, Java, 1, Scala 1, Python 1)
Rows from batch 1 not in batch 2	Redisplayed (programming, language)	Hidden (programming, language)
Changed rows	All rewritten	Only changed rows written

When to pick which: use complete mode when the full state must be visible at a glance (small result tables, dashboards over aggregated state); use update mode when the output should mirror the current input and you want to avoid rewriting unchanged history.

11.3.4 Append Mode and the Watermark Requirement

The third output mode is append mode. Append mode is like a simple insert of unique records: the query appends only new rows to the result, never touching previously emitted rows.

In the demo, the mode was switched from update to append and the application was restarted. The run stopped almost immediately with an error message:

Append output is not supported when there are streaming aggregations on streaming data without watermark.

What the error means. The important part: when the stream performs aggregations — like the running word counts, which combine data across batches — append mode is not supported unless an extra condition is satisfied. That condition is the watermark. The watermark concept is announced here and its mechanics arrive with the window discussion in the next sections: in short, a watermark tells the streaming engine how long to keep waiting for late-arriving data before it finalizes and emits a result — which is precisely what append mode needs to guarantee that a row is never emitted twice.

Assumptions & scope. The rule to remember: append mode is fine for plain new rows (a stream without aggregations, where every record is independent and emitted once as it arrives). But once you add streaming aggregations, you must supply a watermark or the query is rejected. The reason: with an aggregation, the engine cannot know that a row is complete — a late event could still change an earlier window's result — so "append once and never touch again" is only safe when a watermark marks the point after which no late events for that window are expected.

11.3.5 Student Questions and Answers

Q: In complete mode I expected every word to be counted once — why does the output show more than one?

A: Complete mode overwrites the whole result set after each micro-batch. The table is rebuilt from the full history, so a word seen twice across batches shows a count of two, even in the very first output after the second batch arrives. The count is the number of occurrences across the whole stream, not the number of times it appears in the current table.

Q: The append-mode run failed with "Append output is not supported when there are streaming aggregations on streaming data without watermark". Does the missing condition work like a save point?

A: The required condition is the watermark, not a save point. A save point would store progress and let the run resume; a watermark is a time boundary the engine uses to decide when old data can be considered finished, so rows are only appended once. Append mode cannot run with streaming aggregations unless a watermark is set — the engine needs it to know when old data can be considered finished so rows are only appended once. The details of how the watermark works arrive with the window concept.

Exam note: the output modes were flagged as very important from an examination standpoint. Be able to reproduce the word-count tables for update mode (current batch's words shown, running totals incremented) and complete mode (full result set overwritten every micro-batch), and state the append-mode rule: streaming aggregations require a watermark.

Recap + bridge. Complete mode rewrites the full history each batch; update mode writes only new or changed rows; append mode writes only new rows and needs a watermark for aggregations — which is why the water bucket is thrown over to the next sections. Handoff: section 11.4 introduces the two timelines — event time and stream time — that make the watermark's job understandable.

11.4 Event Time, Stream Time, and Time Skew

The hook. A person jumps a signal at a crossing. When did the event happen — the instant the foot crosses the line, or the moment the camera system records it? The answer is not the same, and the difference between those two instants is a quantity you can write down, measure, and quiz students on. This section names both times, defines the gap between them, and shows why the gap matters for every windowed computation that follows.

11.4.1 Two Timelines: Event Time and Stream Time

Streaming processing deals with two different times, and keeping them apart is a core skill. The first is event time: the time at which the event actually occurs in the world. The second is stream time: the time at which the event's information reaches the platform.

Worked example — the traffic monitoring scenario.

Take the demo example: monitoring traffic on a road, and suppose a person jumps the signal at a crossing.

Event time — the time at which the person actually jumped the signal. This is written on the event itself, as a timestamp produced where the event happened.
Stream time — the time at which that event's information reached the streaming platform. The message carrying the jump travels through a data flow layer first, so the platform sees it a moment after it happened.

Everything that happens after the jump — snapping the photo, computing the fine, sending the alert — is part of processing, downstream of the event itself.

Stage	Timeline	Timestamp
Signal jumped	Event time	12:00:03
Photo snapped	Processing	12:00:04
Message reaches platform	Stream time	12:00:07
Fine computed and alert sent	Processing	12:00:08

Sense-check: the event time (12:00:03) is always the earliest timestamp on the record, and stream time (12:00:07) always comes after it, because the message must physically travel to the platform before it can be processed.

Q: Is this like the automatic traffic charging systems we see today, where a broken signal is captured by a camera?

A: Exactly — the broken signal is the event generated over there. The event time is the moment the signal was broken; everything afterward — snapping the photo, computing the fine, sending the alert — is the processing that follows.

Two consequences follow, and both are worth writing down.

First, stream time will always lag a little behind event time. The actual timestamp of an event is earlier than the moment the event reaches the system, because there is a data flow layer the messages must pass through. You cannot expect an event to reach the processing platform at the same instant it occurs, and you do not directly ingest messages straight into the streaming engine — the messages travel through the data flow layer first.

Second, this lag is exactly why the two timestamps must be distinguished. If you process by the arrival time instead of the occurrence time, your windowed results will be shifted. A window keyed to arrival time groups events by when they were seen, not when they happened — and for an out-of-order world, those two groupings disagree.

Q: Streaming time will always be a little behind, right? The event happened, but the system sees it later because of the data flow layer.

A: Yes — the gap exists because events pass through a data flow layer before they reach the platform. You cannot ingest a message directly into the engine at the same instant the event occurs; the arrival is always after the fact, so stream time lags event time.

Exam note: note down the event time / stream time distinction — the session flagged it explicitly as quiz material. A quiz is planned for the coming weekend covering everything up to this point, so treat both definitions and the gap between them as testable.

11.4.2 Time Skew

Because stream time always lags event time, the two timelines do not line up. If you plot the two against each other — event time on one axis, stream time on the other — the ideal situation would be a straight line through the origin: the stream sees every event at the moment it occurs. In practice the observed curve sits below that ideal line: the stream sees each event a little later, and the lag widens as events flow through the data flow layer.

The gap between the ideal time and the observed time — the difference between event time and stream time — is called the time skew:

$t_{skew} = t_{stream} - t_{event} > 0$

where $t_{event}$ is the time at which the event occurred, $t_{stream}$ is the time at which the event's information reached the platform, and $t_{skew}$ is their difference. Since stream time always lags, the skew is always positive.

Formalize — the skew formula, symbol by symbol.

$t_{event}$ — the event timestamp, produced where the event happens. Units: clock time (seconds or milliseconds from a reference).
$t_{stream}$ — the arrival timestamp, produced when the message reaches the streaming platform. Same units.
$t_{skew} = t_{stream} - t_{event}$ — the lag between the two, a duration in the same units.

Because the message cannot arrive before it happens, $t_{stream} \geq t_{event}$ always holds, and in practice the inequality is strict: $t_{skew} > 0$ . The skew is not a fixed constant — it depends on how long the message spends in the data flow layer, so it varies event by event.

Visual intuition. Draw a chart with event time on the horizontal axis and stream time on the vertical axis, both in clock time. The ideal situation is the straight line $y = x$ through the origin: stream time equal to event time at every point, the stream seeing each event at the instant it occurs. The observed line hangs below it — for every event, the observed stream time is greater than the event time, so the observed curve is always below the diagonal. The vertical gap between the diagonal and the curve at any moment is exactly the time skew: the lag between when an event occurred and when the stream saw it. The takeaway: the diagonal is the "same instant" dream, the gap under it is the reality of the data flow layer, and the width of the gap decides how careful windowing must be.

Worked example — measuring the skew with real numbers.

Three events arrive at the platform, each carrying its own event timestamp, with the messages taking different times to travel through the data flow layer:

Event	$t_{event}$	$t_{stream}$	$t_{skew}$
$e_{1}$	12:00:03	12:00:05	2 s
$e_{2}$	12:00:04	12:00:09	5 s
$e_{3}$	12:00:05	12:00:06	1 s

For $e_{2}$ : $t_{skew} = 12 : 00 : 09 - 12 : 00 : 04 = 5$ seconds. Note that $e_{2}$ has a larger skew than $e_{1}$ even though it happened later — the skew is per-event, not a global constant. Also note the ordering effect: $e_{3}$ (event time 12:00:05) reaches the platform before $e_{2}$ (arrives 12:00:09), so the arrival order is $e_{1}, e_{3}, e_{2}$ while the event order is $e_{1}, e_{2}, e_{3}$ . This is out-of-order data: the stream can see $e_{3}$ before $e_{2}$ . Sense-check: on the skew graph, $e_{2}$ sits farthest below the diagonal (5 s of gap), $e_{3}$ sits closest (1 s); both below, never above — which matches the rule $t_{skew} > 0$ .

Pitfalls.

Processing by stream time when the business question is about event time. The classic error: counting "requests per minute" by the minute the request arrived rather than the minute it happened — a stream processor that restarts and replays a backlog makes the arrival rate look like a spike even when the real rate was steady. Window results keyed to the wrong timeline are shifted, and the shift is not constant.
Treating the skew as a fixed number. The lag varies event by event (1 s, 5 s, 2 s in the example above) and can change with network load. A single "average delay" does not describe the stream.
Assuming arrival order equals event order. Because skews differ per event, messages can arrive out of order. Any computation that assumes "first seen = first happened" will be wrong for the stragglers.
Ignoring the clock source. Event timestamps are only trustworthy if the producing device's clock is sane; a mis-set client clock can turn a normal event into a bizarre straggler.

Why this matters for windows: when you group events into time windows, you must decide which timestamp the window reads. The events carry their own timestamps, so a window can be keyed to event time even when the records arrive late — and handling that late-ness is the job of the watermark.

Recap + bridge. Event time is when the event happened; stream time is when its message reached the platform; the skew $t_{skew} = t_{stream} - t_{event}$ is always positive, varies per event, and is why arrival order is not event order. Handoff: section 11.5 uses these two timelines to motivate windows — the finite slices that make an infinite stream computable — and the watermark that governs late events.

11.5 Windows: Why Streams Need Them

The hook. The stream never ends, and the machine's memory is fixed. How can a computation that needs the data finish when the data never finishes arriving? The answer is a trick as old as reading: do not look at the whole page — look through a lens, one stretch at a time. Those stretches are windows, and this section explains why every streaming system runs on them.

11.5.1 The Window as a Lens over an Infinite Stream

A stream is, by nature, infinite — or at least unbounded — and the processing engine's memory is finite. You cannot hold the entire stream for processing, and you do not want to wait for it to end before computing anything. So the processing happens on finite slices of the stream, and those slices are called windows.

Intuition — the magnifying lens. The framing used in the discussion: imagine a magnifying lens held over a page. You cannot read the entire sentence at once — you can only read the words that fall inside the lens diameter. That lens diameter is the window length: whatever the window's capacity is, only that many data points and events are sent to the processing engine at a time. You define the window length to match what your processing speed can handle — you never capture more messages than you can process, because in any streaming application you never capture the entire stream anyway.

Where the analogy holds: the page is the stream, the lens is the window, and moving the lens down the page is the stream flowing past. Where it breaks: a page ends and the stream does not — the lens keeps moving forever, and the words it already passed are gone unless the engine kept a summary of them.

There is a deeper point hiding here: in stream processing, the definition is always that processing happens on a finite amount of data. That is the price of working with an unbounded input and a bounded machine.

A window can be defined by time (the lens covers events from 12:00 to 12:05, say) or by count (the lens covers the next 100 events). The discussion that follows stays with time windows; count-based windows are a simpler variant — how many events you take — and were noted as not important for the current treatment.

Real-world: every production streaming system (click analytics, fraud detection, IoT sensor processing) is windowing by default — no engine can wait for a stream to end, so results are always per-window results.

11.5.2 Streaming Algorithms Are Non-deterministic

Because streaming algorithms work on windows rather than on the whole data set, they are called non-deterministic algorithms.

The reasoning, step by step.

A batch algorithm reads the entire data set, so its answer does not depend on how the data is sliced — the same input gives the same output every run.
A streaming algorithm never sees the entire data — it only collects samples through the concept of the window.
Which window an event falls into depends on windowing choices: the window type, the length, the slide, the trigger.
Therefore the same stream processed twice, with windows sliced differently, can produce different answers.

So a streaming result is not wrong when it differs from another run — it is approximate in a principled way. The approximation is built into the windowing, not a bug in the engine.

Comparison — batch versus streaming determinism.

	Batch processing	Streaming processing
Data seen	Full data set	Only windowed slices
Input	Bounded	Unbounded
Same input, same answer?	Yes, every time	Not guaranteed — answer depends on windowing choices
Result status	Exact	Principled approximation

When to pick which: when the question must be answered exactly over a known, finite data set, batch processing is the right tool; when data arrives continuously and the answer must keep updating, streaming with windows is the only option — and the approximation is the accepted trade-off.

11.5.3 Trigger Policy

Once you have windowed the stream, you still need to decide when the processing runs. The trigger policy defines the rules for when the code should be executed: you have collected the events, and you want to process them; the decision that "I am going to process this window now" is the trigger. You trigger the algorithm on the window data, and the algorithm is run on the processing engine — potentially across the executors that do the distributed work. So the trigger policy is the schedule of when each window gets processed, while the window defines what data gets processed.

Intuition — an alarm clock. The window is the cup of coffee to drink; the trigger is the alarm that says "drink it now." A tumbling window's alarm rings every window length (fixed schedule); a sliding window's alarm rings every slide interval; a count-based window's alarm rings when the window is full. Whatever the rule, the trigger is a when decision — it never changes what data is in the window, only when the engine looks at it.

11.5.4 Eviction Policy

The flip side of the trigger is the eviction policy: what happens to the window's data after processing. The picture used here was a water sampling trip on a river. You dip a mug into the stream, collect one mug of water, and analyze it — what salts are in it, what the pollution levels are. Then you throw the water away. Throwing it away is the eviction policy: you clean up your window to make room for the next set of records, or events. You take the same mug to a slightly different position in the water, collect another mug of water, and analyze that.

The mug-and-river mapping.

The mug is the window capacity: with one mug you cannot store the entire river, just as with finite memory you cannot store the entire stream.
The water is the data.
The analysis is the processing.
Emptying the mug after each analysis is the eviction policy.

Without an eviction policy, windows would pile up and the finite memory would fill. The eviction policy is what keeps the engine's storage bounded: process, empty, refill.

Q: With the mug, you throw away the analyzed water — that throw-away is the eviction policy, right?

A: Yes. You analyze one mug of water, then you throw that water away to clean up your window and store the next set of records. Then you take the same mug to a slightly different position in the stream and collect again. The mug is your window capacity — you cannot store the entire river water in one mug, and in the same sense you cannot store the entire stream data in your window, because your memory is finite while the stream is infinite.

Q: I did not follow the trigger and eviction discussion — what types of windows exist?

A: The window types — tumbling and sliding — come next. What has been covered so far are the two policies: the trigger policy decides when the window's data gets processed, and the eviction policy decides when the processed data is thrown away to free the window for the next records.

11.5.5 Student Questions and Answers

Q: At any one moment, is only one window active, or can several windows be active at the same time?

A: One window only — always one. The window exists because the processing engine has limited storage and is collecting data it cannot hold; the engine collects the data, processes it, and dumps it. With one window, the engine never holds more than it can handle. (Whether the window holds more events is a matter of window size — that is where the different window types come in.)

Pitfalls.

Confusing trigger with eviction. The trigger says when processing runs; the eviction policy says when processed data is thrown away. A window can trigger many times before it evicts (sliding windows are exactly that case).
Forgetting the eviction policy exists. A window that never empties fills the finite memory — the "lens" stops moving because the lens itself is full.
Expecting exact answers from a windowed algorithm. Non-deterministic is not a bug report; it is the definition of streaming computation. Design for it (windows, watermarks, aggregates) instead of against it.
Thinking the window waits for the stream to end. It never does — the whole reason windows exist is that the stream will not end. If a computation needs all the data, it is batch processing, not streaming.

Recap + bridge. The stream is unbounded and the memory is bounded, so processing happens on finite slices called windows; the trigger policy decides when each window runs, the eviction policy decides when its data is emptied, and windowed computation is non-deterministic by definition. Handoff: the two concrete window types — tumbling first, then sliding — turn these policies into numbers.

11.6 Tumbling Windows

The hook. Ask "how many orders arrived in the last five minutes?" and you want one clean number per five-minute slice — 12:00–12:05 gets its own answer, 12:05–12:10 gets another, and no order is ever counted twice. The window type that delivers exactly that is the tumbling window, and its whole character fits in one equation: batch size equals window size.

11.6.1 Tumbling Window Mechanics

The first window type is the tumbling window. A window is simply a collection of events grouped together for processing. In a tumbling window there are two quantities: the batch size and the window size — and the defining property is that in a tumbling window the batch size equals the window size:

$T_{batch} = T_{window}$

where $T_{batch}$ is the batch interval — how often the engine processes — and $T_{window}$ is the window size — how much time's worth of events is collected. When they are equal, every processing run collects exactly the events of the latest window, processes them together, and moves on to the next, non-overlapping window.

Formalize — the equal-interval rule, symbol by symbol.

$T_{batch}$ — the batch interval: the time between one processing run and the next. It answers "how often does the engine run?"
$T_{window}$ — the window size: the span of time whose events each run collects. It answers "how much history does one run cover?"

The equation $T_{batch} = T_{window}$ means the two answers are the same number: the engine runs once per window, and each run covers exactly one window's worth of time. Because the run interval and the covered span are equal, consecutive windows butt up against each other with no gap and no overlap — which is the defining property of the tumbling window.

Example: take a window size of five minutes. The engine processes every five minutes, and each time it collects only the events that arrived inside those five minutes. It is like dipping the mug into the water at 12:00 and collecting only until 12:05 — the events between 12:00 and 12:05 are collected into the window because the window size is five minutes and the batch interval is five, so every five minutes the engine processes the events from the last five minutes.

11.6.2 Worked Example: The Five-Minute Timeline

Worked example — the five-minute timeline.

Here is the timeline the demo worked through. Suppose the time axis runs 12:00, 12:05, 12:10, and the window size is five minutes. Events occur at various moments: call them $e_{1}, e_{2}, e_{3}$ between 12:00 and 12:05, $e_{4}, e_{5}, e_{6}$ between 12:05 and 12:10, and so on, with $e_{7}$ somewhere further along.

Window	Covers	Events processed	When the trigger fires
Window 1	12:00–12:05	$e_{1}, e_{2}, e_{3}$	12:05
Window 2	12:05–12:10	$e_{4}, e_{5}, e_{6}$	12:10
Window 3	12:10–12:15	$e_{7}, \dots$	12:15

Window 1 covers 12:00 to 12:05. The events processed together are $e_{1}, e_{2}, e_{3}$ .
Window 2 covers 12:05 to 12:10. The events processed are $e_{4}, e_{5}, e_{6}$ .
The process repeats: every five minutes, the engine takes the events that happened in the last five minutes and processes them as one batch.

The names matter less than the pattern: the events are grouped in consecutive, non-overlapping slices of the timeline, and each slice is processed as a unit. Sense-check: each event appears in exactly one row of the table — $e_{4}$ lands only in Window 2, never in Window 1 or Window 3 — and the windows cover the timeline with no gaps, because $T_{batch} = T_{window}$ .

Visual intuition. Draw a horizontal time axis from 12:00 to 12:15. Mark five-minute blocks — 12:00–12:05, 12:05–12:10, 12:10–12:15 — like tiles laid edge to edge. Scatter the events $e_{1}$ through $e_{7}$ at their times: each dot falls inside exactly one tile, never on a boundary twice. The takeaway: tumbling windows partition the timeline — tiles that share edges but never overlap, and every event lands in exactly one tile.

11.6.3 Key Properties

The tumbling window has one headline property: any event belongs to only one window. There is no overlap of events across the windows — each event lands in exactly one slice of the timeline. If an event sits between 12:00 and 12:05, it is processed in the first window and never again.

The no-overlap property, stated precisely. For tumbling windows with window size $T_{window}$ , window $k$ covers the half-open interval $[k T_{window}, (k + 1) T_{window})$ . Every event with timestamp $t$ satisfies $k T_{window} \leq t < (k + 1) T_{window}$ for exactly one integer $k$ — so the event belongs to exactly one window. The half-open form is what makes the boundaries work: an event at exactly 12:05 belongs to the 12:05–12:10 window, not both.

11.6.4 Use Cases for Tumbling Windows

When do you reach for a tumbling window? Whenever your question is about a fixed slice of time — for example: how many new orders were received in the last five minutes? That kind of question maps directly onto a tumbling window, because each window cleanly reports on its own slice without double-counting. A moving average, by contrast, is not a tumbling-window job — sliding windows handle that case, and they are the next topic.

Real-world: dashboards that answer "orders in the last five minutes", "errors in the last hour", or "transactions per minute" are tumbling-window reports in production systems.

11.6.5 Student Questions and Answers

Q: Are the windows always continuous? If I process e3 and then the next window begins, could e4 be missed?

A: No — nothing is missed. The tumbling window collects the events between 12:00 and 12:05, then the events between 12:05 and 12:10, and so on. Every event falls into its own five-minute slice; the next window simply starts where the previous one ended.

Q: So any event belongs to only one window, and there is no overlap of events across the windows?

A: Fantastic — exactly that: one event, one window, no overlap. If you see that, you have understood the tumbling window.

Pitfalls.

Counting an event in two windows. With tumbling windows this should never happen — an event near a boundary (at 12:05, say) belongs to one window only, determined by the boundary convention. If a report double-counts a boundary event, the boundary handling, not the window type, is the bug.
Treating a tumbling window as a sliding window. A moving average needs overlap; a tumbling window has none by definition. Reaching for tumbling to compute a moving average silently turns it into a step function of disjoint averages.
Forgetting the trigger. The engine processes at $T_{batch}$ , not continuously — events that arrive after the window closed wait for the next one. A low-latency question needs a smaller window size.

Recap + bridge. A tumbling window is defined by $T_{batch} = T_{window}$ : the engine runs once per window, each window is a non-overlapping slice, and every event belongs to exactly one window. Handoff: section 11.7 relaxes the equal-interval rule — a window that covers more history than each slide, overlapping on purpose, for moving averages.

11.7 Sliding Windows

The hook. A stock trader wants the average of the last five minutes — then the average of the next five minutes, recomputed every minute, always covering the recent past. A tumbling window cannot do that job, because each slice would discard the history the average needs. The sliding window exists for exactly this: windows that overlap on purpose, so every new result still remembers the recent past.

11.7.1 What Sliding Windows Are For

The second window type is the sliding window, also called the hopping window. Think of the magnifying glass again, but this time sliding it along the page: as it moves, the words at the trailing edge fall out while new words enter at the leading edge, and at any moment the lens still shows a continuous stretch. The key difference from the tumbling window is that sliding windows can overlap — the same event can be part of several consecutive windows.

The natural use case is computing a moving average — for example, moving averages in stock prices: what is the average of the last five minutes, then the average of the next five minutes, and so on. There, overlap is exactly what you want, because each new window must include the recent history to make the average smooth. The same idea serves when you want to track the top 10 transactions in the last three hours — a window that slides forward and always covers the recent past.

Comparison — tumbling versus sliding.

	Tumbling window	Sliding window
$T_{batch}$ vs $T_{window}$	Equal	Window larger than slide
Overlap between windows	None	Yes, by design
An event belongs to	Exactly one window	Possibly several windows
Natural question	"New orders in the last 5 minutes"	"Moving average over the last 5 minutes"
An event in several windows	Never	Normal and intended

When to pick which: ask whether the question needs smoothness from shared history — if yes, sliding; if the question wants clean, disjoint slices, tumbling.

Q: Could I not compute the new-orders count, the tumbling-window use case, with a sliding window as well?

A: You would have to keep marking which events were already processed, and the counts would get complicated — the "new orders in the last five minutes" question is naturally a tumbling-window question, because the windows do not overlap and nothing is counted twice. When you want a moving average, or a running window over the last few hours, the sliding window is the natural fit.

11.7.2 Slide Interval and Window Interval

The sliding window is defined by two quantities. One is the slide interval — how often the window moves forward and a new processing run is triggered. The other is the window interval (also called the batch interval here) — how much data each processing run covers. In a sliding window these two are not equal; the window interval is larger than the slide interval, and that difference is what creates overlap:

$T_{window} > T_{slide}$

where $T_{slide}$ is the slide interval (how often the trigger fires) and $T_{window}$ is the window interval (how much history each trigger processes). Because the window covers more history than one slide, the older events of the past window remain part of the current window as well. So a sliding window can contain duplicates across runs — some of the events that were part of the previous processing batch also appear in the current batch.

Formalize — the overlap rule, symbol by symbol.

$T_{slide}$ — the slide interval: how often the trigger fires and the engine runs. It answers "how often does the window move forward?"
$T_{window}$ — the window interval: how much history each run processes. It answers "how far back does the window look?"

The inequality $T_{window} > T_{slide}$ is the entire recipe: the engine moves forward a little ( $T_{slide}$ ) while looking back a lot ( $T_{window}$ ). The overlap between two consecutive windows is $T_{window} - T_{slide}$ , and events inside that overlap are processed in both runs. The special case $T_{window} = T_{slide}$ reduces the sliding window to a tumbling window — one event, one window again — which is the cleanest way to see the two types as one family.

11.7.3 Worked Example: A Five-Minute Slide over a Ten-Minute Window

Worked example — slide 5 minutes, window 10 minutes.

Take the numbers used in the demo: the slide interval is five minutes and the window interval is ten minutes. Every five minutes a trigger fires — that is the sliding. But each trigger processes the data of the past ten minutes — that is the window.

Trigger fires at	Window covers	Events in the overlap with the previous run
12:00	11:50–12:00	—
12:05	11:55–12:05	11:55–12:00
12:10	12:00–12:10	12:00–12:05

So at 12:00 you process events from 11:50 to 12:00; at 12:05 you process events from 11:55 to 12:05; at 12:10 you process events from 12:00 to 12:10; and so on. Events from 11:55 to 12:00 are processed in the 12:00 run and again in the 12:05 run — they appear in two consecutive windows. The window time is not equal to the batch interval, and that is the whole point of the sliding window: you accumulate more data than each slide, so the processing batches overlap.

In the numbers of the demo, ten minutes is a multiple of five — double the slide interval — which makes the overlap tidy and easy to follow. The multiples are a convenience for understanding, not a requirement.

Sense-check: consecutive windows overlap by $T_{window} - T_{slide} = 10 - 5 = 5$ minutes, and the table shows exactly that — each new window shares its last five minutes with the previous window. An event at 11:57 appears in the 12:00 run and the 12:05 run, twice in total — overlap by design, not double-counting by accident.

Visual intuition. Draw the time axis with two horizontal bars. The first bar is the window at 12:00, running from 11:50 to 12:00. The second bar sits five minutes to the right, from 11:55 to 12:05. The two bars overlap between 11:55 and 12:00 — shade that shared stretch. Each new bar is a copy of the old one shifted right by the slide, always longer than the shift, so the shaded overlap is always $T_{window} - T_{slide}$ wide. The takeaway: the window is longer than its step, and that extra length is exactly the overlap.

11.7.4 Student Questions and Answers

Q: Must the window interval always be a multiple of the slide interval? Could the slide be five minutes and the window seven minutes, for example?

A: Yes, you can have that — a five-minute slide with a seven-minute window is fine. It is not a problem. The multiples are simply a convenient way to understand the behavior. The only requirement is $T_{window} > T_{slide}$ ; 7 > 5, so the windows overlap by two minutes and everything else works the same way.

Q: For the sliding window, how does the system decide which events fall between these times — does it take the event time or the streaming time?

A: It looks at when the events occurred — the event time. That timestamp is carried on the event itself, so the event record brings its time along when it passes to the streaming system; the engine does not have to worry about arrival times for this. It only looks at the clock when it wants to refresh, when it wants to trigger — the refresh happens every five minutes here, while you are accumulating data worth of ten minutes.

One assumption to keep in mind: this works cleanly if the time taken to actually process a window is unit time — effectively instant. If you account for the actual processing time, you shift everything by that delta — and the delta is not constant, because some windows have fewer events than others. It keeps moving.

Q: Which timestamp decides which window a late event lands in?

A: The event time, again. The event's own timestamp decides its window, not the moment the stream system happened to receive it. Handling events that arrive late — after their window was already processed — is where the watermark comes in, as announced with append mode.

Pitfalls.

Sliding when the question is tumbling. Counting "new orders per five minutes" with a sliding window means inventing bookkeeping to avoid double-counting overlapping history — the counts get complicated for no benefit.
Assuming the slide must divide the window. A 5-minute slide with a 7-minute window is legal; the overlap is simply 2 minutes. Multiples are convenience, not law.
Keying windows to stream time. If the engine groups by arrival time instead of the event's own timestamp, late arrivals land in the wrong window and the moving average is distorted. The event timestamp travels with the event; use it.
Ignoring processing time. The clean timeline assumes processing a window takes unit time. Real processing time shifts every result by a delta — and the delta varies with window load, so the shift is not a constant you can subtract once.

11.7.5 The Connection to What Comes Next

Count-based windows, stream joins, and the full watermark mechanics are the next items on the path: stream joins are planned as the next topic, and the watermark's role is to tell the engine how long to wait for late events before a window is finalized. With tumbling and sliding windows plus the two timelines under your belt, the watermark will feel like the missing piece clicking into place.

Recap + bridge. A sliding window is defined by $T_{window} > T_{slide}$ : the trigger fires every slide, each run processes the last window-interval's worth of history, and the difference between the two intervals is the deliberate overlap that moving averages need. Handoff: next session brings the watermark — the clock that decides when a window is old enough to finalize — and then stream joins.

Exam Guidance Summary

Output modes are exam-critical. The session stated in so many words that complete, update, and append modes are very important from an examination standpoint. Be able to reproduce the word-count tables for update mode (current batch's words shown, running totals incremented) and complete mode (full result set overwritten every micro-batch), and state the append-mode rule: streaming aggregations require a watermark.
Event time vs stream time is quiz material. Both definitions were flagged to be noted down because a quiz is coming. You should be able to define event time (when the event occurs), stream time (when the event's information reaches the platform), time skew (their difference, $t_{skew} = t_{stream} - t_{event} > 0$ ), and why stream time always lags — including the data-flow-layer reason.
Quiz logistics. A quiz is planned for next week, covering everything covered up to this point. The window for taking it is four days, including Saturday and Sunday of the coming weekend (28th–29th). The session said the 29th (Monday) would not be needed. An announcement will be posted on the course platform (Canvas) once the quiz is uploaded, so the posting itself gives the official start time.
Assignment. The assignment was reported as nearly finished; the deadline gives time till next weekend (the 28th). Both the quiz and the assignment round out the evaluation components, which the session plans to finish next week.
Window concepts to review for the quiz: the window as a lens over an infinite stream, trigger policy (when processing runs), eviction policy (cleaning the window), tumbling windows (batch size equals window size, no overlap, one event in one window), and sliding windows (slide interval vs window interval, overlap, moving averages). Expect questions connecting these definitions to small timeline examples like the 12:00–12:10 five-minute case, and be ready to state which of the two window types a given question calls for.
Practical tip for rerunning streaming apps: remove the checkpoint directory (or use a fresh one) before each rerun; otherwise the application resumes old streaming state and the demo output will not restart cleanly.

Key Industry Applications

Streaming word count over a TCP socket — the classic first streaming application; netcat (from nmap.org) generates the message stream, Spark reads it on port 9999. The same question is a standard interview starter for streaming roles, and the mini-pipeline it demonstrates — a data source, micro-batches, state, and output modes — is a faithful miniature of production ingestion through a data flow layer.
Traffic monitoring and automatic traffic charging systems — a person jumping a signal is the event; snapping the photo, computing the fine, and sending the alert are downstream processing. Event time is when the signal was jumped; stream time is when the system sees it. The gap between the two is the time skew, and windowing by the wrong timeline shifts every report.
Moving averages in stock prices — the canonical sliding-window application: the average of the last five minutes, sliding forward every five minutes, with overlapping windows that keep the average smooth over shared history.
"Top 10 transactions in the last three hours" — another sliding-window use case over recent history: the window always covers the recent past, so the ranking never forgets the transactions that slid in minutes ago.
New orders in the last five minutes — the canonical tumbling-window dashboard question: each window reports its own non-overlapping slice, so nothing is double-counted and each five-minute block has one clean answer.
Water pollution monitoring (sample collection) — the mug-and-river picture of window capacity and eviction: analyze one mug of water, throw it away, collect the next at a slightly different position. The same pattern shows up in any stream where finite memory meets an unbounded feed: process, empty, refill.
Development caution — socket sources like netcat are for development and testing; production streaming applications should not rely on raw socket feeds, which offer no durability, replay, or ordering guarantees.

SPA Lecture 11 notes · Spark Structured Streaming: Word Counts, Output Modes, and Windows

Stream Processing and Analytics· postgraduate· 2026-08-07

Sections Breakdown

1Spark Structured Streaming and the Streaming Word Count

The classic streaming word count: transformations and actions, the SparkSession singleton, the netcat TCP socket data source, and the pipeline end to end.

2The Streaming Word Count Demo, Batch by Batch

Micro-batches and how counts roll in, the full four-micro-batch worked example, and the whitespace-tokenization gotcha.

3Output Modes: Complete, Update, and Append

The three output modes, the complete-mode rerun, update mode, and append mode's watermark requirement for aggregations.

4Event Time, Stream Time, and Time Skew

The two timelines, why stream time always lags, and the time skew formula.

5Windows: Why Streams Need Them

The window as a lens over an infinite stream, non-deterministic streaming algorithms, and the trigger and eviction policies.

6Tumbling Windows

Tumbling window mechanics, the five-minute timeline worked example, and the no-overlap property.

7Sliding Windows

Sliding (hopping) windows, the five-minute slide over a ten-minute window, and moving averages.

8Exam Guidance Summary

Exam strategy from the session: output modes, event time versus stream time, quiz logistics, and window concepts to review.

9Key Industry Applications

Real-world connections: socket word counts, traffic charging systems, moving averages, dashboards, and water pollution sampling.

Postgraduate students learning stream processing and analytics

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Spark Structured Streaming and the Streaming Word Count

Must-know: The streaming word count pipeline: ncat -lk 9999 listens on port 9999, Spark reads the socket (format socket, host localhost, port 9999), and each Enter press becomes one micro-batch; the SparkSession is a singleton per use case; a socket source is for development, not production.

⚠️ Top pitfall: Forgetting the listener must run before the application; using a raw socket feed in production; mixing up -l (listen) and -k (keep connection open).

Self-check: What does -k do in the ncat -lk 9999 command?

Connects to: 11.2

The Streaming Word Count Demo, Batch by Batch

Must-know: Update mode rule: the words shown are always the current batch's words, and the counts shown are running totals across all batches so far; a word seen before but not in the current batch is absent from the output.

⚠️ Top pitfall: Whitespace-tokenization gotcha: 'Java,' with a comma attached is a different token from Java, so the count stays 1 until the plain word is typed; preprocessing normalizes tokens (lowercase, strip punctuation).

Self-check: In update mode, why does Spark show count 2 in the second micro-batch while programming and language disappear from the output?

Connects to: 11.1, 11.3

Output Modes: Complete, Update, and Append

Must-know: Output modes are exam-critical: complete mode rewrites the full history (8 rows after two batches: 5 + 5 - 2 overlaps), update mode writes only the current batch's words with running totals, and append mode requires a watermark whenever the stream performs aggregations.

⚠️ Top pitfall: Rerunning a streaming query without removing the checkpoint directory resumes old state; the watermark is a time boundary for finalized windows, not a save point.

Self-check: Why did the append-mode run fail with 'Append output is not supported when there are streaming aggregations on streaming data without watermark'?

Connects to: 11.2, 11.4

Event Time, Stream Time, and Time Skew

Must-know: Event time is when the event occurs; stream time is when its information reaches the platform; t_skew = t_stream - t_event > 0 because messages pass through a data flow layer; on the skew graph, event time is on the x-axis, stream time on the y-axis, the ideal is the y = x line, and the observed line hangs below it by the skew.

$t_{skew} = t_{stream} - t_{event} > 0$

⚠️ Top pitfall: Processing by stream time when the question is about event time — windowed results shift by a non-constant amount; treating the skew as fixed; assuming arrival order equals event order.

Self-check: If an event happens at 12:00:04 and reaches the platform at 12:00:09, what is the time skew, and can another event that happened at 12:00:05 arrive before it?

Connects to: 11.5

Windows: Why Streams Need Them

Must-know: Windows are finite slices of an unbounded stream: the window length defines how much data each slice holds; the trigger policy decides when processing runs; the eviction policy decides when processed data is thrown away; and because only windowed samples are ever processed, streaming algorithms are non-deterministic while batch algorithms always see the full data set.

⚠️ Top pitfall: Confusing trigger (when processing runs) with eviction (when data is thrown away); expecting exact answers from a windowed algorithm; thinking the window waits for the stream to end.

Self-check: With the river-water mug, what do the mug, the water, the analysis, and throwing the water away each stand for?

Connects to: 11.4, 11.6

Tumbling Windows

Must-know: Tumbling window: T_batch = T_window; consecutive non-overlapping slices (12:00-12:05 gets e1,e2,e3; 12:05-12:10 gets e4,e5,e6); headline property: any event belongs to exactly one window; natural question is 'how many new orders in the last five minutes?'

$T_{batch} = T_{window}$

⚠️ Top pitfall: Counting a boundary event in two windows; using a tumbling window to compute a moving average (needs overlap); forgetting that the engine only runs at the trigger, so events arriving after a window closes wait for the next one.

Self-check: With five-minute tumbling windows starting at 12:00, which window does an event at 12:07 belong to?

Connects to: 11.5, 11.7

Sliding Windows

Must-know: Sliding window: T_window > T_slide; with a 5-minute slide and a 10-minute window, triggers fire at 12:00 (11:50-12:00), 12:05 (11:55-12:05), 12:10 (12:00-12:10), and each run's history overlaps the previous by T_window - T_slide; the event's own timestamp (event time) decides its window; the clean timeline assumes unit processing time.

$T_{window} > T_{slide}$

⚠️ Top pitfall: Sliding when the question is tumbling (new-orders counting gets complicated); assuming the slide must divide the window (5-minute slide with 7-minute window is fine); keying windows to stream time; ignoring that non-unit processing time shifts results by a non-constant delta.

Self-check: With a 5-minute slide and a 7-minute window, by how much do consecutive windows overlap?

Connects to: 11.6, 11.4

Exam Guidance Summary

Must-know: Quiz logistics: quiz next week covering everything so far, four-day window including the weekend (28th-29th), announcement on Canvas gives the official start; review output modes, the two timelines, trigger/eviction policies, and tumbling vs sliding windows with small timeline examples; remove the checkpoint directory before rerunning a streaming app.

⚠️ Top pitfall: Rerunning a streaming application without removing the old checkpoint directory — it resumes old state and the demo output does not restart cleanly.

Self-check: What must you do before rerunning a streaming query so that it starts fresh instead of resuming old state?

Connects to: 11.3, 11.4, 11.6, 11.7

Key Industry Applications

Must-know: Named anchors: netcat socket word count (interview starter); traffic charging (event time vs stream time); moving averages in stock prices and top-10 transactions over recent hours (sliding); new orders in the last five minutes (tumbling); water pollution sampling (window capacity and eviction); socket sources are development-only, not production.

⚠️ Top pitfall: Relying on raw socket feeds in production — they offer no durability, replay, or ordering guarantees.

Self-check: Which industry use case is the canonical sliding-window application mentioned in the session?

Connects to: 11.1, 11.4, 11.6, 11.7

Previous LectureLecture 10 · Spark Streaming: Architecture, Setup, DataFrames, and RDDs

Next LectureLecture 12 · Window Concepts and Streaming Algorithms

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

11.1 Spark Structured Streaming and the Streaming Word Count

11.1.1 Where We Start: Transformations, Actions, and the Two Environments

11.1.2 The SparkSession Singleton

11.1.3 The Data Source: Netcat and a TCP Socket

11.1.4 What the Application Does End to End

11.2 The Streaming Word Count Demo, Batch by Batch

11.2.1 Micro-Batches and How the Counts Roll In

11.2.2 Worked Example: Four Micro-Batches of Streaming Word Count

11.2.3 Student Questions and Answers

11.3 Output Modes: Complete, Update, and Append

11.3.1 Complete Mode

11.3.2 Worked Example: The Complete-Mode Rerun

11.3.3 Update Mode

11.3.4 Append Mode and the Watermark Requirement

11.3.5 Student Questions and Answers

11.4 Event Time, Stream Time, and Time Skew

11.4.1 Two Timelines: Event Time and Stream Time

11.4.2 Time Skew

11.5 Windows: Why Streams Need Them

11.5.1 The Window as a Lens over an Infinite Stream

11.5.2 Streaming Algorithms Are Non-deterministic

11.5.3 Trigger Policy

11.5.4 Eviction Policy

11.5.5 Student Questions and Answers

11.6 Tumbling Windows

11.6.1 Tumbling Window Mechanics

11.6.2 Worked Example: The Five-Minute Timeline

11.6.3 Key Properties

11.6.4 Use Cases for Tumbling Windows

11.6.5 Student Questions and Answers

11.7 Sliding Windows

11.7.1 What Sliding Windows Are For

11.7.2 Slide Interval and Window Interval

11.7.3 Worked Example: A Five-Minute Slide over a Ten-Minute Window

11.7.4 Student Questions and Answers

11.7.5 The Connection to What Comes Next

Exam Guidance Summary

Key Industry Applications

SPA Lecture 11 notes · Spark Structured Streaming: Word Counts, Output Modes, and Windows

Sections Breakdown

Exam Revision Notes

Spark Structured Streaming and the Streaming Word Count

The Streaming Word Count Demo, Batch by Batch

Output Modes: Complete, Update, and Append

Event Time, Stream Time, and Time Skew

Windows: Why Streams Need Them

Tumbling Windows

Sliding Windows

Exam Guidance Summary

Key Industry Applications

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat