Take a Break
5:00
Inhale…
Give your mind a break — no phone, no music, just idle time or a quick walk.
Structured Streaming and Spark Streaming
Prerequisite Knowledge
This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.
Previously Covered in This Subject
- Apache Spark DataFrames and the Catalyst Optimizer — covered in Lecture 10 (Apache Spark DataFrames, Catalyst Optimizer, PySpark Operations, and Spark SQL)
- Spark Session vs. Spark Context entry points — covered in Lecture 10 (Apache Spark DataFrames, Catalyst Optimizer, PySpark Operations, and Spark SQL)
- Explicit StructType schema definition — covered in Lecture 10 (Apache Spark DataFrames, Catalyst Optimizer, PySpark Operations, and Spark SQL)
- Streaming log ingestion with Flume — covered in Lecture 5 (Distributed Resource Management, SQL-to-Hadoop Ingestion, and Streaming Event Collection)
11.1 Recap of Spark Foundations and DataFrame Architecture
Before streaming engines make sense, the cluster must already speak Apache Spark's language: resilient partitions, tabular schemas, and a shared optimizer. This section rebuilds that foundation so later micro-batches and watermarks sit on solid ground.
11.1.1 RDD Low-Level APIs vs High-Level DataFrames
Hook: If Spark can already process terabytes with RDDs, why did the ecosystem invent DataFrames at all?
A Resilient Distributed Dataset (RDD) is an immutable, fault-tolerant collection of objects partitioned across cluster nodes. You transform RDDs with functional operators such as map and filter, and you materialize results with actions such as collect and count. RDDs carry no table schema: each partition holds raw Scala, Java, or Python objects, so the developer owns every type cast and every join strategy.
Intuition: Think of an RDD as a warehouse of unlabeled shipping crates. Workers can open every crate, inspect contents by hand, and move crates between docks. The work is flexible and parallel, but nobody has a printed inventory of column names—so automated route planning is weak. A DataFrame is the same warehouse after you paste a labeled packing list on every crate: named columns, typed fields, and a planner that can skip whole aisles.
Where the analogy breaks: crates do not automatically recover from dock fires; Spark RDDs rebuild lost partitions from lineage. Also, DataFrames still compile down to RDD plans under the hood—the labels do not replace resilience; they add optimization.
Spark's speed edge over Hadoop MapReduce comes from in-memory intermediate storage. MapReduce writes shuffle outputs to disk between stages. Spark keeps those intermediates in RAM when memory allows (spilling to disk only under pressure). Reference texts state that the in-memory Spark engine can deliver up to about \(100\times\) performance versus Hadoop for suitable workloads.
Formalize — speedup ratio. Let \(T_{\text{Hadoop}}\) be wall-clock latency of a disk-bound MapReduce job and \(T_{\text{Spark}}\) the latency of the equivalent in-memory Spark job. The theoretical speedup is \[ \text{Speedup} = \frac{T_{\text{Hadoop}}}{T_{\text{Spark}}} \approx 100\times \] Texts phrase this as "up to 100 times performance with respect to Hadoop." The ratio is an order-of-magnitude claim for iterative and memory-resident workloads, not a guarantee for every query (I/O-bound cold reads from HDFS still pay disk cost).
Worked example — speedup arithmetic. Suppose a nightly MapReduce aggregation finishes in \(T_{\text{Hadoop}} = 200\) minutes. The same logic on Spark completes in \(T_{\text{Spark}} = 2\) minutes. Then \[ \text{Speedup} = \frac{200}{2} = 100 \] Final answer: \(100\times\). Sense-check: cutting latency by two orders of magnitude matches the textbook "up to \(100\times\)" claim when intermediates stay in RAM.
Despite raw RDD speed, hand-written object pipelines demand high developer effort and skip automatic query planning. To serve data science teams used to Python Pandas tables, Spark introduced the DataFrame API: a distributed collection organized into named columns—a schema-aware layer over RDDs.
Q: Why did the Spark ecosystem evolve from RDDs to DataFrames? A: RDDs are low-level, un-schematized APIs. Developers write execution logic by hand with no Catalyst optimization. DataFrames give a Pandas-like tabular interface so engineers write declarative queries while the engine applies automated optimizations underneath.
Scope: The \(\approx 100\times\) figure assumes iterative or intermediate-heavy jobs that benefit from RAM caching. Single-pass scans of cold data on slow disks may show much smaller gains. RDDs remain available for custom low-level control; DataFrames are the default for structured analytics and streaming.
Visual intuition: plot job stage latency on the vertical axis (minutes) against stage index on the horizontal axis. A MapReduce timeline shows tall bars at every shuffle because each stage waits on disk. A Spark timeline shows short bars once data is cached—the landmark is the collapse of inter-stage wait time when RAM holds the shuffle.
Pitfalls:
- Treating "\(100\times\)" as a universal SLA rather than an upper-bound marketing and textbook figure.
- Assuming DataFrames abandon RDDs—Catalyst still lowers plans to RDD/Tungsten execution.
- Using RDDs for every new pipeline and missing schema + optimizer benefits.
- Confusing immutability of RDDs with "cannot transform"—you transform by creating new RDDs, not by mutating old ones.
Recap: RDDs give resilient parallel collections; DataFrames add named columns and optimizer-friendly structure. Bridge: Once queries are declarative, Catalyst can rewrite them—the next subsection.
Real-world & domain: Batch ETL teams that migrated MapReduce nightly jobs to Spark DataFrames routinely cut multi-hour Hadoop runs to minutes when aggregates fit in cluster memory—exactly the setting where the \(100\times\) narrative was born.
11.1.2 Catalyst Optimizer and Spark SQL Session Architecture
Hook: You write df.filter("city = 'Pune'").select("user_id") in Python—what actually runs on the executors?
DataFrames do not execute your Python or Scala loops line by line. Each transformation builds a logical plan. The Catalyst Optimizer analyzes that plan, applies rewrites such as predicate pushdown (filter early) and projection pruning (read only needed columns), then produces a physical plan of efficient operators. Spark SQL—an ANSI SQL interface—shares the same Catalyst pipeline, so spark.sql(...) and DataFrame calls converge on one engine. Texts describe Spark SQL as running query processing "using catalyst optimizer and tungsten" for structured data.
Intuition: Catalyst is the air-traffic controller for your query. You declare destinations ("filter city, keep user_id"); Catalyst chooses taxiways (push the filter into the Parquet scan, drop unused columns) so planes do not taxi across the whole airport unnecessarily.
Entry points in the driver program:
| Entry point | Typical variable | What it unlocks |
|---|---|---|
SparkContext |
sc |
Low-level RDD APIs only |
SparkSession |
spark |
DataFrames, Datasets, Spark SQL, Structured Streaming; encapsulates SparkContext |
Formalize — session layering. Modern applications start a single SparkSession. Internally it owns a SparkContext. Structured Streaming reads and writes hang off spark.readStream / spark.writeStream, so the streaming path rides the same Catalyst stack as batch DataFrames.
Q: What is the structural difference between SparkContext and SparkSession in a driver application? A: SparkContext (sc) is the legacy entry for unoptimized RDD work. SparkSession (spark) is the unified entry that wraps SparkContext and opens DataFrames, Spark SQL, and Structured Streaming under Catalyst.
Pitfalls: Creating both a standalone SparkContext and a separate SparkSession in one app and wondering why configs diverge; assuming Spark SQL is a different cluster product rather than another face of Catalyst.
Exam note: Identify sc → RDDs and spark → DataFrames / SQL / Structured Streaming. Expect questions that ask which object starts a streaming query.
Recap: Declarative DataFrame/SQL plans enter Catalyst; SparkSession is the front door. Bridge: Literature and version choice decide whether the APIs you study still match production Spark 3.x.
11.1.3 Literature and Ecosystem Versioning
Hook: A code sample that "worked in the book" can fail on today's cluster—why?
A primary reference is Spark: The Definitive Guide by Bill Chambers and Matei Zaharia (O'Reilly; Databricks / Spark co-creator lineage). Secondary books and blogs often freeze on Spark 2.1 or 2.2. Production estates commonly run Spark 3.3+. Structured Streaming watermark rules, continuous-mode maturity, and Catalyst improvements changed across those releases.
Professor warning: Do not buy or trust examples locked to Spark 2.1/2.2 without checking the target version. Prefer documentation and samples aimed at Spark 3.x so Catalyst, watermarking, and continuous execution behavior match what enterprises run.
Pitfalls: Copy-pasting DStreams tutorials as if they were current Structured Streaming; assuming inferSchema defaults match batch DataFrame habits when you later call readStream.
Recap: Match learning materials to Spark 3.x. Bridge: With foundations fixed, the lecture shifts from data at rest to data in motion.
Real-world & domain: Platform teams pin Spark minor versions in CI and reject blog snippets that still construct StreamingContext for greenfield pipelines—version hygiene is part of production reliability.
11.2 Fundamentals of Stream Processing vs Batch Processing
Batch systems wait for data to settle on disk. Streaming systems compute while data is still moving. Confusing a shorter cron schedule with true streaming is the classic trap this section removes.
11.2.1 Conceptual Distinction: Data at Rest vs Data in Motion
Hook: If you shrink a nightly batch job to run every five minutes, have you built a streaming platform?
Until now, big data designs centered on batch processing. Raw events land in persistent storage—HDFS, a NoSQL store, or a file directory—and sit at rest. On a schedule (daily, weekly, or every \(N\) hours), a job reads that frozen snapshot, aggregates it, and writes reports.
Stream processing instead ingests, transforms, and acts on data in motion as events arrive—often before long-term storage commits. Texts define a stream as a potentially unbounded, time-ordered sequence of data items; processing happens on pieces of recent data because the full history may never be available at once.
Intuition: Batch is like photographing a highway once a day and counting cars in the photo. Streaming is standing beside the road and counting cars as they pass. Shortening the photo interval (hourly photos) still leaves you with still images of data at rest; it does not put you on the roadside.
Where the analogy breaks: stream engines still may write results to storage; "in motion" describes the compute path, not a ban on disks.
Batch Processing: Incoming Data ---> Persistent Storage (Data at Rest) ---> Periodic Compute Job (e.g., 24h) ---> Reports
Stream Processing: Incoming Data ---> In-Memory Stream Buffer (Data in Motion) ---> Real-Time Compute Engine ---> Instant Output/AlertsThe architectural boundary is not "24 hours versus 10 minutes." It is whether computation reads static persisted data or continuously processes event streams.
Q: Is reducing a batch job execution window from 24 hours down to 5 minutes considered true stream processing? A: No. Frequency alone does not change the paradigm. Batch still reads data already at rest. True streaming operates on continuous event streams in motion before or as they reach storage layers.
Q: Can we "push it to a data lag" and still treat that as live batch/stream buffering? A: Several students used the phrase data lag. The professor corrected the vocabulary: the intermediate layer is a queue (in-memory message buffer such as Kafka), not a data lake. A data lake is persistent long-term storage. Confusing the two mixes the batch "at rest" path with the streaming buffer path.
Scope: Near-real-time micro-batch streaming still processes short tables of recent events; it remains streaming if the engine continuously drains a live buffer rather than only scheduling against a closed file dump. Hard real-time systems with fixed deadlines are a stricter class—stream processing texts note that streaming often has no fixed output deadline, only low-latency goals.
Visual intuition: on a timeline axis (hours), batch shows sparse vertical spikes at job start times over a flat "data sitting on HDFS" band. Streaming shows a continuous ribbon of events flowing through a buffer into compute—the landmark is uninterrupted flow, not spike frequency.
Pitfalls: Calling any sub-hour cron job "streaming"; storing everything in a lake first and claiming the architecture is streaming; using "data lag" when you mean "message queue."
Recap: At rest vs in motion is the defining split. Bridge: Real apps fuse both—fleet maps join live GPS to static profiles next.
11.2.2 Dynamic Streaming and Static Data Fusion: The Fleet Management Paradigm
Hook: Why does a cab app need three different data tempos on one map pin?
Production systems rarely pick pure batch or pure streaming. They compose three tempos:
- Dynamic streaming data: GPS coordinates from the driver's phone—high velocity, second-by-second, data in motion.
- Static data at rest: Cab registration, vehicle model, driver identity—slow-changing rows in databases or caches.
- Batch-aggregated data: Monthly trip counts, weekly earnings, rating averages—refreshed by nightly or weekly batch jobs.
When a rider opens the map, the platform performs on-the-fly joins: live GPS \(\bowtie\) static driver metadata \(\bowtie\) batch ratings → one vehicle marker.
Worked example — fleet fusion. At time \(t\), stream event (driver_id=D42, lat=18.52, lon=73.85) arrives. Static table row: (D42, name=Asha, car=Sedan). Batch table row: (D42, trips_30d=180, rating=4.8). Join on driver_id yields one UI record: Asha's Sedan at (18.52, 73.85) with rating 4.8. Final answer: one fused marker from three tempos. Sense-check: if GPS stalls, the pin freezes even though static/batch rows remain valid—streaming freshness owns the location field.
Assumption: Static and batch sides are treated as slowly changing dimensions; if driver metadata updates every second, that "static" join key itself becomes a stream and join semantics change.
Recap: Uber/Ola-style fleets prove streaming + static + batch coexist. Bridge: Industry patterns cluster into six canonical streaming use cases.
Real-world & domain: Ride-sharing and logistics control rooms render live fleets by joining mobile telemetry to reference stores—the same pattern later appears as streaming–static joins in Structured Streaming APIs.
11.2.3 The Six Canonical Streaming Use Case Categories
Hook: Fraud alerts, COVID dashboards, and Google Analytics counters feel different—do they share one taxonomy?
Per Spark: The Definitive Guide, industrial streaming apps fall into six patterns:
- Event notifications / real-time alerting — Watch logs or metrics; fire alerts (e.g., HTTP 500 spikes to SREs).
- Real-time reporting / live dashboards — Continuously refresh ops metrics (RPS, 5-minute error rate, revenue/minute, air-traffic trajectories).
- Real-time ETL — Parse and clean raw log streams into schematized lake tables (e.g., public pandemic trackers).
- Real-time data serving layers — Maintain pre-aggregated serving tables (active users, page views) so dashboards avoid
SELECT COUNT(*)over billions of raw rows. - Real-time decision / rule engines — Apply rules on the stream (auto buy/sell if price drops \(>10\%\)).
- Real-time ML (online inference / training) — Score features on arrival (card-swipe fraud) or update online model weights.
| Pattern | Typical output | Latency pressure |
|---|---|---|
| Alerting | Page/Slack/ticket | Seconds |
| Dashboards | Chart refresh | Seconds–minutes |
| ETL | Clean files/tables | Minutes acceptable |
| Serving layer | Key-value metrics | Sub-second reads |
| Decisions | Orders/actions | Sub-second |
| Online ML | Scores/alerts | Sub-second |
When to pick which: choose alerting/decisions/ML when missing a single event costs money or safety; choose ETL/serving when the goal is curated tables for many downstream readers.
Exam note: Classify a scenario into one of the six patterns (fraud alert vs log ETL vs analytics serving).
Pitfalls: Mixing "serving layer" with "run a heavy SQL count on click"; labeling every dashboard as ML; forgetting ETL is still streaming when it drains live logs into a lake.
Recap: Six named patterns organize exam scenarios. Bridge: Streams need a buffer between chaotic producers and compute—queuing infrastructure next.
Real-world & domain: Credit-card fraud sits in decisions/ML; IRCTC-scale booking bursts need queues before any of these six patterns can run safely.
11.3 Architectural Queuing Infrastructure and Buffer Layer Design
Producers surge and idle. Compute clusters cannot resize instantaneously. A queue sits between them so neither side dictates the other's clock.
11.3.1 The Necessity of Queuing Buffers
Hook: Can Spark pull events straight from the phone app or ticket API with no middle layer?
Production designs avoid wiring raw sources directly to compute. Generation rates are unpredictable: peaks can flood executors; troughs idle them. An intermediate queuing buffer decouples ingestion speed from processing capacity. Stream-processing texts flag frequency variation (festivals, evenings, viral spikes) as a first-class issue—queues absorb that variance.
Intuition: Without a queue, every booking click is a firehose aimed at the kitchen. With a queue, tickets land on a ticket rail; cooks pull at a sustainable pace.
Purpose: Protect consumers from producer burstiness, allow replay within retention, and let producers acknowledge writes quickly while Spark catches up asynchronously.
Scope: Queues add latency (milliseconds to seconds) and operational cost. Ultra-low-latency paths may still use short buffers, but skipping durable messaging entirely trades safety for speed.
11.3.2 Physical Analogies: The Water Tank and Solar Battery Storage Systems
Hook: What everyday systems already solve "spiky supply, smoother demand"?
Water tank analogy. A tap wired straight to a deep-well pump sees fluctuating well output and bursty household use. An overhead tank between pump and taps stores excess during quiet hours, avoids pressure spikes, and feeds peak showers from reserve.
Solar battery analogy. A \(10\,\text{kW}\) peak rooftop array at midday can exceed a \(5\,\text{kW}\) house load.
Formalize — excess power. \[ P_{\text{excess}} = P_{\text{solar}} - P_{\text{house}} \] with \(P_{\text{solar}}\) the instantaneous panel output, \(P_{\text{house}}\) domestic demand, and \(P_{\text{excess}}\) unconsumed power to buffer.
Worked example. Midday: \(P_{\text{solar}} = 10\,\text{kW}\), \(P_{\text{house}} = 5\,\text{kW}\). \[ P_{\text{excess}} = 10 - 5 = 5\,\text{kW} \] Final answer: \(5\,\text{kW}\) excess into the battery path. Sense-check: at night \(P_{\text{solar}} \approx 0\), the battery discharges so load still runs—same role Kafka plays when producers go quiet but consumers still drain backlog.
Panel → inverter (DC→AC) → battery bank → household circuits. Map to data:
Solar Panel (DC Gen) ---> Inverter (DC to AC) ---> Storage Battery (Buffer) ---> Household Electricals (Compute Load)
Raw Data Source ---> Message Queue ---> In-Memory Queue (Kafka) ---> Stream Compute Engine (Spark)Where analogies break: water/energy are continuous fluids; Kafka stores discrete ordered records with retention policies and consumer offsets.
Q: Why is an intermediate queuing buffer (like Apache Kafka) required between raw data sources and streaming compute engines? A: Direct source-to-compute links fail under variable load. The queue buffers spikes (e.g., 6 AM–10 PM IST traffic) and serves a regulated stream so compute neither crashes nor silently drops events.
Pitfalls: Treating the queue as optional "nice to have"; sizing retention too short so replay windows vanish; equating the queue with the data lake (persistent analytics store).
Exam note: Explain Kafka's necessity with water-tank and solar-battery analogies and the \(P_{\text{excess}}\) calculation.
11.3.3 Enterprise Queuing Engines: Apache Kafka Integration
Apache Kafka is a distributed, fault-tolerant, publish–subscribe commit-log system for high-throughput ingestion. Texts describe Kafka as capturing web activity, stock ticks, and instrumentation data with partitioned, replicated low-latency logs; Spark Streaming / Structured Streaming connect via Kafka sources and sinks (alongside Flume, Kinesis, sockets, files).
Kafka retains messages for a configurable window (commonly on the order of 7 days by default in many deployments). Spark can read topics and write results back to Kafka or other stores. ZooKeeper (or KRaft in newer Kafka) coordinates cluster metadata—mentioned in stack diagrams beside Kafka as the messaging backbone.
Assumption: Consumers must commit offsets correctly; at-least-once delivery plus non-idempotent sinks can still create duplicates unless exactly-once / idempotent patterns are designed (next section).
Recap: Queues decouple spiky producers from Spark. Bridge: Even with Kafka, streams face out-of-order arrival, state growth, and exactly-once demands.
Real-world & domain: IRCTC booking surges and financial transaction feeds place Kafka between frontends and Spark so peak ticket or trade bursts become drained micro-batches instead of cascading outages.
11.4 Operational Challenges in Stream Processing Systems
Queues solve burstiness. They do not erase late phones, growing state, duplicate debits, or the need to join live events to reference tables.
11.4.1 Out-of-Order Data Arrival and Event Time vs Processing Time
Hook: A violation happened at 1:30 PM but the server saw it at 2:00 PM—which clock should a fine use?
Two timestamps matter:
- Event time — when the fact occurred at the source (violation at 1:30 PM).
- Processing / ingestion time — when the engine saw the record (upload at 2:00 PM after a cellular outage).
Network latency, offline devices, and clock skew shuffle arrival order. Generated times 1, 2, 5, 10, 7 may arrive as 1, 2, 10, 5, 7. Correct analytics must reorder by event time, not arrival order.
Intuition: Imagine students submitting homework by email after a storm. Grading "as emails land" rewards whoever regained signal first; grading by the assignment's written timestamp restores true sequence.
Worked example — out-of-order traffic events. True event times: \(1, 2, 5, 10, 7\). Arrival order at Kafka: \(1, 2, 10, 5, 7\). A naive "last seen" detector might think \(10\) precedes \(5\). An event-time window that sorts or buffers until watermarks advance restores \(...,5,7,10\). Final answer: process by event time \(1\!\rightarrow\!2\!\rightarrow\!5\!\rightarrow\!7\!\rightarrow\!10\), not arrival order. Sense-check: processing-time plots would show a spike when the backlog flushes, falsely implying a real-world burst at 2:00 PM.
Scope: Event-time correctness needs embedded timestamps and watermark policies. Processing time suits ops metrics where "when we saw it" is the signal (e.g., requests/sec for autoscaling).
Visual intuition: plot event time on the \(x\)-axis and arrival time on the \(y\)-axis. In-order points lie near the diagonal; late events sit above it. The landmark is the vertical gap (lateness) for point \(7\) arriving after \(10\).
11.4.2 State Management, State Remembrance, and Memory Bounding
Hook: To know that \(2\) was followed by \(10\) then \(5\), what must the engine remember—and for how long?
Pattern detection needs state: running counts, last-seen keys, open windows. State remembrance asks how long keys live—1 hour, 24 hours, 30 days, forever? RAM is finite; unbounded state exhausts executors. Engines bound retention with time windows and watermarks that finalize and evict stale keys.
Q: What is state remembrance in stream processing, and why must it be bounded? A: It means keeping intermediate query variables (counts, sequences) across micro-batches in cluster memory. Because RAM is finite, retention must be capped with windows or watermarks to purge keys and avoid memory leaks.
Pitfalls: Infinite groupBy keys without watermarks; storing full raw history in state instead of aggregates; forgetting state rebuild cost after failure without checkpoints.
11.4.3 High Throughput, Scalability, and Exactly-Once Guarantees
Streaming platforms target three operational guarantees:
- High throughput & scalability — add commodity workers as ingestion grows without dropping messages.
- Exactly-once processing semantics — each event affects outputs once: not zero (loss) and not twice (duplication).
Credit-card duplicate debit. A retry re-delivers a ₹10,000 swipe. Without exactly-once or idempotent sinks, the cardholder sees two charges. Coordination uses transactional writes, idempotent keys, and checkpointed offsets.
Assumption: "Exactly-once" end-to-end requires sink support; source-side replay alone yields at-least-once unless outputs are idempotent or transactional.
Texts stress scalable processing for growing volume and near-real-time completion—matching these operational pillars.
11.4.4 Streaming-Static Data Joins and Output Diversity
Pipelines must:
- Join streaming DataFrames (in motion) to static reference DataFrames (at rest) — same fleet-map idea, now as an engine feature.
- Write diverse sinks — console, memory tables, files, Kafka, and beyond.
Recap: Order, state, exactly-once, and join/sink flexibility define streaming hardness. Bridge: Architects encode those needs into three design decisions next.
Real-world & domain: Automated traffic citations and card payments fail loudly when late events or duplicate processing corrupt money or legal records—so event-time and exactly-once emphasis.
11.5 System Design Decision Frameworks for Stream Architectures
Three knobs dominate stream architecture choice: how you program, which clock you trust, and how the engine schedules work.
11.5.1 Imperative Record-at-a-Time vs Declarative API Abstractions
Hook: Should your code see every event as a callback, or declare groupBy().avg() and let the engine manage buffers?
Decision dimension 1 — processing abstraction.
| Style | What the developer owns | Example engines |
|---|---|---|
| Record-at-a-time (imperative) | State, windows, buffers, recovery by hand | Apache Storm |
| Declarative API | Query intent (groupBy, count, avg); engine manages state/execution |
Spark Structured Streaming, Apache Flink |
Worked example — running average. In Storm-style imperative code you maintain counters and totals in bolt state, handle failures, and emit updates yourself. In Spark you write df.groupBy("sensor_id").avg("reading") on a streaming DataFrame; Catalyst plans stateful aggregation. Final answer: declarative APIs shrink boilerplate; imperative APIs maximize control at higher cost. Sense-check: exam contrasts often name Storm vs Spark/Flink on this axis.
Intuition: Imperative streaming is hand-driving every gear change; declarative streaming is setting "maintain 60 km/h" and letting cruise control manage throttle—until you need a maneuver cruise cannot express.
When to use / alternatives: Prefer declarative engines for analytics-style aggregations and SQL familiarity. Prefer record-at-a-time when ultra-custom per-event logic dominates and team expertise is already Storm-shaped.
11.5.2 Temporal Paradigm Selection: Event Time vs Ingestion Processing Time
Decision dimension 2 — temporal alignment.
- Event time: Evaluate using timestamps inside the payload. Required when correctness tracks real-world order (cardiac monitors, traffic tickets).
- Processing (ingestion) time: Evaluate using the compute node's clock at intake. Fine for ops telemetry (HTTP RPS → VM autoscaling) where "seen by the platform" is the metric.
Comparison: pick event time when late data must still land in the correct historical window; pick processing time when simplicity beats chronological fidelity.
Pitfalls: Building legal or clinical timelines on processing time; ignoring embedded device clocks that are skewed without synchronization.
11.5.3 Execution Engine Models: Continuous Processing vs Micro-Batching
Decision dimension 3 — execution model.
- Continuous processing: Each record (or tiny group) runs with sub-millisecond end-to-end latency; often needs dedicated threads per partition.
- Micro-batching: Collect events for a short interval \(N\) (e.g., \(100\,\text{ms}\)–\(2\,\text{s}\)), form a micro-batch table, run the batch engine. Texts note SparkStreaming divides streams into micro-batches of \(N\) seconds; very small \(N\) may starve each batch of useful data volume.
Spark Structured Streaming defaults to high-throughput micro-batching and also offers an experimental continuous mode.
Continuous Processing: Incoming Event ---> Instant Thread Execution ---> Sub-millisecond Latency
Micro-Batching: Incoming Events (Interval: e.g., 2s) ---> Micro-Batch Table ---> Spark Engine ---> Result SinkQ: What is the difference between continuous processing and micro-batching in stream processing engines? A: Continuous mode evaluates records as they arrive for sub-millisecond latency at higher resource cost. Micro-batching bundles records over short intervals (about \(100\,\text{ms}\) to \(2\,\text{s}\)) into tables and reuses optimized batch execution for maximum throughput.
Scope: Micro-batch latency is bounded below by the trigger interval; continuous mode trades engineering maturity and resource use for lower latency.
Exam note: Be ready to contrast Storm vs Spark/Flink (abstraction), event vs processing time, and continuous vs micro-batch.
Recap: Abstraction, clock, and scheduler form the design triad. Bridge: Spark itself walked from DStreams to Structured Streaming along these axes.
Real-world & domain: Autoscaling web tiers often tolerate processing-time micro-batches; hospital monitors and ticket cameras demand event-time correctness even when phones reconnect late.
11.6 Architectural Evolution of Spark Streaming: DStreams vs Structured Streaming
Spark's streaming story has two generations: RDD micro-batches without an optimizer (DStreams), then DataFrame/SQL streaming with Catalyst, event time, and watermarks (Structured Streaming).
11.6.1 Legacy DStreams Architecture (2012)
Hook: How did Spark do streaming before DataFrames existed?
Discretized Streams (DStreams), introduced in 2012, model a stream as a sequence of RDD micro-batches. Texts describe SparkStreaming as processing a DStream—a series of RDDs—from sources such as Kafka, Flume, Kinesis, files, or sockets, with operators like map, reduceByKey, updateStateByKey, and windowed variants.
Key traits:
- Built on low-level RDD APIs; elements are raw language objects.
- No Catalyst—efficiency depends on manual coding and memory discipline.
- Micro-batching only (no continuous mode).
- Processing-time windowing only—no native event-time or watermarking for late data.
Purpose (procedural view): Turn an unbounded input into repeating RDD batches so the core Spark engine can run familiar transformations in near real time.
11.6.2 Modern Structured Streaming Architecture (2016 / Spark 2.2+)
Hook: What changed when streaming became "just another DataFrame query"?
Structured Streaming (2016; Spark 2.2+) supersedes DStreams for new work:
- Built on Spark SQL / DataFrame / Dataset APIs.
- Schematized tables under Catalyst (and Tungsten execution).
- High-level ops (
groupBy,filter, …) compile to optimized plans. - Micro-batch default plus experimental continuous mode.
- Native event-time processing, late-data handling, and watermarks.
Intuition: DStreams are handwritten assembly on conveyor RDDs. Structured Streaming is SQL on an endless table—the optimizer and watermark clock come free with the abstraction.
11.6.3 Comparative Feature Matrix and Industrial Adoption
Evolutionary comparison.
| Architectural Feature | Legacy DStreams (2012) | Modern Structured Streaming (2016+) |
|---|---|---|
| Primary Abstraction API | Low-level RDDs (DStream) |
High-level DataFrames / Datasets |
| Optimization Engine | None (manual tuning) | Catalyst & Tungsten |
| Temporal Support | Processing time only | Event time & watermarking |
| Execution Model | Micro-batching only | Micro-batching & continuous mode |
| Schema Awareness | Unschematized objects | Explicit tabular schema |
| Ecosystem Status | Legacy / maintenance | Industrial standard |
Final answer: greenfield pipelines choose Structured Streaming. Sense-check: textbooks still teach DStream APIs for SparkStreaming history, but production guidance treats Structured Streaming as current.
Pitfalls: Following outdated DStream tutorials for new projects; assuming DStreams magically gained event-time watermarks; mixing StreamingContext and SparkSession readStream patterns without clarity.
Exam note: Contrast DStreams (RDD, processing time, micro-batch only, manual tuning) with Structured Streaming (DataFrames, Catalyst, event time, watermarking, micro-batch + continuous).
Recap: Structured Streaming is the successor API. Bridge: Next—the unbounded-table programming model and concrete readStream code.
Real-world & domain: Enterprises keep DStreams only for legacy codepaths; new Uber-scale style telemetry pipelines standardize on Structured Streaming over Kafka.
11.7 Structured Streaming Programming Model and API Implementation
Structured Streaming asks you to pretend a stream is an ever-growing table, then runs your DataFrame query continuously on each micro-batch.
11.7.1 The Unbounded Table Abstraction
Hook: How can one groupBy query serve both a finite Parquet folder and an infinite Kafka topic?
The unbounded table: An input stream is an infinitely growing, append-only table. Each new event is a new row. You write ordinary DataFrame transformations; Spark turns them into a continuous plan over micro-batches.
Intuition: Imagine a spreadsheet that never stops gaining rows at the bottom. Your pivot table formula stays the same; the engine refreshes it as rows appear. The analogy breaks when rows must be updated or deleted—output modes and watermarks define those rules later.
Stream Ingestion: Event 1, Event 2, Event 3...
Unbounded Table: +--------------------+--------------------+
| Row ID | Event Payload |
+--------------------+--------------------+
| Row 1 | Event 1 Payload |
| Row 2 | Event 2 Payload |
| Row 3 (Appended) | Event 3 Payload |
+--------------------+--------------------+11.7.2 Explicit Schema Enforcement and ReadStream Configuration
Hook: Batch jobs happily call inferSchema=True—why does readStream refuse by default?
Static reads (spark.read) may infer schemas by scanning files. Streaming reads (spark.readStream) disable automatic schema inference by default. Live sources (IoT firmware v1 with 8 columns vs v2 with 10) can mutate; inferring on an endless stream is expensive and unsafe. Developers must supply a StructType up front.
Trace — activity sensor pipeline setup.
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, LongType
# Explicit schema definition for activity sensor JSON streams
userSchema = StructType([
StructField("Arrival_Time", LongType(), True),
StructField("Creation_Time", LongType(), True),
StructField("Device", StringType(), True),
StructField("Index", LongType(), True),
StructField("Model", StringType(), True),
StructField("User", StringType(), True),
StructField("gt", StringType(), True), # Ground-truth activity (e.g., walking, standing)
StructField("x", DoubleType(), True),
StructField("y", DoubleType(), True),
StructField("z", DoubleType(), True)
])
# Initializing streaming DataFrame read
streamingDF = (
spark.readStream
.schema(userSchema)
.option("maxFilesPerTrigger", 1) # Reads 1 file per micro-batch trigger for testing
.json("cloud_data/activity_data")
)Inputs: JSON files under cloud_data/activity_data, explicit userSchema, maxFilesPerTrigger=1 for slow testing. Output: a streaming DataFrame with columns including gt, x, y, z. Sense-check: without .schema(...), Spark should refuse or fail inference rather than silently guessing types mid-stream.
Q: Why does Spark Structured Streaming require explicit schema definition (StructType) instead of auto-inferring schema? A: External streams (IoT, apps) may change shape across versions. Auto-inference on continuous data is costly and unsafe. Explicit StructType locks consistency across triggers.
Exam note: readStream requires explicit schema; do not rely on batch inferSchema habits.
11.7.3 Streaming Transformations, Aggregations, and WriteStream Execution
Steps (procedural):
- Build
streamingDFwith schema + source options. - Apply relational ops identical to batch (
filter,groupBy, …). - Configure
writeStream(name, format, output mode). .start()the background query;.awaitTermination()so the driver stays alive.
# Grouping and aggregation transformation on streaming DataFrame
activityCounts = streamingDF.groupBy("gt").count()# Initializing WriteStream execution pipeline
activityQuery = (
activityCounts.writeStream
.queryName("activity_counts") # In-memory table identifier
.format("memory") # Output sink destination (RAM memory)
.outputMode("complete") # Output mode: write full aggregated table
.start() # Begins background micro-batch execution
)
# Blocking driver program to prevent process termination
activityQuery.awaitTermination()Without awaitTermination() in a standalone app, the main thread exits and tears down the async streaming job. Texts similarly show streamingContext.awaitTermination() for classic SparkStreaming—same driver-lifetime idea.
Pitfalls: Forgetting awaitTermination; using complete mode on sinks that cannot rewrite full tables; assuming transformations differ between batch and streaming DataFrames (they share API names; supported ops and modes still have rules).
11.7.4 Streaming-Static Joins and Interactive Sink Querying
With format("memory"), query the live result table:
spark.sql("SELECT * FROM activity_counts").show()Re-running every few seconds shows counts as new files arrive. Join streams to static reference data:
staticMetadataDF = spark.read.json("cloud_data/static_metadata")
joinedStreamingDF = streamingDF.join(staticMetadataDF, "Device", "inner")This is the fleet-management join expressed in API form: motion \(\bowtie\) rest.
Recap: Unbounded tables + explicit schemas + writeStream lifecycle implement Structured Streaming. Bridge: Sources, sinks, and output modes (complete/update/append) plus watermarks finish the picture.
Real-world & domain: Phone accelerometer JSON (x,y,z + ground-truth activity labels) is a teaching stand-in for wearable and IoT activity recognition pipelines that must schema-lock device firmware fields.
11.8 Input Sources, Output Sinks, and Output Modes
Sources feed the unbounded table; sinks and output modes decide what leaves each trigger; watermarks decide when state may die.
11.8.1 Classification of Ingestion Sources and Destination Sinks
Hook: Where can events enter Spark, and where can results go?
Sources:
- File source — watch a directory for new JSON/CSV/ORC/Parquet files.
- Socket source — UTF-8 TCP text (local tests).
- Apache Kafka source — subscribe to topics for production velocity.
Sinks:
- Console —
format("console")to stdout (debug). - Memory —
format("memory")interactive SQL table (debug). - File sinks — JSON/CSV/Parquet/Avro directories.
- Kafka sink — fan-out to databases and apps via Kafka connectors.
Texts list basic sources (filesystem, sockets) and advanced sources (Kafka, Flume, Kinesis) for the older SparkStreaming stack—the same source taxonomy carries into Structured Streaming connectors.
Scope: Console and memory sinks are for development, not durable production serving.
11.8.2 The Three Output Modes: Complete, Update, and Append
Hook: After each micro-batch, should Spark rewrite the whole answer, only changed rows, or only brand-new finalized rows?
Complete Mode: Write FULL updated aggregated result table every trigger.
Update Mode: Write ONLY rows that CHANGED/UPDATED since last trigger.
Append Mode: Write ONLY FINALIZED new rows appended since last trigger (No row updates).- Complete (
outputMode("complete")) — Emit the entire result table each trigger. Natural for live aggregations (leaderboards) that rewrite counts. - Update (
outputMode("update")) — Emit only rows that changed since the last trigger. - Append (
outputMode("append")) — Emit only new finalized rows; once written, a row will not be revised later.
Q: Contrast Complete, Update, and Append output modes in Spark Structured Streaming. A: Complete outputs the full updated table every trigger (aggregations). Update outputs only changed rows. Append outputs only newly finalized rows that will never be modified again (immutable log-style writes).
| Mode | Emits | Typical use |
|---|---|---|
| Complete | All aggregation keys | Live totals dashboard |
| Update | Changed keys only | Efficient partial refresh |
| Append | Finalized new rows only | Immutable event/history tables |
When to pick which: complete when consumers need a full snapshot; update when bandwidth matters; append when sinks are append-only and keys must be finalized first (often with watermarks).
Pitfalls: Using append on aggregations without watermarks (rows never finalize); using complete on huge key spaces every second; assuming every sink supports every mode.
11.8.3 Watermarking Mechanics and State Bounding
Hook: How long may an NDTV headline keep changing before the archive locks it?
Scenario — news title updates. At 10:00 AM, Content_ID = 1 titled "Bangalore airport witnesses massive crowd ahead of Diwali." At 11:00 AM the title becomes "Traffic jam in Bangalore airport," later "Bangalore airport traffic choked." If updates stop mattering after 24 hours, set a 24-hour watermark.
Formalize — watermark threshold. \[ T_{\text{watermark}} = t_{\text{max}} - \Delta t_{\text{allowed}} \] where \(t_{\text{max}}\) is the maximum event timestamp seen so far and \(\Delta t_{\text{allowed}}\) is the allowed lateness (here \(24\) hours). Events older than the watermark are late beyond policy; associated state can finalize and evict.
Watermark Window (24h): [ 10:00 AM ------------ Updates Allowed ------------ 24 Hours Later ]
State Finalization: After 24h threshold expires ---> Output Finalized Title (Append Mode) ---> Purge KeyInside the window, title versions stay in state. When the watermark passes, Spark emits the latest title for Content_ID = 1 in Append mode and drops the key from RAM. Later updates are discarded.
Worked example — 24-hour append lifecycle.
- 10:00 — insert state
(1, title=A). - 11:00 — update state
(1, title=B). - Later — update
(1, title=C). - When \(t_{\text{max}} - t_{\text{event}} > 24\,\text{h}\) for this key's window, emit
(1, title=C)once in append mode; purge state.
Final answer: one finalized append of title C after the watermark; further edits dropped. Sense-check: this is how append mode stays compatible with aggregations/updates—emit only after finalization.
Q: Please explain the previous slide again on the white canvas—how does append mode work with the news title updates? A: In append mode, output is produced ONLY when a record key is finalized. Once a key's final value is emitted at the end of the watermark threshold (e.g., 24 hours for NDTV news title updates), subsequent changes for that key are discarded.
Assumption: Event timestamps must be present and roughly trustworthy. Too-tight watermarks drop legitimate late phones; too-loose watermarks bloat state.
Visual intuition: draw event time rightward and a moving watermark line at \(t_{\text{max}} - 24\,\text{h}\). Keys left of the line are closed; keys to the right remain open for edits. Takeaway: the watermark is a moving fence that trades lateness tolerance for memory.
Exam note: Know complete vs update vs append, and \(T_{\text{watermark}} = t_{\text{max}} - \Delta t_{\text{allowed}}\) as the state-bounding rule for append finalization.
Recap: Sources/sinks plus output modes and watermarks complete Structured Streaming operations. Bridge: Use the exam summary and industry list to revise cross-cutting themes.
Real-world & domain: Newsrooms and CMS pipelines watermark content IDs so live headline edits settle into immutable archives without unbounded state growth—the same mechanism bounds IoT session state in Spark.
Exam Guidance Summary
- Spark vs Hadoop performance ratio. Memorize
\[ \text{Speedup} = \frac{T_{\text{Hadoop}}}{T_{\text{Spark}}} \approx 100\times \] Spark wins mainly by keeping intermediates in RAM instead of disk-bound MapReduce stage I/O. Treat \(100\times\) as an order-of-magnitude upper claim for suitable workloads.
- Context vs session entry points.
SparkContext(sc) → low-level RDDs.SparkSession(spark) → DataFrames, Spark SQL, Structured Streaming under Catalyst.
- Batch vs stream boundary. Streaming = data in motion; batch = data at rest. Shortening cron intervals does not create streaming. Do not call a message queue a data lake.
- Six canonical streaming use cases. Classify scenarios into: Event Alerts, Real-time Reporting/Dashboards, Real-time ETL, Real-time Serving Layers, Decision Engines/Rules, Real-time ML Inference.
- Queuing buffers and analogies. Explain Kafka with the water-tank (well → tank → tap) and solar-battery (\(P_{\text{excess}} = 10\,\text{kW} - 5\,\text{kW} = 5\,\text{kW}\)) stories.
- Design decision dimensions. Record-at-a-time (Storm) vs Declarative (Spark/Flink); Event Time vs Processing Time; Continuous (sub-ms) vs Micro-Batch (\(100\,\text{ms}\)–\(2\,\text{s}\) tables).
- DStreams vs Structured Streaming. DStreams (2012): RDDs, processing time only, micro-batch only, manual tuning. Structured Streaming (2016+): DataFrames, Catalyst, event time, watermarks, micro-batch and continuous modes.
- Schema on
readStream. AutoinferSchemais off by default; supply explicitStructTypeso IoT/version drift cannot silently break types.
- Output modes. Complete = full table each trigger; Update = changed rows; Append = newly finalized immutable rows.
- Watermarking. \(T_{\text{watermark}} = t_{\text{max}} - \Delta t_{\text{allowed}}\) bounds state and decides when Append mode may emit (e.g., 24-hour NDTV title finalization).
Key Industry Applications
- Fleet tracking and logistics (Uber/Ola): Join live mobile GPS streams with static driver profiles and batch-calculated ratings for map markers.
- Credit card fraud detection: Score swipes with rules and ML in real time to raise instant alerts (decision + online ML use cases).
- Site reliability and log analytics: Watch server logs and page SREs when HTTP 500 counts cross thresholds (alerting).
- Live air traffic control: Continuous dashboards of aircraft/rocket telemetry (real-time reporting).
- Automated renewable energy management: Buffer \(10\,\text{kW}\) solar peaks through inverter–battery banks against \(5\,\text{kW}\) household load—physical twin of Kafka buffering.
- High-velocity message buffering: Kafka retaining on the order of days of logs (commonly ~7-day defaults) between IRCTC-scale frontends and Spark clusters.
- Media ingestion and activity tracking: Stream JSON accelerometer axes \(x,y,z\) to classify walking, standing, sleeping, cycling in real time.
- News publishing and content lifecycle: Apply 24-hour watermarked append windows to NDTV-style title edits before archiving finalized records.
BDA Lecture 11 notes
Sections Breakdown
RDDs are resilient unschematized partitions; DataFrames add named columns and Catalyst optimization. Covers the ~100x in-memory speedup over Hadoop MapReduce, SparkContext vs SparkSession entry points, and Spark 3.x versioning.
Streaming computes on data in motion; batch on data at rest. Covers fleet management fusion of live GPS with static and batch data, and the six canonical streaming use-case patterns.
Intermediate queues (Kafka) decouple unpredictable producers from compute. Water-tank and solar-battery analogies plus P_excess = 10 - 5 = 5 kW formalize the buffering idea.
Event time vs processing time, bounded state remembrance, exactly-once semantics, and streaming-static joins define streaming operational hardness.
Three decision dimensions: imperative record-at-a-time vs declarative APIs, event time vs processing time, and continuous vs micro-batch execution models.
DStreams (2012) are RDD micro-batches with processing-time only; Structured Streaming (2016+) adds DataFrames, Catalyst, event time, watermarks, and optional continuous mode.
Streams as unbounded append-only tables; readStream requires explicit StructType; groupBy/count with writeStream memory sink and awaitTermination; streaming-static joins.
File/socket/Kafka sources and console/memory/file/Kafka sinks; complete/update/append output modes; watermark T = t_max - delta bounds state and enables append finalization.
Ten consolidated exam bullets covering speedup ratio, session vs context, batch vs stream, six use cases, Kafka analogies, design triad, DStreams vs Structured Streaming, schema, output modes, and watermarking.
Industry mappings from fleet GPS fusion and fraud alerts to Kafka buffering, IoT activity JSON, and watermarked news title archives.
Exam Revision Notes
Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.
Recap of Spark Foundations and DataFrame Architecture
Must-know: SparkSession (spark) is the unified entry for DataFrames/SQL/streaming; SparkContext (sc) is RDD-only; ~100x speedup comes from in-memory intermediates vs MapReduce disk.
\[\text{Speedup} = \frac{T_{\text{Hadoop}}}{T_{\text{Spark}}} \approx 100\times\]
⚠️ Top pitfall: Treating 100x as a universal guarantee, or using outdated Spark 2.1/2.2 docs for Structured Streaming APIs.
Self-check: Which object starts Structured Streaming: sc or spark?
Connects to: Fundamentals of Stream Processing vs Batch Processing; Architectural Evolution of Spark Streaming: DStreams vs Structured Streaming; Structured Streaming Programming Model and API Implementation
Fundamentals of Stream Processing vs Batch Processing
Must-know: Shorter batch schedules are still batch; streaming = data in motion. Classify scenarios into the six Definitive Guide use cases.
⚠️ Top pitfall: Calling a 5-minute cron job streaming, or calling a message queue a data lake/lag.
Self-check: Name the six canonical streaming use-case categories.
Connects to: Architectural Queuing Infrastructure and Buffer Layer Design; Operational Challenges in Stream Processing Systems; Structured Streaming Programming Model and API Implementation
Architectural Queuing Infrastructure and Buffer Layer Design
Must-know: Kafka buffers peak load; explain with water tank and solar battery; compute P_excess = P_solar - P_house.
\[P_{\text{excess}} = P_{\text{solar}} - P_{\text{house}} = 10\,\text{kW} - 5\,\text{kW} = 5\,\text{kW}\]
⚠️ Top pitfall: Wiring sources directly to Spark, or confusing queues with data lakes.
Self-check: Why insert Kafka between IRCTC frontends and Spark?
Connects to: Fundamentals of Stream Processing vs Batch Processing; Operational Challenges in Stream Processing Systems; Input Sources, Output Sinks, and Output Modes
Operational Challenges in Stream Processing Systems
Must-know: Reorder by event time; bound state with watermarks; aim for exactly-once to avoid duplicate side effects.
⚠️ Top pitfall: Using arrival order as truth, or retaining unbounded state in RAM.
Self-check: Events arrive 1,2,10,5,7 — which order is correct for event-time analytics?
Connects to: System Design Decision Frameworks for Stream Architectures; Input Sources, Output Sinks, and Output Modes
System Design Decision Frameworks for Stream Architectures
Must-know: Storm = record-at-a-time; Spark/Flink = declarative. Event time for real-world order; micro-batch is Spark's default high-throughput mode.
⚠️ Top pitfall: Equating micro-batch with batch-at-rest architecture, or using processing time for legally sensitive timelines.
Self-check: Name the three design decision dimensions for stream architectures.
Connects to: Operational Challenges in Stream Processing Systems; Architectural Evolution of Spark Streaming: DStreams vs Structured Streaming
Architectural Evolution of Spark Streaming: DStreams vs Structured Streaming
Must-know: DStreams = RDD + processing time + micro-batch only; Structured Streaming = DataFrame + Catalyst + event time + watermarks.
⚠️ Top pitfall: Treating DStreams as the current recommended API for new projects.
Self-check: Which Spark streaming API supports native watermarking?
Connects to: Recap of Spark Foundations and DataFrame Architecture; System Design Decision Frameworks for Stream Architectures; Structured Streaming Programming Model and API Implementation
Structured Streaming Programming Model and API Implementation
Must-know: readStream needs explicit StructType; awaitTermination keeps the driver alive; memory sink tables are SQL-queryable.
⚠️ Top pitfall: Relying on inferSchema for streaming sources, or exiting the driver without awaitTermination.
Self-check: Why is schema inference disabled by default on readStream?
Connects to: Fundamentals of Stream Processing vs Batch Processing; Architectural Evolution of Spark Streaming: DStreams vs Structured Streaming; Input Sources, Output Sinks, and Output Modes
Input Sources, Output Sinks, and Output Modes
Must-know: Complete = full table; Update = changed rows; Append = finalized rows only. Watermark bounds state and enables append finalization.
\[T_{\text{watermark}} = t_{\text{max}} - \Delta t_{\text{allowed}}\]
⚠️ Top pitfall: Using append without watermarks so aggregations never finalize, or using complete on enormous key spaces every trigger.
Self-check: In the NDTV example, when is the title row written in append mode?
Connects to: Operational Challenges in Stream Processing Systems; Structured Streaming Programming Model and API Implementation
Exam Guidance Summary
Must-know: Memorize the ten exam bullets: 100x speedup, sc vs spark, at-rest vs in-motion, six use cases, Kafka analogies, three design dimensions, DStreams vs Structured Streaming, StructType, three output modes, watermark formula.
\[T_{\text{watermark}} = t_{\text{max}} - \Delta t_{\text{allowed}}\]
⚠️ Top pitfall: Confusing shorter batch schedules with streaming, or DStreams with Structured Streaming.
Self-check: List the three stream architecture decision dimensions.
Connects to: Recap of Spark Foundations and DataFrame Architecture; Fundamentals of Stream Processing vs Batch Processing; Architectural Queuing Infrastructure and Buffer Layer Design; System Design Decision Frameworks for Stream Architectures; Architectural Evolution of Spark Streaming: DStreams vs Structured Streaming; Structured Streaming Programming Model and API Implementation; Input Sources, Output Sinks, and Output Modes
Key Industry Applications
Must-know: Map each industry example to a streaming concept: fleet joins, fraud ML, Kafka buffers, watermarks for news titles.
⚠️ Top pitfall: Memorizing brand names without linking them to the underlying streaming pattern.
Self-check: Which application illustrates watermarked append mode?
Connects to: Fundamentals of Stream Processing vs Batch Processing; Architectural Queuing Infrastructure and Buffer Layer Design; Input Sources, Output Sinks, and Output Modes