Spark Streaming Output Modes, Watermarking, and Machine Learning Fundamentals

12.1 Spark Streaming Architecture and Processing Models

Stream processing is designed to analyze data while it is in motion, before it comes to rest in a persistent database or storage lake. In real-time data architectures, stream processing handles continuous, unbounded streams of incoming records across six major classification categories: sensor data monitoring, financial fraud detection, real-time metrics dashboards, online recommendation engines, traffic management, and security intrusion detection.

12.1.1 Batch Processing vs. Stream Processing

The fundamental distinction between batch processing and stream processing lies in the state of the data during execution:

• Batch processing: Computes transformations on historical, bounded datasets that have already reached rest in disk-based storage engines (such as HDFS, S3, or distributed file systems).
• Stream processing: Computes transformations dynamically on unbounded, real-time data streams in memory before the data arrives at rest.

When to pick which: if you can wait for the answer and you need the entire dataset in the computation, batch; if the answer must be current and data never stops arriving, stream.

In complex enterprise pipelines, stream processing is rarely used in total isolation. Production systems frequently execute stream-to-static joins, where live event streams are joined against static dimension tables stored at rest (for example, joining a live stream of transaction user IDs with a static user profile database).

12.1.2 Evolution of Spark Streaming APIs

Apache Spark evolved its stream processing model across two distinct API paradigms:

1. D-Streams (Discretized Streams API): The legacy low-level streaming API built on top of resilient distributed datasets (RDDs). D-Streams process continuous data streams as sequence chunks of mini-batches over small time intervals. However, D-Streams require complex RDD-level code, explicit state management, and separate codebases for batch and streaming pipelines.
2. Structured Streaming API: The modern high-level streaming engine built directly on the Spark SQL and DataFrame framework. Structured Streaming treats a live data stream as an unbounded input table. As new data arrives, new rows are continuously appended to the unbounded table. Developers interact with stream data using standard DataFrame transformations and SQL queries.

A key advantage of Structured Streaming is unified API parity: code written for batch processing can be converted into stream processing by simply altering the entry reader interface from spark.read to spark.readStream. The downstream transformation, aggregation, and filtering logic remain identical.

12.1.3 Integration with Message Brokers

In modern big data architectures, Apache Kafka serves as the primary ingestion backbone for stream processing. Kafka clusters act as decoupled distributed message buffers, connecting hundreds of heterogeneous data sources to downstream compute engines. Spark Structured Streaming integrates seamlessly with Kafka, consuming partitions in parallel and enabling seamless end-to-end fault tolerance, exact-once processing semantics, and scalable stream ingestion.

Real-world: Apache Kafka is the de facto standard streaming ingestion buffer in industry, feeding live events into Spark Structured Streaming for fraud detection, metrics aggregation, and clickstream analytics. The exact-once semantics matter in finance: if a payment event is processed twice, money is double-counted; Kafka plus Structured Streaming's checkpointing is designed to prevent exactly that.

12.2 Structured Streaming Output Modes

When writing the results of streaming aggregations or queries to external sinks (such as Kafka, databases, or file storage), Structured Streaming provides three distinct output modes. The output mode defines how much accumulated state and which result rows are written to the sink during each trigger execution.

12.2.1 Complete Output Mode

In Complete Mode, the entire updated result table is written to the sink at every trigger interval. The engine retains all historical aggregate state across all micro-batches and re-emits the full cumulative table state upon receiving new data.

• Use case: Complete mode is ideal for real-time summary dashboards or metrics counters where downstream clients require a full state refresh on every query execution (e.g., displaying the top 10 running counters on a web dashboard).
• Resource implications: Because state accumulates continuously and the entire table is outputted repeatedly, complete mode requires sufficient memory to store all unique key states over time.

12.2.1.1 Worked Computation — Complete Mode Word Count Walkthrough

Consider a real-time word counting pipeline configured in Complete Mode across three consecutive micro-batch trigger timestamps \(t_1, t_2, t_3\):

1. At Timestamp \(t_1\): Input micro-batch contains text records (cat: 1, dog: 3).

• Engine maintains internal aggregate table state.
• Output written to Sink:

1. At Timestamp \(t_2\): New input micro-batch arrives with records (owl: 1, cat: 1).

• The engine updates the internal aggregate count for cat (\(1 + 1 = 2\)) and adds new entry owl: 1.
• Output written to Sink (entire cumulative table state is re-written, including unchanged dog count):

1. At Timestamp \(t_3\): New input micro-batch arrives with records (dog: 1, owl: 1).

• The engine updates dog count (\(3 + 1 = 4\)) and owl count (\(1 + 1 = 2\)). cat receives no new occurrences in this micro-batch.
• Output written to Sink (the full updated table is written once again):

12.2.2 Update Output Mode

In Update Mode, only the specific rows that have been modified or added since the previous trigger execution are written to the sink. Rows that remain unchanged across micro-batches are omitted from the output.

• Use case: Update mode is optimal when transmitting full table updates is computationally expensive or bandwidth-prohibitive. For example, tracking user login/logout events across thousands of users where only active users updated during the last trigger window need to be published to a downstream database.
• Behavioral distinction: If an existing key's aggregate value does not change during a trigger interval, that key is excluded from the current micro-batch output.

12.2.2.1 Worked Example — User Activity Tracking in Update Mode

Consider a streaming system maintaining user activity status (user_id, last_login_time):

• At trigger \(t_1\), users Avinash and Tushar log in. Update mode outputs rows for Avinash and Tushar.
• At trigger \(t_2\), user Avinash logs in again, while user Amar (who logged in yesterday) shows no activity.
• Output written to Sink: Only the updated record for Avinash is written. Amar's unchanged state from yesterday is omitted from the sink output, conserving I/O overhead.

12.2.3 Append Output Mode

In Append Mode, only finalized new rows that will never be updated or modified by future data are written to the sink.

• Use case: Append mode is required for file-based sinks (such as Parquet, ORC, or JSON files) where existing files cannot be overwritten or updated in place.
• Finalization constraint: Once a row is emitted in Append Mode, its contents are locked. To determine when a windowed aggregation is finalized and safe to emit, Append Mode relies on watermarking.
• Latency impact: Append mode delays output emission until the engine confirms that late-arriving data for a given window threshold can no longer alter the aggregate result.

When to pick which: need a continuously refreshed full picture → Complete; need to minimize bandwidth on a key-value state → Update; need append-only storage or clean finalized windows → Append.

12.3 Event Time Processing and Watermarking Protocol

Real-world streaming data frequently arrives out of order due to network latency, device connectivity drops, or distributed queue delays. Stream processing systems distinguish between two timelines:

1. Ingestion Time: The timestamp recorded when the processing engine (Spark) receives the event.
2. Event Time: The timestamp embedded directly inside the record payload when the event was originally generated at the source device.

Processing streams based on event time ensures accurate windowed aggregations regardless of arrival delays. To manage unbounded state and decide when late data must be discarded, Spark uses a mechanism called watermarking.

12.3.1 Mathematical Formulation of Watermarks

The verbal definition of a watermark is: the watermark value at any trigger execution is defined as the maximum event time observed by the engine across all processed data up to that point, minus a user-defined delay threshold.

In PySpark, watermarking is declared on a streaming DataFrame using the withWatermark transformation:

streaming_df.withWatermark("event_time", "10 minutes") \
            .groupBy(window("event_time", "10 minutes"), "word") \
            .count()

12.3.2 Watermark Execution Protocol

During streaming execution, the watermarking engine enforces two operational rules at each trigger point:

1. Data Ingestion Filter: Any incoming event whose embedded event time \(t_{\text{event}} < W\) is classified as "too late" and is immediately discarded at the ingestion boundary. It is excluded from state aggregation.
2. Window Finalization: A windowed aggregation covering the time interval \([t_{\text{window\_start}}, t_{\text{window\_end}}]\) is marked as finalized when the updated watermark boundary satisfies \(W \ge t_{\text{window\_end}}\). Once finalized in Append Mode, the aggregate result for that window is outputted to the sink, and its intermediate state is purged from memory.

Visual intuition: draw a horizontal timeline with event time on the \(x\)-axis and the count of arriving events on the \(y\)-axis. As time passes, a vertical watermark line sits \(\Delta_{\text{watermark}}\) behind the rightmost (newest) event that has arrived. Events that arrive to the left of the watermark line are dropped at the door. Windowed buckets whose right edge has slid to the left of the watermark line turn from "open" (grey) to "finalized" (green) and are emitted. The takeaway: the watermark is the moving boundary that separates "still accepting" from "already closed."

Real-world: Ad-click and sensor pipelines routinely use event-time watermarks so that a phone that was offline and reconnects late still contributes its events to the correct 10-minute window, while windows that are truly finished are flushed to data lakes without waiting forever. The same mechanism powers fraud-detection windows where a transaction's own timestamp — not the arrival time — must decide which alerting window it belongs to.

12.4 Step-by-Step Watermarking Walkthroughs

To understand the interaction between system processing time, event time, trigger intervals, batch windows, and watermarking thresholds, consider the following comprehensive worked walkthroughs.

12.4.1 Comprehensive Worked Computation — Append Mode with Watermarking

Execution Setup Parameters:

• Streaming Mode: Append Mode
• System Trigger Interval: 5 minutes (engine evaluates output every 5 minutes of system time)
• Batch Window Duration: 10 minutes (tumbling windows: [0, 10], [5, 15], [10, 20], etc.)
• Watermark Threshold (\(\Delta_{\text{watermark}}\)): 10 minutes

The timeline is tracked across system processing time \(t_{\text{sys}}\) (horizontal axis) and event timestamps \(t_{\text{event}}\) embedded in data tuples (event_time, word).

System Processing Timeline (t_sys):
t_sys = 5 min  ---> Trigger 1: Engine starts; Watermark = None; Output = None
t_sys = 10 min ---> Trigger 2: Max Event Time = 8; Watermark = 8 - 10 = -2 min; Output = None
t_sys = 15 min ---> Trigger 3: Max Event Time = 14; Watermark = 14 - 10 = 4 min; Output = None
t_sys = 20 min ---> Trigger 4: Max Event Time = 21; Watermark = 21 - 10 = 11 min; Output = None
                    [Late event (3, "bat") arrives -> 3 < 11 -> DISCARDED]
t_sys = 25 min ---> Trigger 5: Max Event Time = 26; Watermark = 26 - 10 = 16 min;
                    [Watermark 16 > Window End 10 -> Finalizes Window [0, 10]]
                    OUTPUT: (cat: 1, dog: 2, owl: 1)
                    [Late event (9, "cat") arrives -> 9 < 16 -> DISCARDED]
t_sys = 30 min ---> Trigger 6: Max Event Time = 26; Watermark = 16 min (unchanged);
                    [Watermark 16 > Window End 15 -> Finalizes Window [5, 15]]
                    OUTPUT: (cat: 1, dog: 3, owl: 2)

Step-by-Step Execution Sequence:

1. System Time \(t_{\text{sys}} = 5\) minutes (Trigger 1):

• Current Watermark \(W\): None (initialization phase).
• Batch to Finalize: None.
• Output: None.

1. Events arrive between \(t_{\text{sys}} = 5\) and \(t_{\text{sys}} = 10\):

• Data received: (7, "dog"), (8, "owl").

1. System Time \(t_{\text{sys}} = 10\) minutes (Trigger 2):

• Current Watermark \(W\): None (no prior watermark exists).
• Batch to Finalize: None (cannot finalize window [0, 10] without an established watermark).
• Max Event Time observed so far \(T_{\text{max}}\): 8 minutes.
• Update Watermark: \(W = T_{\text{max}} - \Delta_{\text{watermark}} = 8 - 10 = -2\) minutes. (Negative boundary indicates pipeline warmup).
• Output: None.

1. Events arrive between \(t_{\text{sys}} = 10\) and \(t_{\text{sys}} = 15\):

• Data received: (9, "cat"), (14, "dog").

1. System Time \(t_{\text{sys}} = 15\) minutes (Trigger 3):

• Current Watermark \(W\): \(-2\) minutes.
• Batch to Finalize: None (watermark \(-2 < 10\)).
• Max Event Time observed so far \(T_{\text{max}}\): 14 minutes.
• Update Watermark: \(W = 14 - 10 = 4\) minutes. (Watermark line advances to 4 minutes).
• Output: None.

1. Events arrive between \(t_{\text{sys}} = 15\) and \(t_{\text{sys}} = 20\):

• Data received: (8, "dog"), (13, "owl"), (15, "cat"), (21, "dog").

1. System Time \(t_{\text{sys}} = 20\) minutes (Trigger 4):

• Current Watermark \(W\): 4 minutes.
• Batch to Finalize: None (watermark \(4 < 10\); window [0, 10] remains open for potential late arrivals with \(t_{\text{event}} > 4\)).
• Max Event Time observed so far \(T_{\text{max}}\): 21 minutes.
• Update Watermark: \(W = 21 - 10 = 11\) minutes. (Watermark line advances to 11 minutes).
• Output: None.

1. Events arrive between \(t_{\text{sys}} = 20\) and \(t_{\text{sys}} = 25\):

• Data received: (26, "owl"), (17, "owl"), (3, "bat").
• Late Data Filter Audit: Event (3, "bat") has embedded event time \(t_{\text{event}} = 3\). The current watermark is \(W = 11\) minutes. Because \(3 < 11\), event (3, "bat") is discarded at ingestion and purged immediately.

1. System Time \(t_{\text{sys}} = 25\) minutes (Trigger 5):

• Current Watermark \(W\): 11 minutes.
• Batch Finalization Evaluation: The engine checks window [0, 10]. Because watermark \(W = 11 \ge 10\), window [0, 10] is officially finalized.
• Finalized Window [0, 10] Data Aggregation:
• Included events: (7, "dog"), (8, "owl"), (9, "cat"), (8, "dog").
• Word counts: cat: 1, dog: 2, owl: 1.
• Output written to Sink:

• Max Event Time observed so far \(T_{\text{max}}\): 26 minutes.
• Update Watermark: \(W = 26 - 10 = 16\) minutes. (Watermark advances to 16 minutes).

1. Events arrive between \(t_{\text{sys}} = 25\) and \(t_{\text{sys}} = 30\):

• Data received: Late arriving record (9, "cat").
• Late Data Filter Audit: Event (9, "cat") has \(t_{\text{event}} = 9\). Current watermark \(W = 16\) minutes. Because \(9 < 16\), this record is discarded at ingestion as a late arrival.

1. System Time \(t_{\text{sys}} = 30\) minutes (Trigger 6):

• Current Watermark \(W\): 16 minutes.
• Batch Finalization Evaluation: The engine checks window [5, 15]. Because watermark \(W = 16 \ge 15\), window [5, 15] is officially finalized.
• Finalized Window [5, 15] Data Aggregation:
• Included valid events with \(t_{\text{event}} \in [5, 15]\): (7, "dog"), (8, "owl"), (9, "cat"), (8, "dog"), (13, "owl"), (14, "dog").
• Word counts: cat: 1, dog: 3, owl: 2.
• Output written to Sink:

• Max Event Time \(T_{\text{max}}\): 26 minutes (unchanged).
• Update Watermark: \(W = 26 - 10 = 16\) minutes (remains 16 minutes).

12.4.2 Comparative Analysis — Update Mode with Watermarking

If the exact same scenario above is executed in Update Mode instead of Append Mode:

• At system time \(t_{\text{sys}} = 10\), Update Mode immediately emits intermediate aggregate counts for unfinalized window [0, 10] (dog: 1, owl: 1).
• At system time \(t_{\text{sys}} = 15\), Update Mode emits updated counts for window [0, 10] (cat: 1, dog: 2, owl: 1) and initial counts for window [5, 15].
• Once the watermark boundary advances past a window's end time (e.g., \(W = 11 \ge 10\) at \(t_{\text{sys}} = 25\)), window [0, 10] is finalized. The engine stops accepting changes for window [0, 10] and suppresses future emissions for that window.

When to pick which: if downstream consumers can tolerate (and want) incremental updates, Update Mode gives lower latency; if the sink must contain only final, immutable results — typical for data lakes — Append Mode is the only correct choice.

12.4.3 Student Q&A Exchanges on Watermarking Mechanics

Real-world: Event-time watermarks with Append Mode are what let clickstream and IoT pipelines land clean, deduplicated, finalized windows into Parquet/ORC data lakes — every emitted row is guaranteed not to change later, so lake readers never see corrected or duplicated records.

12.5 Machine Learning Fundamentals and Tool Ecosystem

The accumulated historical datasets stored via batch processing engines and real-time streams processed by Spark serve as the core input data for Machine Learning (ML) algorithms. Machine learning provides automated computational models capable of discovering underlying patterns, predicting future outcomes, and making intelligent decisions from data.

12.5.1 Formal Definitions of Machine Learning

12.5.1.1 Qualitative Definition — Arthur Samuel (1959)

Arthur Samuel, a pioneer in computer gaming and artificial intelligence, defined machine learning as:

"The field of study that gives computers the ability to learn without being explicitly programmed."

Samuel demonstrated this concept by building a self-learning Checkers program. By having two instances of the computer program play thousands of games against each other, the program accumulated experience, learned winning position strategies, and eventually outperformed Samuel himself.

12.5.1.2 Quantitative Definition — Tom Mitchell (1997)

Tom Mitchell formulated a precise, engineering-oriented definition based on three core components: Experience (\(E\)), Task (\(T\)), and Performance Measure (\(P\)):

"A computer program is said to learn from experience \(E\) with respect to some class of tasks \(T\) and performance measure \(P\), if its performance at tasks in \(T\), as measured by \(P\), improves with experience \(E\)."

where:

• \(E\) represents the experience (training dataset or historical environment interactions).
• \(T\) represents the target task (e.g., classifying email as spam vs. non-spam, or controlling a self-driving vehicle).
• \(P\) represents the quantitative performance metric (e.g., classification accuracy percentage, or mean squared error).

12.5.2 Industry Tool Ecosystem for Big Data Machine Learning

The machine learning tool ecosystem spans low-code UI platforms, desktop/single-node scripting libraries, and distributed big data processing engines:

In this course, primary emphasis is placed on Apache Spark MLlib to leverage distributed parallel execution over large-scale big data infrastructure.

Real-world: The same MLlib pipelines that run on Spark clusters power real-time fraud scoring and recommendation engines, consuming the very streaming data that Sections 12.1–12.4 showed being ingested from Kafka.

12.6 Machine Learning Taxonomy and End-to-End Pipeline

12.6.1 The End-to-End Machine Learning Pipeline

Building an enterprise machine learning solution requires executing a structured, six-stage pipeline:

[1. Data Collection] ---> [2. Data Cleaning] ---> [3. Feature Engineering]
                                                          |
[6. Deployment]     <--- [5. Model Evaluation] <--- [4. Model Training]

1. Data Collection & Ingestion: Gathering raw unstructured or structured records from databases, streaming brokers (Kafka), logs, or file storage.
2. Data Preprocessing & Cleaning: Handling missing values, filtering noise, deduplicating records, and resolving data type mismatches.
3. Feature Engineering & Selection: Transforming raw inputs into informative numerical vectors. Redundant or collinear features must be pruned. For example, if a dataset contains patient height in both centimeters and meters, keeping both features introduces numerical redundancy; one representation should be removed.
4. Model Selection & Training: Selecting appropriate candidate algorithms (e.g., Linear Regression, Decision Trees) and fitting model parameters on historical training data.
5. Model Evaluation & Validation: Measuring model generalization performance on unseen test data using statistical metrics.
6. Deployment & Monitoring: Publishing trained models to production microservices or streaming engines for real-time inference.

12.6.2 Taxonomy of Machine Learning Paradigms

Machine learning algorithms are categorized into three major learning paradigms based on the nature of feedback provided during training:

                          Machine Learning
                                 |
      +--------------------------+--------------------------+
      |                          |                          |
Supervised Learning     Unsupervised Learning    Reinforcement Learning
      |                          |                          |
  +---+---+                  +---+---+                  +---+---+
  |       |                  |       |                  |       |
Reg.    Class.           Clust.   Assoc.             Agent   Reward

12.6.2.1 Supervised Learning

In Supervised Learning, the model is trained on a labeled dataset consisting of input-output pairs \(\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{N}\), where \(\mathbf{x}_i \in \mathbb{R}^d\) represents the independent feature vector and \(y_i\) represents the target dependent variable.

Supervised learning is divided into two sub-branches based on the data type of the target variable \(y\):

1. Regression: The target variable \(y \in \mathbb{R}\) is continuous.

• Formulation: Modeling the functional mapping \(y = f(\mathbf{x}) + \epsilon\).
• Linear Regression Equation:

\[ y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_d x_d + \epsilon \] where \(y \in \mathbb{R}\) is the scalar dependent variable, \(\beta_0 \in \mathbb{R}\) is the intercept term, \(\beta_j \in \mathbb{R}\) are feature weights, \(x_j \in \mathbb{R}\) are independent feature inputs, and \(\epsilon \sim \mathcal{N}(0, \sigma^2)\) represents unobserved random error.

• Real-world applications: Predicting house prices based on land area and room count, stock index forecasting, temperature estimation.

1. Classification: The target variable \(y\) is categorical (discrete classes).

• Binary Classification: Target \(y \in \{0, 1\}\) (e.g., Email Spam vs. Non-Spam, Credit Card Fraud vs. Legitimate, Disease Positive vs. Negative).
• Multiclass Classification: Target \(y \in \{1, 2, \dots, K\}\) where \(K > 2\) (e.g., Handwritten digit recognition \(0\text{--}9\), news article topic categorizations).

12.6.2.2 Unsupervised Learning

In Unsupervised Learning, the model is provided with an unlabeled dataset \(\mathcal{D} = \{\mathbf{x}_i\}_{i=1}^{N}\) containing feature vectors without corresponding ground-truth target outputs \(y\). The objective is to uncover intrinsic patterns, structural groupings, or underlying probability densities.

Main sub-branches include:

1. Clustering: Partitioning data points into homogeneous groups such that points within a cluster are highly similar, while points across clusters are distinct.

• Primary Algorithm: K-Means Clustering.
• Real-world applications: Customer market segmentation, grouping documents by topic, anomaly detection.

1. Association Rule Mining (Market Basket Analysis): Discovering co-occurrence relationships and conditional purchase dependencies between items in transactional databases.

• Primary Algorithm: Apriori Algorithm.
• Core Mathematical Metrics for a rule \(A \rightarrow B\):
• Support: The joint probability of finding both itemsets \(A\) and \(B\) in a transaction:

\[ \text{Support}(A \rightarrow B) = P(A \cap B) = \frac{\text{Count}(A \cup B)}{N} \] where \(\text{Count}(A \cup B)\) is the number of transactions containing both \(A\) and \(B\), and \(N\) is total transactions.

• Confidence: The conditional probability that a transaction contains \(B\), given that it contains \(A\):

\[ \text{Confidence}(A \rightarrow B) = P(B \mid A) = \frac{P(A \cap B)}{P(A)} = \frac{\text{Count}(A \cup B)}{\text{Count}(A)} \]

• Lift: The ratio of observed joint occurrence to expected independent occurrence:

\[ \text{Lift}(A \rightarrow B) = \frac{P(A \cap B)}{P(A) \cdot P(B)} = \frac{\text{Confidence}(A \rightarrow B)}{P(B)} \] where a \(\text{Lift} > 1\) indicates a strong positive inferential association between itemset purchase behaviors.

• Real-world retail examples:
• The classic retail finding where diaper purchases strongly correlate with Friday evening beer purchases.
• Predictive analytics revealing an unexpected surge in customer demand for strawberry Pop-Tarts in retail stores immediately preceding major hurricane landfall events.

12.6.2.3 Reinforcement Learning

In Reinforcement Learning (RL), an autonomous agent learns optimal behavior strategies by interacting directly with a dynamic environment through trial and error over discrete time steps.

The agent observes the current environmental state \(s_t \in \mathcal{S}\), selects an action \(a_t \in \mathcal{A}\) based on a policy \(\pi(a \mid s)\), transitions to a new state \(s_{t+1}\), and receives a scalar reward signal \(r_t \in \mathbb{R}\) (or penalty). The goal is to maximize cumulative long-term discounted rewards \(\sum_{t=0}^{\infty} \gamma^t r_t\).

• Real-world applications: Autonomous vehicle navigation, chess/Go game engines (DeepMind AlphaGo), robotic arm manipulation, algorithmic trading strategies.

Real-world: Retailers deploy the Apriori algorithm to optimize shelf layouts and inventory — pairing beer near diaper aisles for weekend shoppers or stocking strawberry Pop-Tarts before forecasted hurricanes — while streaming platforms feed the same event data into supervised fraud classifiers, showing how Sections 12.1–12.6 connect end to end.

12.7 Model Evaluation Metrics

Evaluating model quality requires distinct quantitative performance metrics for classification and regression tasks.

12.7.1 Classification Evaluation Metrics — The Confusion Matrix

For binary classification problems, model predictions are evaluated against actual ground-truth labels using a \(2 \times 2\) Confusion Matrix:

From these four fundamental counts, four primary summary metrics are derived:

1. Accuracy: The proportion of total predictions that were correctly classified:

\[ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} \] where \(TP, TN, FP, FN \in \mathbb{N}\). Accuracy can be misleading on imbalanced datasets (e.g., 99% non-fraud, 1% fraud).

1. Precision: The proportion of positive model predictions that are truly positive:

\[ \text{Precision} = \frac{TP}{TP + FP} \] Precision measures prediction quality and controls False Positive alarms.

1. Recall (Sensitivity / True Positive Rate): The proportion of actual positive cases that were correctly identified by the model:

\[ \text{Recall} = \frac{TP}{TP + FN} \] Recall measures prediction coverage and controls False Negatives (critical in medical diagnostics and fraud detection).

1. F1-Score: The harmonic mean of Precision and Recall, balancing both metrics into a single scalar value:

\[ F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2 \cdot TP}{2 \cdot TP + FP + FN} \]

12.7.2 Regression Evaluation Metrics

For regression tasks predicting continuous target values \(\hat{y}_i\) against ground truth values \(y_i\) over \(N\) evaluation samples:

1. Mean Squared Error (MSE): Average squared difference between actual and predicted values:

\[ \text{MSE} = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2 \] MSE heavily penalizes large outlier errors due to the squaring operation.

1. Root Mean Squared Error (RMSE): The square root of MSE, restoring error metrics back to the original physical units of the target variable \(y\):

\[ \text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2} \]

1. Mean Absolute Error (MAE): Average absolute magnitude of errors:

\[ \text{MAE} = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i| \] MAE provides a linear error score robust to extreme outliers.

1. R-Squared (\(R^2\) Coefficient of Determination): The proportion of variance in the dependent target variable explained by the regression model:

\[ R^2 = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2} \] where \(\bar{y} = \frac{1}{N}\sum_{i=1}^{N} y_i\) is the empirical mean of actual target values. An \(R^2 = 1.0\) indicates perfect model fit, while \(R^2 = 0.0\) indicates the model performs no better than predicting the mean value \(\bar{y}\).

Real-world: Fraud-detection and medical-screening systems deliberately optimize Recall, accepting more false alarms so that almost no real fraud or disease is missed; pricing and demand-forecasting systems report RMSE in the same currency as their forecasts so business users can read "average error per prediction" directly.

Exam Guidance Summary

Exam note: Review the following key exam topics and question patterns derived from this lecture:

1. Spark Structured Streaming Output Modes:

• Be prepared to compare Complete Mode, Update Mode, and Append Mode across memory resource usage, output emission frequency, and sink compatibility (4--5 marks).
• Remember: Append Mode emits output only when a batch window is finalized by the watermark boundary.

1. Watermarking Calculation & Trace Problems:

• Practice multi-step numerical trace questions given trigger interval, batch window size, watermark delay threshold, and event tuples (event_time, key).
• Calculate updated watermark boundaries using \(W = T_{\text{max}} - \Delta_{\text{watermark}}\).
• Identify discarded late records (\(t_{\text{event}} < W\)) and state final window aggregate outputs.

1. Definitions of Machine Learning:

• Memorize Tom Mitchell's quantitative ML definition: Experience \(E\), Task \(T\), Performance Measure \(P\).
• Given a application problem (e.g., self-driving car or email spam filter), identify the specific \(E\), \(T\), and \(P\).

1. Supervised vs. Unsupervised Learning & Market Basket Analysis:

• Distinguish between Regression (continuous target) and Classification (categorical target).
• State formulas and calculate Support \(P(A \cap B)\), Confidence \(P(B \mid A)\), and Lift \(\frac{P(A \cap B)}{P(A)P(B)}\) for Association Rule Mining problems.

1. Evaluation Metrics & Confusion Matrix Computations:

• Construct a \(2 \times 2\) confusion matrix from problem statements.
• Compute Accuracy, Precision, Recall, and F1-Score. Explain why high recall is critical in fraud/disease detection.
• State regression metrics (MSE, RMSE, MAE, \(R^2\)) and explain their physical interpretations.

Key Industry Applications

Real-world: Apache Kafka serves as the standard streaming ingestion buffer, feeding live events into Spark Structured Streaming for fraud detection, metrics aggregation, and clickstream analytics.

Real-world: Complete Output Mode powers live web dashboards and metrics counters requiring a continuous full refresh of top-N key counters at every trigger boundary.

Real-world: Update Output Mode powers enterprise session state tables and user activity trackers (such as last login/logout events), minimizing network I/O by outputting only changed user records.

Real-world: Append Output Mode with event-time watermarking ensures append-only storage formats (such as Parquet or ORC data lakes) receive clean, deduplicated, finalized window outputs without data duplication or late-data corruption.

Real-world: Predictive feature engineering in big data pipelines prunes redundant features (such as height in centimeters vs. meters) to eliminate multicollinearity and improve model stability.

Real-world: Retail chains deploy Association Rule Mining (Apriori algorithm) to optimize store shelf layouts and inventory management—such as pairing beer near diaper aisles for weekend shoppers or stocking strawberry Pop-Tarts prior to forecasted hurricane events.

Real-world: Healthcare diagnostics employ machine learning classification models trained on audiometric decibel frequencies to deliver personalized hearing test calibrations.