—

Stream Processing and Analytics

Spark Streaming: Architecture, Setup, DataFrames, and RDDs

Published: 2026-08-07

Level: postgraduate

Audience: Postgraduate students in Stream Processing and Analytics

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Stream processing vs batch processing — covered in Lecture 1 (1.6) and Lecture 4 (4.1)
Real-time systems: hard, soft, and near — covered in Lecture 3 (3.1)
The rate mismatch and durable storage — covered in Lecture 4 (4.2) and Lecture 7 (7.4)
Spark high-level view: architecture, cluster managers, and libraries — covered in Lecture 9 (9.3)
RDD versus Structured Streaming programming models — covered in Lecture 9 (9.3.3)
Kafka streaming setup and the producer–consumer demo — covered in Lecture 9 (9.2)

10.1 Spark Streaming: Two Ways to Process Data

10.1.1 Batch Processing vs Real-Time Stream Processing

This session opens the Spark streaming module in earnest. The class had already worked through Kafka (installing it, writing producers, producing console messages), and Spark's installation steps were partially covered. A student asked about a console application demo with publish-subscribe messaging and whether that was something different from Spark. The answer: that demo was the Kafka setup. Later in the course the class will see how the messages Kafka sends get handled by Spark, so the Kafka-to-Spark integration is still to come.

Spark offers two ways to process data:

Real-time stream processing — records are handled as they arrive, with low latency.
Micro-batch processing, or plain batch processing — records are grouped into small, discrete batches, and each batch is processed as a unit.

The name "micro-batch" is the clue: it is batch processing applied to tiny, frequent batches instead of huge nightly jobs. Micro-batch processing is the middle ground between true one-at-a-time streaming and large batch processing. Small batches of tuples are processed together, the batches are processed in strict order, and if anything in a batch fails, the entire batch is replayed. This ordering and replay behaviour is exactly what lets micro-batch engines give stronger accuracy guarantees than naive one-at-a-time processing. Spark's classic streaming engine works precisely this way: it cuts the incoming stream into micro-batches and runs each micro-batch through the same batch engine.

Q: The console application demo where we created producers and consumed messages, that was a publish-subscribe thing. Was that different, or was it in Spark context only? A: That was the Kafka setup, the demo part where producers produce to the console. Later on we will also see how the messages sent by Kafka are handled by Spark, so the Spark integration with Kafka will be covered separately. The division of labour behind the answer: Kafka is the message queue, Spark is the computation. Kafka's job is to hold and hand over messages; Spark's job is to run the actual computation on those messages. Whenever the class builds streaming applications, the real-time messages flowing in get computed by Spark itself, not by Kafka.

10.1.2 Where Streaming Is Useful

Spark streaming becomes genuinely useful when you want to process large datasets, especially in a nearly real-time environment. Think of event processing: something like a bank application where every transaction needs handling as it happens, or stock market data where prices and trades stream in continuously. Those are the settings where Spark helps a lot.

Why these are perfect streaming cases: a bank transaction cannot wait in a queue until midnight for the nightly batch run — fraud checks and account balances must react within seconds of the transaction happening. The same is true for stock prices: the value of a trade decision depends on acting while the price is still current. When correctness still matters but the data has a short shelf life, you need near-real-time processing.

Spark also has support for ingestion from various systems, meaning the messages being ingested can come through Kafka or through simple TCP/IP socket programming, among other sources. TCP/IP sockets are the lowest-level transport: a raw network connection that streams bytes as they arrive. Kafka sits far above that, offering a durable, scalable message log. Both feed the same Spark engine downstream.

10.1.3 Why Streaming Needs Long-Term Storage

A key design point the class was told to remember: in stream processing in general, you need long-term storage. You cannot just process all the messages as they come in, because the incoming rate of the messages is much, much higher than the processing speed at which processing can happen. The incoming stream outruns the processor, so messages must sit somewhere durable while the engine works through them.

The rate mismatch: let the ingestion rate be $r_{in}$ (messages per second) and the processing rate $r_{proc}$ (messages per second). Streaming only works in a sustained way when

$r_{proc} < r_{in} is acceptable ⟺ unprocessed messages must accumulate somewhere.$

The mismatch does not go away by ignoring it. If $r_{in} > r_{proc}$ and there is no durable queue, messages are simply dropped — data loss. If the messages sit in a durable store (Kafka's log is the classic example), the processor catches up at its own pace and no message is lost. This asymmetry between ingestion rate and processing rate is the reason storage is a first-class part of any streaming architecture.

Fundamental point: whenever you design streaming solutions, the incoming rate of the messages is much higher than the processing speed, so long-term storage is not optional. The stream must be buffered durably between ingestion and processing, or messages will be lost the moment the processor lags.

Where this is felt in practice: a stock exchange can publish tens of thousands of ticks per second; a single Spark node might process only thousands of records per second. Kafka absorbs the difference: Spark reads at its own speed from the log, and the log itself is the durable buffer that makes the whole system lossless. Durable storage also gives fault tolerance — if a consumer crashes, it can restart and resume from the last position it committed, replaying anything it missed.

10.2 Key Properties of Spark

10.2.1 In-Memory Computation

One of the most important things to keep in mind about Spark is that it performs in-memory computation. Whatever you do, that computation happens in memory. That is what makes Spark fast for iterative and interactive workloads, and it is the property that most shapes how Spark applications feel different from disk-based batch engines.

Why memory changes everything: a disk-based engine (the classic Hadoop MapReduce style) writes intermediate results to disk after each step and reads them back for the next step. Spark keeps intermediate data in RAM. For iterative jobs — ones that loop over the same data many times, like machine learning training — the difference is huge: one loop iteration that costs minutes with disk I/O can cost seconds in memory. The trade-off is that memory is a scarce, expensive resource, which is exactly why the cluster manager (Section 10.3.5) is responsible for memory management.

10.2.2 One Engine from Batch to Streaming

Spark is a kind of generalization. You can use it as a batch application, and when you reduce the batch size, then it becomes streaming. The same engine serves both regimes, and the boundary between batch and streaming is a parameter (the batch size) rather than a different product. If the batch window is small enough, you are effectively doing stream processing with the same code and the same engine. Shrinking the batch size turns batch processing into streaming.

A useful way to picture it: think of the batch size as a slider. Pull it all the way up — one huge batch, say all of yesterday's clicks — and you are doing classic batch processing. Pull it down to a few seconds' worth of data per batch, and you are doing micro-batch streaming. Nothing fundamental changes about the engine: the same transformations run on each batch, only the batch size moves. This is why the professor's phrase "reduce the batch size and it becomes streaming" is exactly how Spark's micro-batch model works: the incoming stream is cut into small batches, and each batch is processed as an ordinary Spark batch job.

10.2.3 Iterative Aggregations and Analytics on Real-Time Data

When you want to apply Spark iteratively, you can also use it for simple aggregations and simple analytics that you want to generate on real-time data. The engine is not only for huge one-shot jobs; it can run repeatedly over live data, computing rolling aggregates and lightweight analytics as the data arrives.

Because data stays in memory between iterations, the engine can re-apply the same computation on every new batch — count of events per second, running average of a sensor reading, sum of transactions per minute — without paying disk costs each cycle. These rolling aggregates are the bread and butter of monitoring dashboards: the batch window decides how fresh the numbers are.

10.2.4 Service Delivery Automation (Data Pipelines)

Spark supports complete service delivery automation. That means pipeline development: automated pipeline development right from the beginning of data ingestion into the system, to data processing, and finally to generating insights. These are the broad-level steps of any data pipeline, and Spark can carry all of them in an automated way.

Pipeline automation in three stages:

Ingestion — data enters the system (from Kafka, sockets, files, and so on).
Processing — Spark runs the transformations, aggregations, and analytics on the data.
Insights — the results are written out to storage or served to dashboards and downstream systems.

The same Spark application describes all three stages in one program, and the engine schedules and executes them automatically. When you want to perform all these steps in a more automated fashion, you can use it.

10.2.5 Language Support: Java, Python, Scala, SQL — and Why Not R

Spark provides support for different programming languages: Java, Python, Scala, and SQL. One interesting observation was raised: you see support for Java libraries and Scala libraries, but you do not see R here suddenly. Earlier in the program, machine learning tasks and data visualization tasks, which are performed for data and analytics, are also popular, and R has a strong footprint there. But for real-time streaming you do not see that kind of footprint from the R ecosystem. Scala definitely is present, Python definitely is present, and SQL and Java are the other tools. So the streaming stack in Spark is Java, Python, Scala, and SQL, with R absent from the streaming ecosystem even though R is popular for ML and visualization work.

Why R is missing from the streaming stack: R's strengths are interactive statistics, machine learning, and visualization — workloads that are analyst-driven rather than production-engine-driven. Real-time streaming libraries and connectors (the Kafka integrations, the streaming APIs) are written and maintained mainly in Java and Scala, with Python bindings as the other first-class citizen. So while R remains a strong tool for the analytics part of the pipeline, the streaming infrastructure itself speaks Java, Scala, Python, and SQL. If you want to do streaming in Spark, pick one of those four.

10.2.6 Integration with Other Platforms

The integration with other platforms is also very natural. The named integrations are Hadoop, Kafka, and AWS. Spark sits comfortably alongside the Hadoop ecosystem, consumes from Kafka, and runs in AWS environments. These integrations are why Spark appears in so many production data stacks.

The Spark recipe in one line: in-memory computation for speed, one engine that spans batch and streaming by tuning batch size, iterative analytics on live data, automated end-to-end pipelines, four languages (Java, Python, Scala, SQL), and natural integration with Hadoop, Kafka, and AWS. These six properties together explain why Spark became the default engine of the modern data stack.

10.3 Spark Architecture: Core, Libraries, and Cluster Managers

10.3.1 Spark Core and the High-Level Libraries

The Spark architecture picture is centered on Spark core. Around the core sit a few high-level libraries. One is the machine learning library. Another is Spark SQL, which includes structured streaming. There is also a graphics library, GraphX, for graph construction and processing, and real-time streaming support. These are the high-level libraries you import while developing an application, but application development itself happens at the core level. Applications can be written in Python, Java, or Scala. Scala is also very popular, because Spark itself is written in Scala. So when you work in Scala you are working in the same language as the engine.

The layered picture (from the top down):

Your application — written in Python, Java, or Scala, importing whichever high-level library you need.
High-level libraries — MLlib (machine learning), Spark SQL (which includes structured streaming), GraphX (graph processing), and real-time streaming support. These are the toolkits you import; they all run on top of core.
Spark core — the foundation: scheduling, memory management, fault recovery, and interaction with storage systems. Every library, every API call, is ultimately executed by core.

Application development happens at the core level: even when you use a library, the library compiles down to core operations that Spark executes across the cluster.

Why Scala gets special treatment: Spark is written in Scala. Writing your application in Scala means you use the same language as the engine itself — every Spark API surface is naturally available to you, and when something goes wrong you can read the engine's own code. That is why the course, and the Spark community, treat Scala as a first-class citizen alongside Python and Java.

10.3.2 The ML Library (MLlib)

The ML library requires special mention because the course assignment is also based on this. It is a built-in library. It contains machine learning algorithm functionality, and the typical algorithms supported here are classification algorithms, regression algorithms, and unsupervised learning algorithms such as collaborative filtering and clustering. So a single built-in library covers supervised learning (classification, regression) and unsupervised learning (collaborative filtering, clustering) without bolting on a separate framework.

What MLlib covers, mapped to ML families:

Supervised learning — classification (assigning labels, e.g., spam or not spam) and regression (predicting continuous values, e.g., a house price).
Unsupervised learning — clustering (grouping similar records, e.g., customer segments) and collaborative filtering (recommending items from user-item ratings, e.g., "people like you also bought…").

Because MLlib is built into Spark, the same in-memory, distributed engine that processes streaming data also trains and applies these models — no separate framework installation needed.

Exam note: the course assignment is based on the ML library (MLlib), so the algorithms named here — classification, regression, collaborative filtering, and clustering — are directly relevant to the assignment work. Keep this list close: it is the syllabus of the assignment.

10.3.3 The Graph Library (GraphX)

The graphics library, GraphX, is for the construction of graphs and for network analysis, including social network analysis. Graphs are normally used for devising parallel computation. GraphX uses the Spark RDD API, the resilient distributed dataset API, and constructs a directed acyclic graph, a DAG, under the hood. There are various algorithms built on it; the famous page ranking algorithm (PageRank) is the named example. PageRank looks at which website is most popular based on the real-time web activities of the users. So the library connects graph structure, parallel computation, and real-time behavior of users into one tool.

Three ideas in one sentence: a graph is a set of nodes (vertices) connected by edges — think of a social network where users are nodes and friendships are edges. Network analysis computes properties of that structure: who is central, which communities exist, what flows where. Parallel graph computation splits the graph across machines so each machine processes its slice of nodes, exchanging messages about the edges in between. GraphX gives you all three on top of Spark's distributed RDD machinery.

GraphX's two key traits:

Built on the RDD API — the graph is stored and processed as a distributed collection, so it inherits Spark's fault tolerance and scalability.
DAG under the hood — GraphX represents its computation as a directed acyclic graph (a chain of steps where no step loops back on itself), which Spark's scheduler can parallelize and recover from failure.

PageRank in plain words: the algorithm asks, "which web pages do users consider important?" The intuition: a page is important if important pages link to it. Each link from page A to page B casts a "vote" for B, weighted by A's own importance, and the process repeats until the scores settle. In the lecture's framing, PageRank looks at which website is most popular based on the real-time web activities of the users — the click and link activity flowing through the network feeds the ranking computation.

Real-world: PageRank over real-time web activity is the named graph use case, and social network analysis is the named application domain for GraphX.

10.3.4 Cluster Managers

At the bottom of the architecture sit the cluster managers. Spark can run on a Yarn cluster manager, which is the default cluster manager, or you can use Mesos, or it can be a simple standalone scheduler. These are the three different varieties when it comes to the cluster manager, and the most common cluster managers overall are Yarn, Mesos, or Kubernetes. The cluster manager is the layer that gives Spark its resources across machines.

The cluster manager's job in one line: it is the layer that hands Spark the machines (or containers) it needs and tracks what resources are available across the cluster. The named options:

YARN — the default and the one tied to the Hadoop ecosystem; Spark lives inside Hadoop's resource layer.
Mesos — a general cluster manager that can run Spark alongside other frameworks on the same machines.
Standalone — Spark's own simple scheduler, used when you just want Spark and nothing else.
Kubernetes — the modern container orchestrator; Spark pods are scheduled like any other container. (Named in the lecture as a common option; the setup walkthrough uses the default YARN path.)

10.3.5 Key Cluster Activities

The cluster manager performs some key activities. One is scheduling of the tasks across the workers. Another is memory management. Then there is fault tolerance, which is the ability of the system to come out of a failure very quickly. Finally, the cluster layer manages how you can interact with other databases, where you normally store the data. So task scheduling, memory management, fault tolerance, and database interaction are the four responsibilities that hang off the cluster manager layer.

The four responsibilities, and why each matters:

Task scheduling — deciding which worker runs which task, and in what order. Bad scheduling means idle machines and slow jobs.
Memory management — tracking and allocating RAM across workers. Remember Section 10.2.1: Spark's speed comes from memory, so memory is the resource that needs the most careful bookkeeping.
Fault tolerance — the ability to recover quickly from a failure. If a worker dies mid-computation, the cluster manager detects it and re-runs that worker's tasks elsewhere.
Database interaction — managing how the cluster reads from and writes to the storage systems where data lives (HDFS, S3, and so on).

Notice how these echo the streaming fundamentals from Section 10.1: storage is a first-class citizen, and the system is designed so a failure is a speed bump, not a restart.

Recap + bridge: the architecture is a stack — applications and high-level libraries (MLlib, Spark SQL, GraphX, streaming) on top of Spark core, and the cluster manager (YARN, Mesos, standalone, Kubernetes) at the bottom supplying scheduling, memory, fault tolerance, and database access. Next we look at the two ways you actually write applications on top of this stack: RDD and Structured Streaming.

10.4 Two Ways to Write Spark Applications: RDD and Structured Streaming

10.4.1 The Resilient Distributed Dataset (RDD)

There are two ways to write Spark applications. One is using RDD, which stands for resilient distributed dataset. The other is Spark Structured Streaming. Spark RDD and Spark Structured Streaming are the two approaches. The RDD is a very low-level API in Spark, and it provides simple operations like map and reduce. RDDs can be written in Python, Java, or Scala.

Reading the name, one word at a time:

Resilient — it can survive failures: if a worker dies, Spark rebuilds the lost partition from the lineage of operations that produced it, so the dataset is reconstructed rather than lost.
Distributed — the data is split across partitions that live on different nodes of the cluster.
Dataset — an in-memory collection of records.

An RDD is the lowest-level abstraction Spark offers. It exposes simple operations such as map (apply a function to every record) and reduce (combine all records into fewer records with an operation like sum). Everything higher in Spark — DataFrames, SQL, the libraries — is ultimately executed on top of RDDs, but as a programmer you rarely need to touch them directly.

Why the low-level API matters even today: the high-level libraries (MLlib, GraphX, streaming) are built on the RDD machinery. Understanding RDDs means understanding what those libraries do under the hood — lineage, lazy evaluation, partitions — which is exactly the vocabulary the next section (transformations and actions) uses.

10.4.2 Structured Streaming and the SQL Story

SQL conflicts with RDD, in the sense that SQL comes for structured streaming, which is altogether a different way of developing applications compared to RDD-based applications. So the presence of Spark SQL means you are in the structured streaming world, not the RDD world. The RDD is a low-level API and it lacks a couple of features that structured streaming provides; those details are explained later in the course. The takeaway for now is that the two API families differ in level of abstraction, in features, and in how you develop the application.

The two API families side by side:

Dimension	RDD	Structured Streaming (via Spark SQL)
Level of abstraction	Low-level, record-oriented API	High-level, table-oriented API
Mental model	Distributed collection of records	Endless table of rows, queried with SQL
Development style	Imperative functions like `map`, `reduce`	Declarative queries: `select`, `group by`, `filter`
Where the name comes from	Resilient distributed dataset	Structured (typed) data flowing through the engine

The presence of Spark SQL means you are in the structured streaming world, not the RDD world. The RDD is a low-level API and it lacks a couple of features that structured streaming provides; those details are explained later in the course. The takeaway for now is that the two API families differ in level of abstraction, in features, and in how you develop the application.

The trap to avoid: SQL is not just "another way to write RDD code." When you use Spark SQL or Structured Streaming, you are operating in a different API family with a different mental model — tables and declarative queries rather than collections and imperative functions. Mixing the two mental models is the source of most early Spark confusion.

10.4.3 Student Questions and Answers

Q: What is this resilient distributed dataset? A: A resilient distributed dataset is a very low-level API in Spark. It provides simple operations like map, reduce, and things like that. It can be written in Python, Java, or Scala, and it lacks a couple of features that structured streaming provides.

Recap + bridge: two application styles exist — the low-level RDD API (simple operations like map and reduce) and the SQL-driven Structured Streaming world. The course will spend most of its weight on structured streaming; RDD basics are the foundation. Next we go from API choice to the practical reality: getting Spark installed and configured on your own machine.

10.5 Setting Up Spark: Installation and Configuration

10.5.1 Downloading Spark and Pairing It with Hadoop

The setup path starts with the Kafka setup from the previous session, then moves to the Spark environment. You download the Spark tar file from the Spark website. A key fact about the download: when you want to download Spark, it comes with a combination of Hadoop as well, because you need the HDFS Hadoop file system. For every tarball you see for Spark, the corresponding Hadoop environment version is also displayed over there, so you have to see the compatible mix of your Spark with the Hadoop file system. The instructor is using Spark 3.0.x, and the Hadoop ecosystem paired with it is Hadoop 2.7.7.

Why Spark ships "with Hadoop": the pre-built Spark tarball is compiled against a specific Hadoop version, because Spark needs the Hadoop client libraries to talk to the HDFS distributed file system (where your data lives in a Hadoop cluster). The version appears right in the file name — a Spark 3.0.x build for Hadoop 2.7 comes as something like spark-3.0.3-bin-hadoop2.7.tgz. That "hadoop2.7" suffix tells you exactly which Hadoop client libraries are inside.

The compatibility rule: for every Spark version there is a pre-built Hadoop pairing listed next to it on the download page. The recommendation is you do not change whatever appears there — download the matching pre-built combination. The pair used in the course, Spark 3.0.x with Hadoop 2.7.7, is exactly such a standard pairing: the Hadoop 2.7 release line (final version 2.7.7) is the classic Hadoop environment that Spark 3.0.x builds target. (The version number was heard as "Hadoop 2.77"; the actual release line is 2.7.7, the standard Hadoop line paired with Spark 3.0.x builds.)

You will see a lot of versions when you open the download page. The current available version at the time of the session is 3.5.5, but there is not much difference, and the course uses a 3.0.x release. You can download whichever version you like as long as the Spark-Hadoop pairing is compatible. For the Kafka integration you can use either Apache Kafka or Confluent Kafka; both are fine.

For the Hadoop side, there are various variants of Hadoop binaries, and you can download any one suitable to you with the right combination of Spark version. The recommendation is you do not change whatever appears there, though you can certainly go for older versions from the archived releases.

10.5.2 Folder Structure and Configuration Files

Once you download the tar.gz file, you extract the binaries and create a simple folder name in your local file directory. The instructor's own machine has a folder for Confluent (the Kafka distribution, community edition, unzipped and renamed to simply "Confluent 5.4") and a folder for Spark. When you download the tar file you get a very long Spark version name with a tar.gz extension, but you simply extract the binary files and rename the folder to something simple like "Spark 3". You rename the folder because you do not want to keep changing it every time the version string changes.

Inside the Spark folder there are many directories: a bin directory, a config directory, a data directory, and so on. The bin directory holds the shell scripts, and inside it there is a Kubernetes folder as well, matching whichever cluster manager you want to use. The configuration folder holds different properties files. The most common one that will be used is spark-defaults.conf.

The two folders you actually touch:

bin/ — the shell scripts: spark-shell (the Scala REPL), pyspark (the Python REPL), spark-submit (to launch applications). Inside bin there is also a Kubernetes-related folder, matching whichever cluster manager you plan to use.
conf/ — the configuration files. The one you will use most is spark-defaults.conf, where Spark settings live as key-value pairs. Next to it sits log4j.properties, which controls Spark's logging noise.

If you open the spark-defaults.conf file you will see key-value pairs, and the hash symbol (#) in front of many lines marks them as comments. You keep all the default settings except the configuration for the key spark.driver.extraJavaOptions, plus the log4j default configuration. The values you store here are JVM parameters, Java Virtual Machine parameters. There is also a directory setting for the Yarn container whose file is referenced in the same file.

Reconciling the Yarn directory setting: the spoken key name was garbled ("directory for yarn container is a file name called Hello Spark"), but the standard Spark configuration keys that point YARN executors at a shared directory of JARs are spark.yarn.jars (a list of JAR paths) and spark.yarn.archive (a path to an archive of JARs). If your setup walkthrough asks for a "Yarn container directory," it is one of these two keys — both tell YARN where the Spark JARs live so every container on every worker can find them.

Later on you will add another key called spark.jars.packages. In that packages value you specify Kafka, because you are integrating Kafka with Spark. The exact file name and package coordinates for the Kafka-Spark connector are part of a separate setup walkthrough, which is covered later when the class reaches Kafka integration. This is the only file you touch in the configuration folder, apart from log4j.properties, which you also modify if required.

10.5.3 Environment Variables: Java, Hadoop, Kafka, and PySpark

Spark requires Java pre-installed, and the download link provides it. After the download, you add Spark to the path. The environment shows Kafka added to the path, Hadoop added to the path, and then Spark added to the path. A folder called "Kafka logs" is created locally as the default directory for the Kafka-Spark integration, and you do not have to do anything for it; it happens automatically.

For Python you need Python workers. You can either download Python directly or use the Anaconda platform. The instructor uses Anaconda 3, so the setup links the Python.exe file available in Anaconda 3 to a new environment variable called PYSPARK_PYTHON. This is how you tell Spark to use the Python flavor: because the Spark engine itself is installed, but you want Python to drive it, so you create the environment variable PYSPARK_PYTHON and point it at the Anaconda Python.exe.

The environment variables, in order:

JAVA_HOME (or just Java on the path) — Spark's engine runs on the JVM, so Java must be present first.
Kafka on the path — from the previous session's Kafka setup.
Hadoop on the path — the HDFS client binaries.
Spark on the path — so pyspark, spark-shell, and spark-submit are reachable from anywhere.
PYSPARK_PYTHON — tells Spark which Python executable to use for Python workers. Set to the Anaconda Python.exe path, e.g. C:\Users\<you>\anaconda3\python.exe.

The "Kafka logs" folder appears automatically as the default directory for the Kafka-Spark integration; nothing to configure.

The actual linking happens when you create a variable called PYTHONPATH along with the Spark python/lib folder and Py4j. Py4j is described as a unit-testing library for Python; it is needed for debugging and other purposes. So you set both paths: one for Python, and within Python, the library folder that contains Py4j.

The Py4j piece: Py4j (Python for Java) is the bridge that lets Python code call into the Java-based Spark engine — every spark.* call you make from a Python notebook travels through Py4j to the JVM underneath. The lecture calls it a unit-testing library for debugging purposes; functionally, it is the glue between your Python session and the Spark JVM. For Python to work at all, PYTHONPATH must include Spark's python/lib folder where Py4j lives. Without it you get import errors at startup even though Spark itself is fine.

10.5.4 Taming Log Noise with log4j

In the log4j.properties file you change the root category to WARNING instead of INFO. The class was also told about a log4j setting that controls the log level for the class that spark-shell uses when it starts, so that the shell does not flood the screen.

The two log4j edits (reconciled to the standard keys):

Root category: change log4j.rootCategory=INFO, console to log4j.rootCategory=WARN, console. This lowers the overall log level from INFO to WARNING, so only warnings and errors print.
Class-level setting for the Spark internals: add log4j.logger.org.apache.spark=WARN. This keeps Spark's own internal messages quiet while still letting warnings from the shell appear.

With both in place, starting pyspark or spark-shell produces a clean prompt instead of hundreds of INFO lines scrolling past. (The spoken class key was garbled; org.apache.spark is the standard logger package for Spark's own classes.)

10.5.5 Verifying the Installation

To check whether the installation works, you can simply type "pyspark" on the command line. When you type pyspark it creates the Spark environment, and you can write a simple Python command right there to test it. The instructor also has Anaconda integrated, so you can work with Jupyter notebooks as well, running Spark cells with Shift+Enter.

Smoke test (worked through): open a terminal and type:

pyspark

The shell starts and prints the Spark version. Then run:

spark.version

You should see the installed version, e.g. '3.0.3'. A trivial sanity check that exercises the engine end to end:

spark.range(1, 4).count()

This asks the engine to count the rows 1, 2, 3 and should return 3. If the prompt returns 3 without a stack trace, Java, Spark, Python, and Py4j are all wired correctly. Sense-check: the number returned must equal the length of the range you asked for.

10.5.6 Student Questions and Answers

Q: For Apache Spark, can we go with Apache Hadoop 3.3? Will that work together? A: Yes, you can. What you need to download is the Hadoop version that matches your Spark. There is a link given over there as well. The .tgz file is there, and that we need to use in the Linux environment only, inside the virtual machine. So pick the Hadoop distribution that pairs with the Spark version you downloaded.

Recap + bridge: the setup recipe — download the Spark tarball pre-built for your Hadoop version (the course uses Spark 3.0.x with Hadoop 2.7.7), extract and rename the folder, set JAVA_HOME/Kafka/Hadoop/Spark on the path plus PYSPARK_PYTHON and PYTHONPATH with Py4j, calm log4j down to WARN, and verify with pyspark. Everything here is mechanical; the interesting part starts in code with the SparkSession entry point.

10.6 SparkSession: The Entry Point

10.6.1 The Singleton Pattern

To start working in code, you create a Spark session. The import is from pyspark.sql import SparkSession. SparkSession is a singleton object, meaning only one instance is created. To ensure that only one instance is created, you enclose all this code in an if __name__ == "__main__" block. Only then do you create the Spark session. The singleton pattern is important in distributed programs: you do not want multiple sessions fighting over the same cluster resources.

Why a singleton? A singleton is a class that allows exactly one instance in the whole program. A Spark session holds a connection to the cluster — the driver, the executor pool, the resource reservations. Two sessions in one program would mean two drivers competing for the same workers, double memory reservations, and confusing scheduling. The if __name__ == "__main__" guard does two jobs at once: it is the standard Python idiom that prevents the block from running when the file is imported instead of executed, and it guarantees the session creation runs exactly once, in one place.

10.6.2 Master URL and Configurations

You create the session with SparkSession.builder. You give the application a name with .appName, and any name works. Then you set the master. The instructor uses master("local[3]"), meaning three processes, creating a mini-cluster on the local machine, which is a laptop or desktop. That is what "local" means: you are using your own machine instead of a real cluster.

The session also carries configurations. Each configuration is a key-value pair: in the key you specify what needs to be done, and in the value you set the corresponding status. One configuration says the session should stop gracefully, without many exceptions, when a shutdown happens. There is also a configuration for the shuffle partitions, setting three partitions for the session.

Building the session (worked through):

from pyspark.sql import SparkSession

if __name__ == "__main__":
    spark = (
        SparkSession.builder
        .appName("Lecture10Demo")
        .master("local[3]")
        .config("spark.sql.shuffle.partitions", 3)
        .getOrCreate()
    )

Walking through the pieces:

SparkSession.builder starts the builder chain.
.appName("Lecture10Demo") names the application — any name works, it shows up in the cluster's application list.
.master("local[3]") runs in local mode with three worker threads, a mini-cluster on your own machine. The professor's phrase was "master of local three"; local[3] is the standard Spark way to request three local worker threads, matching that description of creating three processes. (For comparison, local[*] uses all available cores, and a real cluster would use something like yarn or spark://host:7077.)
.config("spark.sql.shuffle.partitions", 3) sets a session-wide key-value pair: after a shuffle, data is repartitioned into 3 partitions. The professor's phrase "just three partitions" maps to this shuffle-partition setting; spark.sql.shuffle.partitions is the standard key that controls it.
.getOrCreate() returns the existing session if one exists, or creates it — the practical implementation of the singleton guarantee.

The two configurations mentioned in class: a graceful-shutdown setting (so the session stops cleanly with few exceptions on shutdown, e.g. the spark.driver.extraJavaOptions family from the setup section) and the shuffle-partition count. You add more later as key-value pairs exactly like these.

Common traps:

Forgetting getOrCreate() at the end of the builder chain — the session is never created.
Creating the session outside the if __name__ == "__main__" guard, then importing the file elsewhere and accidentally spawning a second session on import.
Expecting local[3] to give three machines. It gives three worker threads on one machine — a mini-cluster for learning, not a real cluster. The same code switches to a real cluster by changing the master URL.

Recap + bridge: every Spark program starts with one line of ceremony — a singleton SparkSession built with .appName (any name), .master("local[3]") (three local threads, i.e., your machine), and key-value .config entries. The session is your handle to everything that follows, starting with DataFrames and schemas.

10.7 DataFrames and Schemas

10.7.1 Creating a DataFrame from a Collection

The Spark DataFrame API is an in-memory storage of your actual data result set. When you store data in it, you need to explicitly specify the data type of each column. That is what a schema is for. There are two types of schema: implicit schema and explicit schema.

DataFrame and schema in one breath: a DataFrame is an in-memory, table-shaped collection of records held inside Spark: rows on top, named columns from left to right. A schema is the description of that shape — for every column, its name and its data type. The schema is not optional decoration: Spark needs it to know how to store, sort, and validate each value. The question is only where you declare it, which is exactly the implicit/explicit distinction that follows.

The first demo creates a DataFrame from a simple list of integers. You take a list of values, pass it to spark.createDataFrame, and show the result:

age = [1, 2, 3, 4, 5]
df = spark.createDataFrame(age, "int")
df.show()

If you print the schema, it shows the data type of the column; if you call the DataFrame's show method, it displays the rows, and printSchema gives you the data types. So the tiny list becomes a single-column DataFrame whose column type was decided at creation time.

Worked example 1 — a list of integers becomes a one-column frame:

age = [1, 2, 3, 4, 5]
df = spark.createDataFrame(age, "int")
df.show()
df.printSchema()

Step by step: the list [1, 2, 3, 4, 5] is handed to createDataFrame together with the type string "int". Spark stores the five values in one column named value with integer type. show() renders the rows:

+-----+
|value|
+-----+
|    1|
|    2|
|    3|
|    4|
|    5|
+-----+

printSchema() renders the type information:

root
 |-- value: integer (nullable = true)

Sense-check: five input values, five output rows — nothing lost, nothing invented, and the column type is the int we declared at creation time.

10.7.2 Implicit Schema

Implicit schema means that when you create the DataFrame, at that time itself you specify what the data type of each column is. The word "implicit" here is easy to misread; the instructor's own definition is that it is implicit because at the time of creating the DataFrame object you are passing the values as hard-coded data types.

The word "implicit" is the trap. Most students read "implicit" as "automatic — you do not define it." That is wrong here. The instructor's definition: implicit means you are passing the data types as hard-coded arguments right inside the createDataFrame call, at the moment of creation. The type information is still stated — it is just stated inline, in the same line of code, rather than in a separate schema object. When the class later works through a big explicit-schema example, you will remember: implicit = types baked into the constructor call.

The next demo uses a list of tuples, which is the natural shape for event records. Each record has three fields: a timestamp (when the event happened), the amount of the purchase, and the location or specific vendor name where it was made. These are the events about various transactions that users perform. Considering these three as individual columns, you store the corresponding rows as tuples, so you have a list of tuples and you pass this list of tuples to the createDataFrame method while supplying the data types. The schema declaration looks like this: the column name "timestamp" has data type string, the column name "amount" has data type float, and the column name "vendor" has data type string.

events = [
    ("2024-05-01 10:15:00", 250.5, "Vendor A"),
    ("2024-05-01 10:18:00", 89.0, "Vendor B"),
]
events_df = spark.createDataFrame(events, ["timestamp", "amount", "vendor"])
events_df.show()
events_df.printSchema()

After running this cell, the show method gives the rows and printSchema gives the schema with the data types, exactly as declared.

Worked example 2 — a list of tuples becomes an events frame (implicit schema):

events = [
    ("2024-05-01 10:15:00", 250.5, "Vendor A"),
    ("2024-05-01 10:18:00", 89.0, "Vendor B"),
]
events_df = spark.createDataFrame(events, ["timestamp", "amount", "vendor"])

The list of tuples is the natural shape for event records: each tuple is one event, and its three elements line up with the three column names in the second argument. show() prints:

+-------------------+------+--------+
|timestamp          |amount|vendor  |
+-------------------+------+--------+
|2024-05-01 10:15:00|250.5 |Vendor A|
|2024-05-01 10:18:00|89.0  |Vendor B|
+-------------------+------+--------+

printSchema() prints:

root
 |-- timestamp: string (nullable = true)
 |-- amount: float (nullable = true)
 |-- vendor: string (nullable = true)

Note the types came from the values themselves: because we only passed column names (no type list), Spark inferred each column's type from the data — "timestamp" is a string, "amount" became float, "vendor" a string. Sense-check: two events in, two rows out, three columns matching the tuple width.

A nuance worth keeping: in the events demo the types are inferred from the values, whereas in the age demo the type was handed over as "int". Both are implicit schemas in the instructor's sense — the type information lives inside the createDataFrame call itself — but the lecture's definition is the hard-coded one: implicit means you pass the types at creation time, in the constructor call.

10.7.3 Explicit Schema with StructType and StructField

With an explicit schema you create a schema object outside the constructor and pass that object to the constructor of createDataFrame. The schema is defined like a structure in the C language: you have a struct type, and in the struct type you have struct fields. The example defines three struct fields. The first is "timestamp" with the corresponding data type StringType; you create the object of StringType. The second struct field is "price" with the corresponding data type FloatType. The third struct field is "vendor" with the corresponding data type StringType. You create this schema explicitly and pass that object to the constructor:

from pyspark.sql.types import StructType, StructField, StringType, FloatType

schema = StructType([
    StructField("timestamp", StringType()),
    StructField("price", FloatType()),
    StructField("vendor", StringType()),
])

events_df = spark.createDataFrame(events, schema)
events_df.printSchema()

Here also you can say printSchema on the events frame and see the types. The same data can be loaded through either schema style; the difference is where the type information lives.

Worked example 3 — the same events through an explicit schema object:

from pyspark.sql.types import StructType, StructField, StringType, FloatType

schema = StructType([
    StructField("timestamp", StringType()),
    StructField("price", FloatType()),
    StructField("vendor", StringType()),
])

events_df = spark.createDataFrame(events, schema)
events_df.printSchema()

Instead of the types living inside the createDataFrame call, we build a standalone schema object first. StructType is the outer container (the struct, like a C struct). Inside it, each StructField pairs a column name with a type object: StructField("timestamp", StringType()) means the column "timestamp" is a string. Then createDataFrame(events, schema) receives the data and the ready-made schema object.

printSchema() shows exactly the three declared fields:

root
 |-- timestamp: string (nullable = true)
 |-- price: float (nullable = true)
 |-- vendor: string (nullable = true)

Sense-check: same data, same three columns — the only difference from Example 2 is that the type information lives in a reusable object outside the constructor instead of inside it.

10.7.4 Student Questions and Answers

Q: Is this called implicit schema or inline schema? I was thinking implicit means you don't have to define it. A student proposed the name inline schema. A: This is implicit. Implicit means when you are defining the data frame, at that time itself you are giving the column type, like this. It is called implicit because at the time of creating the data frame object you are passing these values as hard-coded data types. It is not "inline"; the distinction is between declaring the types at creation time (implicit) versus building a separate schema object and passing it (explicit). The accepted term is implicit schema, because the types are given at creation time. You have already done the explicit thing in the assignment, so you are familiar with that object-based schema.

The correction is worth restating: the student's doubt is natural — "implicit" usually means "you do not have to define it." But the professor's accepted term is implicit schema, defined by where the types are given: at creation time, hard-coded inside the constructor call. The rejected term, inline, would suggest the types appear next to the data as annotations; that is not the Spark reality. So: implicit = types passed at creation time; explicit = types in a separate StructType object passed to the constructor. This vocabulary correction is worth remembering — the word means the opposite of what intuition suggests.

10.7.5 Student Questions and Answers

Q: Which one is advantageous among implicit schema and explicit schema, and why? A: Let us hear your thoughts first. Q (student analysis): Implicit would be advantageous because it is faster coding; you do not have to explicitly define that struct type and struct field. But I think any complex type is probably difficult to define in implicit, so in that case we will have to define it in the explicit way. From this exercise, implicit seems advantageous from the speed of development point of view. A: Yes. Implicit is quite faster for development and quite suitable for the streaming processes here, because streaming is very fast and you do not want to pause to declare schemas. Explicit is more relevant where criticality is high, where you require the explicit orders, like a near real-time system or a soft real-time system where the data streaming requires the explicit declaration. Also for complex data types you need the explicit declaration. If you have many fields, say 20, 25, 30 fields, defining them in implicit is very cumbersome and does not look good from a maintenance point of view; it looks neat in the explicit way. Explicit suits fixed scenarios, like IoT levels things, the streaming signals, which are fixed. When the number of nodes and the sequence are not fixed and the structure is quite dynamic, implicit is more useful; explicit means fixed kinds of structures. And one more advantage you missed: in a distributed environment, with distributed datasets, it is very difficult if you hard-code this into an implicit schema. With explicit schema you can pass a file with the schema definition and simply say "schema is equal to that schema", so everyone need not change the code within the constructor of createDataFrame, because this schema is shared across the system. That is the main advantage of explicit schema. The problem with implicit schema, as you rightly mentioned, is that you need to specify each data type, you are hard-coding it, so reusability is a challenge and it is cumbersome when there are so many fields. If those fields are likely to be modified in the course of time, you never know which part of the code you need to modify. For simple log files, where you know with certainty what schema structure they will have, you can probably use implicit schema, especially when there is not much dependency on the rest of the people in a distributed environment. But if you see exchange of files in a distributed platform, it is always better to use the explicit way.

Implicit vs explicit — the full decision table (from the professor's analysis):

Criterion	Implicit schema	Explicit schema
Development speed	Fast — types written inside the call	Slower — build a schema object first
Streaming suitability	Suited to fast-moving streaming	Needed where criticality is high (near-real-time / soft real-time systems)
Complex types	Hard to declare	Required
Many fields (20–30+)	Cumbersome, poor readability	Neat, easy to maintain
Structure that changes	Better — no fixed structure	Suits fixed structures (e.g., IoT sensor signal streams)
Sharing across a distributed team	Hard — types buried in code	Can pass a schema file; everyone points at the same schema object

The one-line rule: use implicit for speed and dynamic structure; use explicit when types are complex, many, fixed, or must be shared across a distributed team.

10.7.6 Student Questions and Answers

Q: Application-wise, how do we figure out which one to use? Will it be a mixture of both? A: No. In application-wise, either you use this way or you use this way; it is up to you. Suppose you are developing some ML library application. When you are using the ML library, it is always better to use this explicit way, because tomorrow your dataset schema or structure might be different. Then you know: this is the schema, if you just modify here, automatically everything gets the same. You do not have to peep into this constructor of createDataFrame; you modify the schema definition and everything gets done. You can use the DataFrame with a list of tuples, a list of records, and so on and so forth. So it is not a mixture: in any given application you pick one style and stay consistent — and for ML library work the choice is explicit.

Application-wise answer: it is not a mixture. In any given application you pick one style for your schemas, and you stay consistent. The professor's concrete guidance: for ML library work, prefer explicit — because your dataset schema is likely to change tomorrow, and with explicit schemas you modify one schema definition and every DataFrame that uses it updates automatically. You never have to hunt through createDataFrame calls.

Recap + bridge: a schema is the name-and-type description of every DataFrame column; implicit schemas declare types inside createDataFrame (fast, good for streaming and changing structure), explicit schemas build a reusable StructType/StructField object (needed for complex types, many fields, fixed IoT-style streams, and team-shared schema files). Pick one style per application. Next: the RDD world's working example, reduceByKey on key-value pairs.

Real-world: in distributed teams, sharing a schema file across systems is the practical argument for explicit schemas; IoT sensor signal streams with fixed structures are the named explicit-schema case, and fast-changing streaming pipelines are the named implicit-schema case.

10.8 RDD Operations: Key-Value Pairs and reduceByKey

10.8.1 Parallelizing a Collection

The RDD demo works on key-value pairs. In RDD, you pass a list of values, and whenever you use an RDD you just say parallelize, because you want to run across different nodes. Parallelizing a collection spreads the elements of the collection across the nodes of the cluster so each node works on its slice.

Why parallelize? spark.sparkContext.parallelize(list) takes an ordinary in-memory Python list and hands it to Spark as a distributed RDD: the elements are split into partitions, and the partitions are spread across the available nodes (here, the local[3] threads from Section 10.6). Each node then owns a slice of the collection and can work on its slice in parallel. The word is the instruction: make this collection parallel.

10.8.2 Worked Example: reduceByKey and groupByKey

The collection holds pairs of a key and a value, called x and y in the walkthrough. The instructor ran a small list through reduceByKey with the sum operation and walked through every group. (The exact input tuples were partially garbled in the session; the collection below is the reconstruction that reproduces the results narrated in class — key 1 sums to 6, key 2 sums to 7, key 3 sums to 5.)

pairs = [(2, 4), (1, 6), (2, 3), (3, 6), (3, -1)]
rdd = spark.sparkContext.parallelize(pairs)
result = rdd.reduceByKey(lambda a, b: a + b).collect()
print(result)

What reduceByKey does: it reduces the values by the key, and the operation is the sum. For each key, wherever there is a match, the values get combined with the given operation. Walking through the pairs:

Key 1 appears once, with value 6, so the result is (1, 6).
Key 2 appears twice, as (2, 4) and (2, 3). Summing the values: 4 + 3 = 7, so the result is (2, 7).
Key 3 appears twice, as (3, 6) and (3, -1). Summing the values: 6 + (-1) = 5, so the result is (3, 5).

So the collected output is (1, 6), (2, 7), (3, 5). The instructor also pointed out that if you add another pair with an existing key, say a second pair with key 2, that key's total changes accordingly, because now three values must be combined for that key. You do not expect the results in a particular order, because the work is distributed; the output order is not guaranteed. In short, reduceByKey groups the values by key and then reduces each group with the sum operation. The formula for the operation is:

$result (k) = v \in values (k) \sum v$

described in class as "reducing the values by the key" where "the operation is the sum": every key keeps the total of all its values.

Worked example — every step, every key:

Input pairs: (2, 4), (1, 6), (2, 3), (3, 6), (3, -1)

Step 1 — group the pairs by key:

Key	Values
1	[6]
2	[4, 3]
3	[6, -1]

Step 2 — apply the sum to each key's values:

$result (1) result (2) result (3) = 6 = 6 = 4 + 3 = 7 = 6 + (- 1) = 5$

Step 3 — the collected output: [(1, 6), (2, 7), (3, 5)]

Sense-check: every input pair contributed exactly once — 3 + 2 = 5 pairs in, 3 keys out — and each key's total equals the sum of its own values only.

Why reduceByKey is more efficient than groupByKey here: groupByKey would first gather each key's values into a list — key 2 becomes the list [4, 3] — and only then sum the lists. reduceByKey combines values on the way, before and during the shuffle, so far fewer records cross the network. For a sum, the two give the same answer; for huge datasets the difference in shuffled data is enormous. (The groupByKey description — values collected into a list per key, then reduced — is the standard Spark behaviour and matches the instructor's comparison of the two methods.)

10.8.3 Transformations vs Actions

From this example comes the vocabulary of Spark operations. There are two types of tags in Spark: one set of operations is called transformations, and the other set is called actions. reduceByKey and groupByKey are transformations, because they build up the computation without pulling results back. collect is an action, because it actually pulls the data back to the driver. The distinction matters because transformations are lazy, and actions trigger the real computation.

The two tags, with the example in mind:

Transformation — an operation on an RDD that returns a new RDD (e.g., reduceByKey, groupByKey, map, filter). Transformations are lazy: they only build a plan (the lineage) and do no real work yet.
Action — an operation that pulls results back to the driver or writes them out (e.g., collect, count, show). Actions are the moment the engine actually executes the accumulated plan.

In the example, the chain parallelize → reduceByKey → collect does nothing at all until collect is reached. That is when the engine looks at the whole plan and runs it. If the collect() were missing, result would be a plan, not a list — print it and you would see an RDD description, not the numbers.

The beginner trap: a pipeline with only transformations produces no output — nothing is computed until an action fires. Forgetting the final action (like collect()) is the classic "my Spark code does nothing" bug.

10.8.4 Student Questions and Answers

Q: What does this RDD code do? Can you tell me? We have a list of values and call reduceByKey on it. A: We have an RDD, a resilient distributed dataset, and we pass a list of values, then parallelize it because we want to run across different nodes. When I call a method called reduceByKey, what is it doing? It is reducing the elements by the key, and the operation is the sum. X is the key and Y is the value, so wherever there is a match the values are reduced by the key. That is group by, in effect.

Recap + bridge: an RDD is created by parallelize, reduceByKey combines each key's values with an operation (here sum), giving result(k) = sum of values(k); groupByKey collects the values into lists first, and collect is the action that executes the whole lazy plan and returns the output to the driver. The lazy/transformations-vs-actions vocabulary carries straight into the next section's wide and narrow transformations.

10.9 Wide and Narrow Transformations

10.9.1 Narrow Transformations

There are two kinds of transformations: wide transformations and narrow transformations. A transformation is narrow when a single output partition can be computed from a single input partition. It does not require data sets from the other partitions. Narrow transformations operate on one partition and produce a resulting output without any exchange of data while producing the output. Typical examples are filter and contains. filter means a subset: you apply a logic on a given data frame and you filter out the information; by definition you are filtering the given data frame, so that has nothing to do with other partitions. contains checks whether a particular item is part of a data set. Because they operate on a single partition, these are narrow transformations, and in narrow transformations there is no exchange of data.

The narrow rule, one sentence: a transformation is narrow when each output partition depends on exactly one input partition — the node can finish its slice alone, with no data coming in from other partitions. filter (keep only the rows that satisfy a condition) and contains (check whether an item is present) both qualify: every row is decided by its own value. And because no data moves between partitions, a narrow transformation costs no shuffle — no exchange of data while producing the output.

10.9.2 Worked Example: The 18% Tax

The instructor's example of a narrow operation: suppose you have a column of amounts, and for each amount you want to find the tax amount, which is a percentage, let us say 18 percent of it. You transform each row independently; you do not group by anything. For a row with amount $a$ , the tax is:

$tax = 0.18 \times a$

because 18 percent of the amount means multiplying the amount by 0.18. Every row produces its tax from its own value alone, so this transformation is narrow: single output partition from single input partition, no exchange of data.

Worked example — the 18% tax on a small column:

Suppose the amounts in one partition are 100, 250, and 500. Applying the rule row by row:

$tax (100) tax (250) tax (500) = 0.18 \times 100 = 18 = 0.18 \times 250 = 45 = 0.18 \times 500 = 90$

Output: 18, 45, 90. Every output row came from its own input row — no row needed a value from another partition, no grouping, no reshuffling. Sense-check: 18% of 100 is a fifth of the way to 100, and 18 is exactly that; scaling the input by 2.5 (100 → 250) scales the tax by 2.5 (18 → 45), as a multiplication must.

10.9.3 Wide Transformations

A transformation is wide when the operation depends upon data frames from other partitions. Concretely: suppose the data set has two columns, and it is distributed. On machine one we have one data frame, on another machine we have a second data frame, on a third machine a third data frame. All of them have the same columns, but they hold different instances. When you do some computation that requires combining results from all these partitions, you need a wide transformation. Wherever that kind of reshuffling of the individual data frames across partitions is required, the transformation is called wide. Operations like group by or order by require output from the other partitions as well, so they are wide transformations.

The wide rule, one sentence: a transformation is wide when each output partition may depend on data from several input partitions, so the engine must move data between machines — a shuffle (reshuffling of the individual data frames across partitions). group by and order by are the named examples: the total for Chennai cannot be computed from one machine's slice alone; the rows scattered across every partition must first be brought together.

10.9.4 Worked Example: Group By Across Partitions

Suppose machine 1 holds a partial data frame with city and amount columns: Chennai has 2000 and Hyderabad has 5500. Another machine holds another partial data frame with the same columns: again Chennai, and then Bangalore. Because the frames are distributed, the full data for any single city is split across machines. When you apply group by followed by sum, you need to calculate the total of all Chennai rows, the total of all Bangalore rows, and so on. But that requires reshuffling these individual data frames across partitions: the Chennai rows scattered on both machines must come together. Wherever that kind of reshuffling is required, the transformation is wide.

Worked example — group by city across two machines:

Machine 1 holds:

city	amount
Chennai	2000
Hyderabad	5500

Machine 2 holds:

city	amount
Chennai	3000
Bangalore	1200

(These are the demo frame's values as narrated in class; the exact amounts on screen were partially garbled, but the structure — partial frames per machine with the same columns — is as described.)

Step 1 — each machine computes its local partials for group by city, sum:

$Machine 1: Machine 2: Chennai: 2000, Hyderabad: 5500 Chennai: 3000, Bangalore: 1200$

Step 2 — the reshuffle: every machine's Chennai partials move so that all Chennai rows sit together for the final combine. This is the shuffle — the exchange of data that makes the transformation wide.

Step 3 — combine the partials per city:

$total(Chennai) total(Hyderabad) total(Bangalore) = 2000 + 3000 = 5000 = 5500 = 1200$

Final totals: Chennai 5000, Hyderabad 5500, Bangalore 1200. Sense-check: every row was counted exactly once — 2000 + 3000 = 5000 Chennai total matches the sum of the two Chennai rows — and the shuffle is precisely the step a narrow transformation never needs.

Real-world: group-by-city totals across a distributed click or transaction log is exactly the pattern behind dashboards that show regional sums; the reshuffle cost is why such queries get tuned in production.

10.9.5 Student Questions and Answers

Q: Maybe a wide transformation is when multiple values, more than one value, are transformed, like X and Y where I transform X as well as Y. A student guessed that wide transformations transform several values. A: That is not the definition. The key is not how many values you transform; it is whether the transformation depends on data frames from other partitions. If I have DF1 on machine 1 with city and amount, and a partial data frame on another machine, then to compute group by city with sum I need to bring all Chennai rows together, and that requires reshuffling of these individual data frames across partitions. Wherever that kind of reshuffling is required, such transformations are called wide transformations. The definition is whether data from other partitions is needed. A narrow transformation, on the other hand, does not require data sets from the other partitions.

The correction, restated: the number of values transformed is irrelevant. A transformation that processes two columns per row (e.g., tax = 0.18 × amount) is still narrow — every row still resolves locally. The test is the shuffle, not the arity.

Q: For the contains method: let us say a partition has a list of values like 2, 3, 4, 5, 6, and I want to count how many records are within that partition with city Chennai. I can use contains within that partition only, right? My confusion: partition 1 has some Chennai records and partition 2 also has Chennai records. Don't I need to combine those two and then check whether it contains? A: It depends upon the use case. If your use case definitely requires the data frames from other partitions, anyway you need to use wide transformations. But if you are just checking for some filter, then probably you may not require the other partitions; the filtering or checking is within that partition, or you are checking locally. That is why contains and filter are normally applied on local partitions. Filter means subset: you are applying a logic on a given data frame and you are filtering out the information, and that has nothing to do with other partitions, because by definition you are filtering the given data frame. The way to think about contains: given this data frame, you want to check whether a particular thing is part of this data frame or not. That is the more natural instinct, rather than combining all the data. Whereas for group by or order by, the natural instinct would be combining by city or pin code, because this is a distributed data set across multiple partitions; group by within that particular data frame is not what you expect, you expect it for the entire original data frame, which means you need to pull the data from the remaining partitions as well.

The resolution, restated: the answer depends on the use case, and the professor's mental model is what decides it — contains asks "is this item part of this data frame?" — a local question answered within each partition. group by / order by ask about the entire original data frame — a global question that by definition needs rows from the remaining partitions. Same dataset, different questions: the local question stays narrow, the global question becomes wide. If your use case genuinely requires data from other partitions, then you need a wide transformation — but a plain filter or membership check does not.

Recap + bridge: narrow transformations compute each output partition from one input partition with no exchange of data (filter, contains, per-row tax); wide transformations need data from several partitions and trigger a shuffle (group by, order by, the Chennai totals). The wide/narrow vocabulary completes the RDD foundation — next session: what comes after the RDD basics.

10.10 What Comes Next: RDDs Are Redundant

10.10.1 The Roadmap

The instructor closed the hands-on part with the plan for the coming sessions. The next session will brief the class on some Spark RDD basic operations, how exactly RDD operations happen from what was discussed. Then a couple of sessions will be dedicated to Spark Structured Streaming, maybe half a lecture on RDD. There is a clear signal about where effort should go: RDDs are not being used, and the instructor says the reasons will be explained next time because RDDs are actually redundant. The focus will be predominantly on structured streaming. So the study priority is set: understand the RDD basics as a foundation, but expect the real course weight on Spark Structured Streaming.

Why RDDs are called redundant: the high-level APIs — DataFrames, Spark SQL, Structured Streaming — already do everything RDDs do, with more features and less boilerplate: automatic schema handling, query optimization, and declarative queries (Section 10.4). An RDD still underlies all of them, but as a programming surface it has been superseded. That is why the course weight sits on structured streaming: the same ideas (partitions, transformations, actions, shuffles) reappear there in a friendlier form.

Study priority, in one sentence: learn the RDD basics well enough to understand what is happening under the hood — partitions, lazy transformations, actions, wide vs narrow — but put the real effort into Spark Structured Streaming, where the course (and the exam-relevant work) actually lives. Do not go deep into hand-written RDD pipelines; they are the redundant layer.

The instructor also acknowledged that the material can feel a little rusty after a long gap, with sessions spaced out, and promised that the concrete examples to come will consolidate the concepts. When you get into concrete examples, the definitions settle. So the message to students is to keep the definitions from this session in hand, and the practice sessions will make them stick.

Keep these definitions fresh — they are the ones the practice sessions will consolidate:

Batch vs real-time/micro-batch processing, and why streaming needs long-term storage (10.1)
Implicit vs explicit schema (10.7)
Transformations vs actions (10.8)
Wide vs narrow transformations (10.9)

The coming hands-on examples turn each of these from definitions into reflexes.

Exam note: no exam-specific guidance was given in this session beyond the assignment note. The assignment is based on the ML library (MLlib): Part A is a demo, which can be shown with screenshots, and Part B is a Jupyter notebook plus an HTML file, submitted together as a single artifact; screenshots or images can be embedded in the notebook or HTML instead of sending separate image files.

Exam Guidance Summary

No exam-specific guidance was given in this session. What students were told that matters for assessment:

The course assignment is based on the ML library (MLlib), and the algorithms named in the architecture discussion — classification, regression, and unsupervised learning like collaborative filtering and clustering — are the ones to work with.
The assignment submission is structured in two parts: Part A is a demo that can be shown with screenshots, and Part B is a Jupyter notebook plus an HTML file. Both parts are submitted together as a single artifact. Images (architecture diagrams, screenshots) can be embedded in the notebook or the HTML rather than attached as separate files, so one file can be good enough.
The study priority for the module is Spark Structured Streaming. The RDD basics are worth learning as the low-level foundation, but the class was told the focus will be predominantly on structured streaming and that RDDs are redundant in practice.
Keep the definitions from this session fresh — batch vs streaming, implicit vs explicit schema, transformations vs actions, wide vs narrow transformations — because the coming hands-on examples are what consolidate them.

Key Industry Applications

Bank applications and stock market data are the named near-real-time environments where Spark streaming's event processing is very helpful. (10.1)
Ingestion into Spark can come through Kafka or simple TCP/IP socket programming, and the sources named include Kafka, the Hadoop file system, Amazon S3, and social platforms like Twitter. (10.1, 10.4)
Downstream, processed output can be stored into a database or turned into a visualization dashboard. (10.4)
Spark integrates naturally with Hadoop, Kafka, and AWS. (10.2)
The ML library (MLlib) is built in and covers classification, regression, collaborative filtering, and clustering, and it is the basis of the course assignment. (10.3)
GraphX, built on the RDD API, handles network analysis and social network analysis, and the named application is PageRank, finding the most popular websites based on real-time web activity of users. (10.3)
Kafka itself is available in two flavors for integration: Apache Kafka or Confluent Kafka. (10.5)
In distributed teams, explicit schemas are shared as a file across the system, which is the practical reason to prefer them over hard-coded implicit schemas; IoT signal streams with fixed structures are the named explicit-schema scenario. (10.7)
Group-by-total queries across distributed data require reshuffling, which is the wide-transformation pattern behind regional aggregate dashboards in production. (10.9)

SPA Lecture 10 notes · Spark Streaming: Architecture, Setup, DataFrames, and RDDs

Stream Processing and Analytics· postgraduate· 2026-08-07

Sections Breakdown

110.1 Spark Streaming: Two Ways to Process Data

Batch vs real-time processing, micro-batches, Kafka as queue and Spark as compute, and why streaming needs long-term durable storage.

210.2 Key Properties of Spark

In-memory computation, one engine from batch to streaming, iterative analytics, automated pipelines, four languages, and Hadoop/Kafka/AWS integration.

310.3 Spark Architecture: Core, Libraries, and Cluster Managers

Spark core, MLlib, Spark SQL with structured streaming, GraphX and PageRank, and the cluster manager's scheduling, memory, fault tolerance, and database duties.

410.4 Two Ways to Write Spark Applications: RDD and Structured Streaming

The low-level RDD API with map and reduce versus the high-level SQL-driven Structured Streaming world.

510.5 Setting Up Spark: Installation and Configuration

Spark-Hadoop pairing, folder structure, environment variables (PYSPARK_PYTHON, PYTHONPATH, Py4j), log4j tuning, and the pyspark smoke test.

610.6 SparkSession: The Entry Point

The singleton pattern, if __name__ == '__main__', appName, master('local[3]'), and key-value configurations.

710.7 DataFrames and Schemas

DataFrame basics, implicit schema declared at creation time, explicit StructType/StructField schemas, and when to use each.

810.8 RDD Operations: Key-Value Pairs and reduceByKey

parallelize, reduceByKey with sum, result(k) = sum of values(k), and transformations versus actions.

910.9 Wide and Narrow Transformations

Narrow transformations with no data exchange, wide transformations with reshuffling, the 18% tax and group-by-city worked examples.

1010.10 What Comes Next: RDDs Are Redundant

The roadmap: RDD basics then structured streaming, why RDDs are redundant, and the study priority.

11Exam Guidance Summary

Assignment structure on MLlib: Part A demo with screenshots, Part B notebook plus HTML as one artifact, structured streaming study priority.

12Key Industry Applications

Banking and stock-market event processing, Kafka and socket ingestion, MLlib assignment basis, GraphX PageRank, and regional aggregate dashboards.

Postgraduate students in Stream Processing and Analytics

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Spark Streaming: Two Ways to Process Data

Must-know: Kafka is the message queue; Spark does the computation. Streaming needs long-term storage because the incoming rate of messages is much higher than processing speed.

⚠️ Top pitfall: Assuming streaming systems can process messages as fast as they arrive; without durable storage the stream overruns the processor and messages are dropped.

Self-check: Why does a streaming architecture need long-term storage even though it is processing in real time?

Key Properties of Spark

Must-know: Spark does in-memory computation; reducing the batch size turns batch processing into streaming; supported languages are Java, Python, Scala, and SQL, with R absent from the streaming stack.

⚠️ Top pitfall: Thinking streaming needs a different engine than batch; it is the same engine with a smaller batch size.

Self-check: Why is R missing from Spark's streaming language list even though it is popular for ML and visualization?

Connects to: 10.1

Spark Architecture: Core, Libraries, and Cluster Managers

Must-know: The course assignment is based on MLlib, covering classification, regression, collaborative filtering, and clustering. Cluster managers are YARN (default), Mesos, standalone, or Kubernetes.

⚠️ Top pitfall: Forgetting that YARN is Spark's default cluster manager and that GraphX is built on the RDD API.

Self-check: Name the four key activities of the cluster manager layer.

Connects to: 10.2; 10.4

Two Ways to Write Spark Applications: RDD and Structured Streaming

Must-know: RDD is a very low-level API providing simple operations like map and reduce, writable in Python, Java, or Scala; Spark SQL belongs to the structured streaming world, a different API family.

⚠️ Top pitfall: Treating SQL as another way to write RDD code; they are different API families with different mental models.

Self-check: What does RDD stand for and what does each word mean?

Connects to: 10.3

Setting Up Spark: Installation and Configuration

Must-know: Spark tarballs are pre-built against a Hadoop version; keep the displayed pairing (Spark 3.0.x with Hadoop 2.7.7). PYSPARK_PYTHON points Spark at the Anaconda Python; PYTHONPATH includes the python/lib folder with Py4j.

⚠️ Top pitfall: Downloading a Spark build paired with the wrong Hadoop version, or skipping PYTHONPATH so Py4j cannot be found and Python workers fail.

Self-check: Why does the Spark download page show a Hadoop version next to each Spark tarball?

SparkSession: The Entry Point

Must-know: SparkSession is a singleton built with .appName, .master('local[3]') (three local worker threads on your own machine), and .config key-value pairs; code goes inside an if __name__ == '__main__' block.

⚠️ Top pitfall: Confusing local[3] with three machines; it is three worker threads on one machine. Forgetting getOrCreate() means no session exists.

Self-check: Why is the SparkSession restricted to one instance in a program?

Connects to: 10.7

DataFrames and Schemas

Must-know: Implicit schema means types are passed hard-coded at creation time inside createDataFrame (fast, good for streaming and dynamic structure); explicit schema builds a StructType object passed to the constructor (needed for complex types, many fields, fixed structures, and shared schema files).

⚠️ Top pitfall: Reading 'implicit' as 'no types needed'; implicit still declares types, just inside the constructor call.

Self-check: Why is explicit schema better in a distributed team even though implicit is faster to write?

Connects to: 10.6

RDD Operations: Key-Value Pairs and reduceByKey

Must-know: reduceByKey reduces the values by the key with a given operation: result(k) = sum over values(k). reduceByKey and groupByKey are transformations (lazy); collect is an action that pulls results to the driver.

$result (k) = v \in values (k) \sum v$

⚠️ Top pitfall: Forgetting the final action: a chain of transformations alone computes nothing because they are lazy.

Self-check: Why is reduceByKey usually more efficient than groupByKey for a sum?

Connects to: 10.9

Wide and Narrow Transformations

Must-know: Wide vs narrow is decided by whether the transformation depends on data from other partitions (requiring reshuffling), not by how many values are transformed. Narrow: filter, contains, per-row tax. Wide: group by, order by.

$tax = 0.18 \times a$

⚠️ Top pitfall: Defining wide by the number of values transformed; the real test is whether other partitions' data is needed (shuffle).

Self-check: Why is computing the total of all Chennai rows across two machines a wide transformation?

Connects to: 10.8

What Comes Next: RDDs Are Redundant

Must-know: RDDs are redundant in practice; the focus will be predominantly on Spark Structured Streaming. The assignment is based on MLlib: Part A demo (screenshots allowed), Part B Jupyter notebook plus HTML submitted together.

⚠️ Top pitfall: Spending study effort on hand-written RDD pipelines instead of structured streaming, where the course weight lies.

Self-check: Why does the instructor call RDDs redundant even though they are still used under the hood?

Connects to: 10.4

Previous LectureLecture 09 · Kafka Streaming Demo and a High-Level View of Spark

Next LectureLecture 11 · Spark Structured Streaming: Word Counts, Output Modes, and Windows

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

10.1 Spark Streaming: Two Ways to Process Data

10.1.1 Batch Processing vs Real-Time Stream Processing

10.1.2 Where Streaming Is Useful

10.1.3 Why Streaming Needs Long-Term Storage

10.2 Key Properties of Spark

10.2.1 In-Memory Computation

10.2.2 One Engine from Batch to Streaming

10.2.3 Iterative Aggregations and Analytics on Real-Time Data

10.2.4 Service Delivery Automation (Data Pipelines)

10.2.5 Language Support: Java, Python, Scala, SQL — and Why Not R

10.2.6 Integration with Other Platforms

10.3 Spark Architecture: Core, Libraries, and Cluster Managers

10.3.1 Spark Core and the High-Level Libraries

10.3.2 The ML Library (MLlib)

10.3.3 The Graph Library (GraphX)

10.3.4 Cluster Managers

10.3.5 Key Cluster Activities

10.4 Two Ways to Write Spark Applications: RDD and Structured Streaming

10.4.1 The Resilient Distributed Dataset (RDD)

10.4.2 Structured Streaming and the SQL Story

10.4.3 Student Questions and Answers

10.5 Setting Up Spark: Installation and Configuration

10.5.1 Downloading Spark and Pairing It with Hadoop

10.5.2 Folder Structure and Configuration Files

10.5.3 Environment Variables: Java, Hadoop, Kafka, and PySpark

10.5.4 Taming Log Noise with log4j

10.5.5 Verifying the Installation

10.5.6 Student Questions and Answers

10.6 SparkSession: The Entry Point

10.6.1 The Singleton Pattern

10.6.2 Master URL and Configurations

10.7 DataFrames and Schemas

10.7.1 Creating a DataFrame from a Collection

10.7.2 Implicit Schema

10.7.3 Explicit Schema with StructType and StructField

10.7.4 Student Questions and Answers

10.7.5 Student Questions and Answers

10.7.6 Student Questions and Answers

10.8 RDD Operations: Key-Value Pairs and reduceByKey

10.8.1 Parallelizing a Collection

10.8.2 Worked Example: reduceByKey and groupByKey

10.8.3 Transformations vs Actions

10.8.4 Student Questions and Answers

10.9 Wide and Narrow Transformations

10.9.1 Narrow Transformations

10.9.2 Worked Example: The 18% Tax

10.9.3 Wide Transformations

10.9.4 Worked Example: Group By Across Partitions

10.9.5 Student Questions and Answers

10.10 What Comes Next: RDDs Are Redundant

10.10.1 The Roadmap

Exam Guidance Summary

Key Industry Applications

SPA Lecture 10 notes · Spark Streaming: Architecture, Setup, DataFrames, and RDDs

Sections Breakdown

Exam Revision Notes

Spark Streaming: Two Ways to Process Data

Key Properties of Spark

Spark Architecture: Core, Libraries, and Cluster Managers

Two Ways to Write Spark Applications: RDD and Structured Streaming

Setting Up Spark: Installation and Configuration

SparkSession: The Entry Point

DataFrames and Schemas

RDD Operations: Key-Value Pairs and reduceByKey

Wide and Narrow Transformations

What Comes Next: RDDs Are Redundant

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat