Apache Spark: Fast, General-Purpose Cluster Computing

7.1 The Four Kinds of Big Data Processing

We finished the cloud computing module with the different kinds of storage available on the cloud, and we also covered the network and compute resources on the cloud. That discussion was focused on storage. We looked into the Dynamo paper published by Amazon — its requirements, and some technologies built to deal with those requirements. Exam note: a few questions from that portion are still open, and they will be picked up once the syllabus is complete.

The four kinds are batch, stream, interactive, and graph processing. They are not four brands of software; they are four patterns of work. One engine may support several, and the same piece of data can be handled in more than one pattern depending on the question being asked.

7.1.1 Batch Processing

Batch processing works on blocks of data that have already been stored for a period of time. The name says it all: the data sits around in a "batch," and only when the time is right does a program chew through the whole block at once.

Suppose a financial organization has logged transactions over the past one week, and it wants to process those transactions on a weekly basis. Every Sunday night, a program opens the week's stored transactions, runs over all of them, and produces the summary report. Any kind of program will do — for example, a MapReduce job written to carry out that task. That is batch processing, and we already know how to do it with Hadoop.

The key property to notice is the delay between data arrival and processing. The transactions arrive continuously on Monday through Saturday, but nothing touches them until the batch run. Batch processing is the right tool whenever the answer can wait — payroll summaries, end-of-day sales reports, index rebuilding. The data is already on stable storage, so the engine reads it from disk, processes it, and writes results back to disk.

7.1.2 Stream Processing

Stream processing carries out certain analytics in real time. The system processes the data as it arrives and quickly detects deviations or conditions within a small period of time from the point of receiving the data. Whatever data is in memory, you run a computation on top of it and produce real-time analytics. That is stream processing, and Spark helps you to do that.

Where batch says "wait until the data piles up," stream says "act while the moment is still alive." The window between arrival and response is seconds, not days. That is what "small period of time" means: the system must finish the check before the condition it is checking for stops mattering.

Real-world: Fraud detection is one good example of stream processing. A transaction arrives, and the system must decide almost immediately whether it is fraudulent or genuine, before the moment passes. A fraud check that runs a week after the purchase protects nobody — the money is already gone.

7.1.3 Interactive Processing

Interactive processing means the user works with results one query at a time. I have written one query; based on the results of that, I want to further dig down; based on the results of the second, I want to dig down again. Each step depends on the previous one, so the engine has to answer quickly.

Think of it as a conversation with the data: query → look at the answer → ask a sharper question → look again. The user, not the clock, drives the sequence, and the wait between query and answer must be short enough to keep the human engaged. If every query took ten minutes, the dig-down flow would die. Tools such as Impala and Presto support interactive processing, with Presto being a distributed SQL query engine designed for exactly this query-while-you-think workflow. These tools are not part of this course, but they show what big data computing engines support.

Notice how stream and interactive differ even though both are "fast." Stream processing runs continuously and automatically on incoming data. Interactive processing runs only when a user fires a query, and each query is a new step decided by the previous answer.

7.1.4 Graph Processing

Graph processing suits relationship-heavy data, where one record may have multiple relationships. We have seen that such data is easy to model using graph databases like Neo4j rather than storing it in an RDBMS, and that the data stored in the graph is processed using the Cypher query language.

In an RDBMS, a person, a product, and a company each live in a table, and the links between them need join tables and join queries. In a graph model, the links are first-class citizens: a person record can point directly at dozens of products, friends, and companies. The work here is not "sum all rows" — it is "follow the connections," like finding the shortest path between two people in a social network, or detecting that one account sits suspiciously close to many others.

7.1.5 The Case for a Unified System

So far we have different kinds of storage and different kinds of processing, each with its own tooling and programming interface. Batch has MapReduce, streaming has its own pipelines, interactive queries have their own engines, graphs have their own databases. Every switch of workload means learning a new tool, moving data between systems, and paying the transfer cost.

Can we bring all of this into one system, where the underlying storage layer is the same, the programming platform is the same, and we can still do all four kinds of computation? That question is where Spark comes into the picture.

7.2 What Apache Spark Offers

Spark is the answer to the unified-system question: one engine, one programming platform, and all four kinds of computation on top of the same storage layer. This section looks at what Spark is and what it promises.

7.2.1 A Fast, General-Purpose Cluster Platform

Spark is a cluster computing platform — software that spreads computation over a group of machines — designed to be fast as well as general purpose. "General purpose" means it is not specifically a graph processing application, not specifically batch processing, not specifically stream processing: you can do any kind of processing using Spark. Speed matters for large data sets, because it creates a difference when data is being explored. Spark extends the MapReduce model to efficiently support more types of computations and runs computations in memory. We will look at the challenges that exist with Hadoop MapReduce and at how Spark deals with those challenges to speed up operations.

7.2.2 Generality Across Workloads

Generality means one engine covers a wide variety of workloads: batch applications, iterative algorithms, and interactive queries all fall under Spark. It is also easy and inexpensive to combine the different processing types in pipelines, so a job can read a batch, filter it interactively, and stream the output without moving between systems.

Take that pipeline example slowly. A company stores a week of click logs as a batch file. A Spark batch job reads the file and cleans it. An analyst then asks interactive questions about the cleaned data — "which products did users from India click?" — and the answers come back quickly because the data is already in memory. The same application can then stream the recommended products out to a live dashboard. In the Hadoop world, that journey would mean at least three different tools and several data handovers. In Spark, it is one application.

7.2.3 Language Support and Ecosystem Integration

Spark offers simple APIs for Java, Python, Scala, and SQL — a good amount of language support. It can integrate with Hadoop and other big data ecosystem tools, with streaming platforms such as Kafka, and with other big data tools offered on the cloud. Good language support and good integration with tools that already exist.

This matters for a practical reason: a team rarely starts from zero. They already have data in HDFS, events flowing through Kafka, and a cloud storage bucket. Spark does not ask them to re-architect; it talks to what is already there. And because the API exists in several languages, a team of Java developers and a team of Python developers can work on the same engine without fighting over language.

7.2.4 Spark and Hadoop: Friends, Not Replacements

Spark was introduced to address the speedup of Hadoop computation, but it is not a modified version of Hadoop. You can still use Hadoop as an option for cluster management using YARN, or for HDFS storage. The point is that you can run Spark on top of your Hadoop cluster as well. Recall the Hadoop 1.0 versus 2.0 story, where some decoupling happened between the two: version 2.0 started supporting non-HDFS, non-Hadoop storage as well. So if you have a Hadoop cluster, you do not need to move away from it — YARN keeps managing the cluster and its resources, and Spark takes over the processing that MapReduce used to do, which speeds up operations further.

Spark has its own in-memory cluster computing engine built around the RDD — the resilient distributed dataset, which we will examine in detail. It supports more than MapReduce: SQL, streaming, machine learning (regression, classification, and so on), and graph processing are all available.

7.3 Why MapReduce Is Slow, and How In-Memory Computing Fixes It

Before looking at Spark's machinery, we need to see the problem it solves. Hadoop MapReduce is not slow because the idea is wrong; it is slow because of how the data moves — through the disk, again and again. This section walks the MapReduce pipeline, names each cost, and then shows the in-memory architecture that removes most of those costs.

7.3.1 The MapReduce Pipeline Moves Data Through Disk

To see why Spark exists, follow what a Hadoop MapReduce job does when iterations happen. The data is read from the disk, which is the stable storage, and then divided into splits. Based on the splits, a number of map tasks run — say m1, m2, m3. The output of those map tasks is stored on the disk through an HDFS write operation. A shuffle and sorting step happens under the covers, some data is given to the reduce step, which groups that data based on a particular key, and finally the data is stored on the hard drive again.

In short: you read the data from a file, you write the intermediate data into a file, you read it again, and you write it again. Reading and writing a disk is a costly and time-consuming operation, and that is what reduces the performance of MapReduce algorithms.

Picture the flow as a chain with a hard drive at every link:

file on disk  →  splits  →  map tasks (m1, m2, m3)  →  disk (HDFS write)
    →  shuffle and sort  →  reduce (group by key)  →  disk (final write)

For a single pass this is already expensive. Now imagine an algorithm that needs ten iterations — an iterative job such as a machine learning routine that refines a model ten times. Each iteration repeats the whole chain, so the data travels to disk and back ten times. The disk, not the computation, becomes the bottleneck.

7.3.2 Replication Factor Three

There is a second cost on top of the disk traffic. HDFS has a default replication factor of three, so whatever read or write operation you perform, the system maintains three copies. For one write, three writes are actually happening. Imagine that added to the read-write pipeline just described.

The replication factor exists for a good reason — it is how HDFS survives a machine dying without losing data. But the price is physical: every logical write of 100 GB means \(3 \times 100 = 300\) GB actually written to disk across the cluster. Reads can sometimes be satisfied from the nearest replica, but the writes always pay the full triple cost. In the example above, the 400 GB of logical disk traffic becomes roughly 1.2 TB of physical disk traffic per iteration.

7.3.3 Serialization and Deserialization

MapReduce applications also carry serialization and deserialization overheads. Serialization is converting an object, or data, into a byte stream in order to transfer it. Deserialization is converting that byte stream back into data at the other end. Every transfer between stages pays both conversions, and these are significant overheads.

A record in memory is a rich object: it has fields, types, and structure. Before the record can cross the network or sit in a file, the framework must flatten it into a plain sequence of bytes — that is serialization. The receiver must then rebuild the object from those bytes — deserialization. Neither step is free, and MapReduce performs both at every stage boundary: map to disk, disk to reduce, reduce to disk.

7.3.4 Interactive Queries and Time-Critical Jobs

The same problem hits interactive work. There is input on the stable storage, you write queries on top of it, and you get results. Each query reads from stable storage, so disk I/O dominates the application execution time. This applies even to tools you would not expect: Hive, and SQL written on top of it, runs MapReduce under the covers to retrieve the data from the tables. The underlying structure is Hadoop based and MapReduce based, so reading and writing from the disk poses inefficiencies and is likely to increase latency.

Interactive work is hurt twice. First, each query is a fresh disk read — the "conversation with the data" from section 7.1.3 never warms up. Second, the user is waiting, and a human cannot sustain a dig-down flow when every step takes minutes. The latency problem infects even Hive, which looks like a query tool but quietly executes MapReduce jobs on the disk underneath.

Real-world: Some applications are time critical. Specifically, fraud detection: as the data comes in, you need to analyze it and provide certain insights — whether this is a fraudulent transaction or a general transaction. Waiting for disk reads at every step is not acceptable there. In-memory computing using RDDs is the feature of Spark that eliminates these kinds of inefficiencies.

7.3.5 The In-Memory Computing Architecture

Picture the in-memory architecture. The data is read from the disk for the first time and given to the first MapReduce iteration. The intermediate results are not stored on the disk. Instead they are put into a distributed memory — the main memory of the cluster nodes, pooled together. The second iteration of MapReduce reads its input from that memory, puts its results back into the distributed memory, and so on across any number of iterations. Only at the end are the final results saved onto a stable storage.

With this kind of architecture, operations speed up, because you are reading and writing from memory rather than from the disk at every iteration. You read from the disk initially and put final results onto a disk at the end; you do not store every iteration's results on the disk. The disk overhead drops significantly.

disk →  iteration 1 →  distributed memory →  iteration 2 →  distributed memory
     →  ... →  final iteration →  disk

The distributed memory is just the RAM of all cluster nodes treated as one shared pool. A node writes its intermediate result to its own RAM (or a neighbor's, when the computation demands it), and the next iteration reads from RAM at memory speed — roughly a thousand times faster than a disk seek, and without the replication multiplier.

7.3.6 Spillover to Disk

Memory may be limited. If the data is so large that it cannot fit into this memory, then a certain portion of the data is saved on the disk. This is spillover, and the spilled portion can be transparently stored on the disk. The system keeps working, though performance degrades as more data spills.

The word to notice is transparently: the program does not crash, and the developer does not write any special code. The engine quietly moves the overflow to disk and carries on. The cost is hidden performance: every spilled byte reintroduces the disk latency the architecture was built to avoid. This is the trade-off behind the later advice to size cluster memory to the data — spillover is the in-memory architecture's safety valve, not its normal mode.

7.3.7 In-Memory Architecture for Interactive Queries

The same architecture works for interactive query processing. Intermediate results stay in the distributed memory without complicating the user's programming. Once the read from the disk into the distributed memory happens, it is a one-time processing. Queries fired by the end user run against the data in the distributed memory time and again — the engine does not go into the hard drive for each query. Results are computed from memory and passed to the users each time.

This is what rescues the interactive dig-down flow. The expensive disk read happens once, up front. After that, every query is a memory lookup, and the user gets the fast back-and-forth that interactive work needs. The programmer does nothing special to get this behavior — it is built into the engine.

7.4 RDD: The Resilient Distributed Dataset

The in-memory architecture is implemented through one central data structure: the RDD. Every other Spark abstraction — DataFrame, Dataset, stream, graph — is built on it, so getting a feel for the RDD is getting a feel for Spark itself.

7.4.1 What an RDD Is

The RDD (resilient distributed dataset) is the fundamental data structure in Spark — a distributed, immutable collection of objects. Immutable means you cannot change it once you have created it. If you want to update an RDD, you do not modify it; you create a new RDD whenever you want updated values.

Three words carry the whole definition. Distributed: the collection is spread across the memory of several machines, not sitting in one process. Immutable: once created, its contents never change — no in-place editing exists. Collection of objects: it holds data items, and the items can be anything the language supports, from strings to whole records.

The immutability rule sounds like a restriction, but it is the source of the structure's strength. Because an RDD never changes, it can be safely shared across machines with no risk of one machine corrupting what another reads — and if any copy is lost, the original description of how to build it is still valid, because nothing was overwritten.

7.4.2 Creating an RDD

You can create an RDD from other RDDs, or you can parallelize an existing collection in the driver program. Suppose your data is a list with the values 1, 2, 3, 4, 5. You create an RDD over it like this:

sc.parallelize(data)

Here sc is the SparkContext. Once you install Spark on your system and move to the Spark shell, this variable already exists, and you call certain methods on it — that is what we are doing here. Parallelizing spreads the data across the memory of the different nodes, which is why the collection becomes a distributed dataset. You can also refer to a data set in external storage such as HDFS or HBase and create an RDD from that; that is another way to create an RDD.

The SparkContext is the front door of a Spark application: it holds the connection to the cluster, tracks every RDD the application creates, and starts the jobs that compute them. In the interactive shell the context is created for you and exposed as sc; in a standalone program your code builds it explicitly.

7.4.3 Why "Distributed": Partitions

Each RDD is divided into logical partitions so that it can be computed on in parallel across the cluster. The cluster has multiple nodes, each node has memory, and the RDD's partitions are spread over them so you can operate on the RDD in parallel. An RDD can contain any type of objects, depending on the language.

A partition is a logical slice of the RDD — "logical" because it exists in the engine's plan, not as a physical file. Each partition can be processed independently by a separate task, and independent tasks can run at the same time on different cores and different machines. More partitions generally mean more parallelism: an RDD split into 40 partitions can in principle keep 40 tasks busy, while a single-partition RDD is processed by one task no matter how big the cluster is. When an RDD is read from HDFS, a natural partition boundary is the HDFS block — one block, one partition — so the engine processes exactly where the data lives. This is the same "move compute to the data" principle we used in Hadoop.

7.4.4 Why "Resilient": The Lineage Graph

Resilience is the very important part. When an RDD is created, and every time an operation is applied to it, those operations are recorded. If the RDD gets corrupted, a lineage graph of operations helps to reconstruct it when a node fails and a part of the RDD is lost. The RDD is distributed across multiple nodes; if one node fails, you trace back through the operations stored in the graph and replay them to reconstruct the RDD that was lost. That is why the structure is called resilient.

Every RDD remembers its own family tree. The tree's nodes are operations — "created from a list," "mapped with this function," "filtered with this predicate" — and its edges point from each RDD to the RDDs it was built from. When a node dies and a partition disappears, the engine finds the lost partition in the lineage, locates its parent partitions, and replays the recorded operations to rebuild it — on another healthy machine if needed.

original list [1, 2, 3, 4, 5]  →  parallelize  →  RDD-A
RDD-A  →  map(x => x * 2)  →  RDD-B
RDD-B  →  filter(x => x > 5)  →  RDD-C

If a partition of RDD-C is lost, the engine rebuilds it from RDD-B's surviving partitions by re-running the filter — and if RDD-B's needed partition is also gone, it replays the map over RDD-A, whose contents are still on disk or in the original program. The recipe metaphor holds exactly: to recover a dish, you do not need a backup of the finished dish; you need the recipe and the raw ingredients.

7.4.5 A Short History: RDD, DataFrame, Dataset

The RDD concept started in 2011, and it was meant to deal with unstructured data: we do not want to impose a schema, we are not interested in the optimizations done for structured data, and we want low-level operations. In 2013, people started putting some structure onto it, and the DataFrame came into the picture. In 2015, the Dataset came along, where we became concerned about type safety — what kind of data we are putting into the data set — so it is type safe and fast. An RDD is a distributed collection of raw objects; a Dataset is a distributed collection of JVM objects with type checking. DataBricks is the organization that contributed to RDD and this ecosystem.

The three abstractions form a timeline of one idea maturing. First came raw power: RDDs, deliberately schema-free, for programmers who want to control every operation. Then came convenience: DataFrames added a schema and column-like organization, which let the engine optimize queries the way databases do. Then came safety: Datasets added compile-time type checking, so mistakes about data shapes are caught before the job runs. The old abstraction never dies — each new one sits on top of the RDD engine.

7.4.6 The Features of an RDD

First, an RDD keeps the data in memory as much as possible — this is the in-memory computing. If a huge amount of data arrives and cannot be retained in memory, it spills over to the disk, with the downsides noted earlier.

Second, evaluation is lazy: an RDD is evaluated only when an action triggers. When you apply an operation on an RDD, a new RDD is not created immediately. The operations are stored in a graph, and the intermediate RDDs are not computed. Only when an action — a certain trigger — is performed does Spark trace back through the graph and create the new RDD. This may not be fully clear yet, but it will become clearer with some programming constructions.

Third, a failed or lost RDD partition on a worker can be recovered from the lineage of operations. Every operation done on an RDD is stored in the form of a graph, so if some portion of the RDD is lost because a node failed, you can always reconstruct it from the lineage of operations.

Fourth, an RDD is immutable. You cannot update an RDD once it is created; you always create a new RDD with the new values.

Fifth, persistence is supported. You can process an RDD in memory for storage, or save an RDD onto a stable storage — that possibility exists.

Sixth, parallelism comes from partitioning. When you create an RDD, it is partitioned and put onto different nodes in the memory, which supports parallelism. As in Hadoop — where we always move the compute closer to the data — here also tasks are put closer to the data location.

Seventh, operations are coarse-grained: you apply operations on the entire set of data, not on a data item within the RDD. It is not a fine-grained control — you cannot pick a single row and apply an operation on that.

Persistence deserves one more line because it is the feature that makes iteration fast. If a job will reuse an intermediate RDD several times, a program can cache it — marking it so that after the first computation, its partitions stay in memory instead of being recomputed from the lineage every time. Recomputing a ten-step lineage for each reuse would be wasteful; caching turns the first computation into the only one. The same choice drives interactive queries: cache the loaded dataset once, and every query runs against memory.

7.4.7 The Limitations of RDDs

For structured data, RDDs do not exploit any optimizer, and in that case it is better to use a DataFrame or a Dataset. RDDs were typically for unstructured data, where you do not put a structure on it; with a DataFrame you try to put a structure on it; with a Dataset you keep type checking at compile time, which is why it is the safer thing. RDDs are in-memory JVM objects, they are garbage collected, and they carry serialization and deserialization overheads. Also, since data that cannot fit in memory is pushed onto the disk and that slows performance, machines need to have enough memory given the data size and the kind of analysis we are going to do. These are the limitations, but in comparison to Hadoop there are advantages as well.

The first limitation is a design trade-off, not a bug. RDDs deliberately skip schema and query optimization, so for tidy, structured data they leave speed on the table — the engine cannot reorder and prune work it knows nothing about. The second limitation is physical: an RDD's objects live in the JVM heap, so they are created, garbage collected, and serialized, and all of that costs CPU. The third limitation is the one from section 7.3.6 in reverse: the whole design assumes data fits in memory, so clusters must be sized to the data, or performance pays for the spill.

7.5 RDD versus Distributed Shared Memory

In an earlier discussion we saw parallel systems where memory is shared across all the parallel systems. The RDD architecture looks very similar — all the systems are connected across a common memory — but actually it is not the same system. In distributed shared memory (DSM), the address space is shared by multiple nodes. RDDs also share the same memory system across nodes. The differences show up in five dimensions.

The resemblance is real: in both designs, every node can reach data held by other nodes, and both pool the machines' memories into one logical space. But the family resemblance hides a fundamental difference in how operations are allowed to touch that shared memory. Going through the five dimensions one by one makes the split obvious.

7.5.1 Grain of Read and Write Operations

RDD operations are coarse-grained, as they work at the data set level; you cannot access the individual variables, so Spark systems are not meant for fine-grained access. In distributed shared memory, you can access the data items — a specific data item. The grain of read and write operations is coarse for RDDs and fine for DSM.

This is the deepest difference, and the others mostly follow from it. A DSM program reads and writes a single memory cell — "increment that counter," "update that entry" — exactly as if it were a local program touching a shared variable. An RDD program can only say "apply this function to the whole dataset," producing a new whole dataset. You can update every record that matches a rule, but you cannot reach in and edit record number 42 on its own. If your problem needs fine-grained point updates — a live shared ledger, a chat room's presence map — an RDD is the wrong shape of tool.

7.5.2 Consistency

RDDs are immutable, so they are fairly consistent by construction. In DSM, it is the responsibility of the programmer to follow a certain set of rules to make sure consistency is there: there is a mechanism of locks, and critical sections — these are all operating system concepts.

Consistency asks: "when several nodes read and write shared data, can any node see a half-finished or contradictory state?" The RDD answers with its immutability: shared data never changes, so there is nothing to contradict — consistency is guaranteed by the data structure itself, not by careful programming. DSM hands the problem to the programmer: you must wrap shared regions in locks, guard them with critical sections, and get the order right, or two nodes will trample each other. Locking works, but it is exactly the kind of error-prone manual discipline that distributed systems try to remove.

7.5.3 Fault Recovery

New RDDs are created on each transformation, and the lineage of operations taken while the RDD was being manipulated is stored. Using that lineage of operations, RDDs can be recovered after a fault. Distributed shared memory needs a certain form of checkpointing or rollback mechanisms to recover from faults.

When a node dies, the two systems lose different things. An RDD node loses only computed results, and those results have recorded recipes — the lineage replays the operations, and the partition is rebuilt from the raw data and functions that are still safe elsewhere. A DSM node that dies may have held the only current copy of some shared state; there is no recipe to replay, so the system must have periodically snapshotted the memory (checkpointing) or must undo recent operations (rollback) — both expensive, and both leave a window where the lost writes are simply gone.

7.5.4 Staggler Mitigation

A staggler is the slowest performing node in the system. If you have heterogeneous nodes, the slowest performing node slows down the performance of the whole system, so it becomes a staggler. The problem is that slow tasks slow down end-to-end performance, and we want to mitigate that. RDDs make this easier with backup tasks. In DSM it is very difficult, which is why in distributed shared memory we always want the nodes to be homogeneous — all the same hardware and software configurations — while the distributed systems we deal with, like the Hadoop systems, allow heterogeneous nodes.

A job is only as fast as its slowest task: the engine waits for every partition before reporting the final answer, so one sluggish node drags everyone else down. Spark's lineage gives it a cheap fix — backup tasks. When a task runs suspiciously slow, the engine starts a duplicate of that task on a healthy node and uses whichever finishes first. This works because tasks are independent recomputations of the same partition. DSM cannot do that: slow shared-memory nodes are entangled in every other node's operations, and you cannot just "re-run" a task in a world of mutable shared state. The DSM world's workaround is prevention — buy identical machines — which is why DSM clusters favor homogeneous hardware, while Hadoop-style clusters tolerate mixed hardware.

(Note: the standard spelling of the term is straggler; the lecture uses staggler. Both refer to the same phenomenon — the slowest node throttling the whole job.)

7.5.5 Out-of-Memory Behavior

When RDD data runs out of memory, the spillover goes to the disk, and performance degrades gradually. In DSM, an out-of-memory situation starts swapping: heavy page-in and page-out activity begins, the system hangs, and performance goes down significantly. That is the difference between an RDD and distributed shared memory.

The two systems do not merely slow down differently — they fail differently. Spark spills the excess to disk and keeps going, trading speed for survival: you can watch performance degrade gracefully as more data spills. A DSM system starts swapping pages between RAM and disk, and as thrashing sets in the whole machine effectively hangs; performance collapses instead of degrading. For a production system, the difference matters: gradual slowdown can be monitored and mitigated; a hang is an outage.

7.6 The Spark Unified Stack

Spark is built as a unified stack. At the bottom sit the schedulers; in the middle is Spark Core; on top sit the four processing supports: Spark SQL for structured data, Spark Streaming for real-time data, machine learning, and graph processing.

Picture the stack as four stacked shelves. The bottom shelf holds the cluster managers — the schedulers that decide which application gets which machine resources. The shelf above holds Spark Core, the common engine and RDD machinery every workload shares. The top shelf holds the four specialized libraries — Spark SQL, Spark Streaming, machine learning (MLlib), and graph processing (GraphX) — which all talk to the same core. The point of the design is that everything above the bottom shelf speaks one programming model: once you can use an RDD, you can use every library.

┌─────────────────────────────────────────────┐
│  Spark SQL │ Spark Streaming │ MLlib │ GraphX│   ← four processing supports
├─────────────────────────────────────────────┤
│                  Spark Core                 │   ← scheduling, memory, RDD API
├─────────────────────────────────────────────┤
│   YARN │ Mesos │ Standalone │ Kubernetes     │   ← cluster managers (bottom)
└─────────────────────────────────────────────┘

7.6.1 Cluster Managers at the Bottom

It is not necessary to use an in-house cluster manager; a variety exist. Hadoop YARN based cluster managers can be used, Apache Mesos can be used, and there is a simple cluster manager included in Spark itself called the standalone scheduler. Kubernetes is also an option. If you are installing Spark on an empty set of machines, the standalone scheduler is easy to get started with. If you already have a Hadoop YARN or Mesos cluster, Spark supports these cluster managers and allows your application to run on them as well. This is all about decoupling: if there is a tight coupling, we cannot do this; if the resource management layer is separate from the compute layer and the storage layer, these things become possible. From Hadoop 2.0 onwards these things started, which is why YARN was made a separate layer — and that is why, using Hadoop YARN for cluster management, processing can be done on Spark.

The cluster manager is the landlord of the machines: it hands out CPU and memory to competing applications and reclaims them when applications finish. Spark deliberately has no opinion about which landlord you use, because the compute layer (Spark) is decoupled from the resource layer (the manager) and the storage layer (HDFS and friends). That decoupling is the same design move made in Hadoop 2.0, when YARN was split out as its own layer — and it is exactly why Spark can sit on top of an existing Hadoop cluster instead of demanding its own.

7.6.2 Spark Core

Spark Core contains the basic functionalities of Spark, including the components for task scheduling, memory management, fault recovery, and interacting with the storage systems. It is also the home of the API that defines RDDs — Spark's main programming abstraction — with many APIs for building and manipulating these collections.

Everything the RDD section described lives here: the driver program that holds the SparkContext, the scheduler that splits work into tasks, the memory manager that decides what stays in RAM and what spills, the lineage machinery that rebuilds lost partitions, and the connectors that read from and write to HDFS, HBase, and other storage. When a Spark SQL query or an MLlib model is running, Spark Core is the engine under the hood.

7.6.3 Spark SQL and HiveQL

Spark SQL is the package for working with structured data. It allows querying the data via an SQL interface, and a Hive variant is also there, termed HiveQL — the Hive Query Language. It also allows developers to intermix SQL queries with programmatic data manipulation supported by RDDs in Python, Java, and Scala, all within a single application. That tight integration with the rich computing environment provided by Spark is what makes Spark SQL unlike any other open source data warehouse.

The headline feature is mixing: one application can run SELECT ... FROM sales WHERE ... and then take the result and process it with ordinary Python or Scala code over RDDs — no separate query engine, no data handover. Hive users get a familiar entry point too, since HiveQL queries run in Spark SQL. Remember the earlier warning about Hive on MapReduce being slowed by disk at every step; Spark SQL gives the same SQL experience without the disk-bound MapReduce underneath.

7.6.4 Spark Streaming

Spark Streaming enables the processing of live streams of data. It integrates with various streaming tools such as Kafka, Flume, and Kinesis. Streams like tweets coming from Twitter can be analyzed on the go. It provides an API for manipulating the data stream that closely matches the Spark Core RDD API, and it is designed to provide a degree of fault tolerance, throughput, and scalability. Data can come from Kafka, Flume, Kinesis, Twitter, HDFS, or S3; the results can go to a dashboard or be saved onto databases or HDFS storage.

The design trick is that streaming reuses the RDD machinery instead of inventing a second engine: the incoming stream is chopped into small batches, each batch becomes a distributed collection of RDDs (the DStream), and each batch is processed with the exact RDD operations already covered. Because the batches are computed through the same lineage machinery, stream processing inherits the fault tolerance of batch processing — a lost batch is recomputed like a lost partition. This is the fraud-detection engine from section 7.1.2 made concrete: Kafka or Kinesis delivers transactions as they arrive, Spark Streaming analyzes each batch, and alerts flow to a dashboard or a database while the moment is still live.

7.6.5 MLlib: Machine Learning at Scale

Spark has strong support for building machine learning applications. MLlib is a built-in library containing common machine learning functionality. It provides multiple types of machine learning algorithms, including classification, regression, clustering, and collaborative filtering, plus supporting functionality such as data import and model evaluation. Later we will see how to write a machine learning application using Spark. All of these methods are designed to scale out across a cluster.

MLlib is the machine learning shelf of the stack: classification (predict a category), regression (predict a number), clustering (discover groups without labels), and collaborative filtering (the technique behind "users like you also bought…"). Around the algorithms sit the supporting pieces a real project needs — loading data, splitting into training and test sets, and evaluating how good a model is. The scaling promise is the same one the whole stack makes: an algorithm written once runs on one machine or a hundred, because every method is built on distributed RDD-style computation.

7.6.6 GraphX: Graph Processing

GraphX is Spark's library for manipulating graphs — specifically social network relationships — with graph-parallel computation. It extends the Spark RDD API and allows creating directed graphs with arbitrary properties attached to each vertex and edge; we saw the Neo4j style of thinking before. It provides various operators for manipulating graphs and a library of common graph algorithms such as PageRank and triangle counting.

GraphX brings the graph-processing pattern from section 7.1.4 into the unified stack. A graph has vertices (people, pages, accounts) and directed edges (follows, links, transfers), and both can carry arbitrary data — a person's age on the vertex, the time of a transfer on the edge. The library provides graph operators plus ready-made algorithms: PageRank scores vertices by how much traffic flows through them (the ranking idea behind web search), and triangle counting counts how many triplets of vertices are all mutually connected — a measure of how tightly knit a social neighborhood is.

7.7 Where Spark Fits: Use Cases and Limits

7.7.1 Spark Across Domains

Spark has use cases across domains. In banking: customer segmentation, credit risk assessment, and targeting advertisements on products. In the e-commerce and retail domain: clustering on streaming data for identifying trends, and recommendations. In travel: personalized travel recommendations. Media and entertainment, healthcare, and IoT — analyzing the sensor data on the edge — all fit.

Read the list as two recurring patterns rather than a menu. The first pattern is learn from stored history: banking clusters customers into segments and scores credit risk from past records; retail and travel recommend products and trips from purchase histories. The second pattern is react to the live stream: retail clusters streaming click data to spot trends while they form, and IoT engines analyze sensor readings at the edge as they arrive. Both patterns are the same Spark — batch-style model building plus streaming-style scoring — combined in one platform.

Real-world: Fraud detection again shows the pattern: streaming data in, immediate analytics out, at the scale of a bank's transaction flow. The bank's historical data trains a fraud model; the streaming engine applies that model to every new transaction in real time; and a flagged transaction is stopped before the money moves.

7.7.2 When Not to Use Spark

Avoid Spark in two situations. First, large batch processes with high memory requirements. Since Spark is in-memory computing, a huge amount of data that cannot sit in the memory must spill over to the disk, and the performance goes down; it is better to go ahead with Hadoop batch processing, which reads and writes from the disk by design. Second, multi-user analysis environments where the concurrent demand for memory is high. With many users reading data from and writing data to the memory, demand for a particular kind of data becomes very high, it becomes difficult to manage, and the system may not scale with the number of concurrent users. If that is the case, we probably should not use Spark.

The first situation is the memory budget from section 7.3.6 in its extreme form. Spark's advantage is holding the working set in RAM; if the working set cannot fit, spillover erodes the advantage until the job is basically disk-bound anyway — in which case Hadoop's honest disk-based design is just as fast and far simpler to reason about. The second situation is contention: Spark's speed assumes each application can grab memory when it wants it. Under many concurrent users all caching different datasets, memory becomes a scarce, contested resource, and the engine spends its energy shuffling data in and out instead of computing.

Exam Guidance Summary

A few questions from the Dynamo paper portion may be picked up once the syllabus is complete, so keep the material on the Dynamo paper's requirements and the technologies that meet them ready. The remaining RDD material — DataFrames and Datasets, and the differences between RDD, Dataset, and DataFrame — will be covered later in the course.

For revision, the lecture's own structure is a good checklist:

• The four kinds of processing (batch, stream, interactive, graph) — what defines each, and the timing that separates them.
• Why MapReduce is slow: disk traffic at every stage, replication factor three, serialization and deserialization overhead, and how the in-memory architecture (with spillover) addresses each.
• The RDD: what the three words mean (resilient — lineage; distributed — partitions; dataset — collection), creation, laziness, immutability, persistence, coarse-grained operations, and the RDD-vs-DSM five dimensions.
• The unified stack: cluster managers, Spark Core, Spark SQL/HiveQL, Spark Streaming, MLlib, GraphX.

Keep the earlier Dynamo material ready as advised; the DataFrames/Datasets comparison arrives in a later lecture.

Key Industry Applications

• Fraud detection — the stream processing example: analyze each transaction as it arrives and flag fraudulent ones in real time.
• Dynamo — Amazon's paper on storage requirements; a few questions from that portion may be picked up later.
• Neo4j and Cypher — the graph database and query language used for relationship-heavy data.
• Impala and Presto — SQL-on-Hadoop engines for interactive processing of queries.
• Kafka, Flume, Kinesis, Twitter, HDFS, S3 — streaming and storage sources integrated with Spark Streaming.
• DataBricks — the organization that contributed the RDD and its ecosystem.
• MLlib — classification, regression, clustering, and collaborative filtering at cluster scale.
• GraphX — PageRank and triangle counting over social network graphs.
• Cluster managers — YARN, Mesos, the standalone scheduler, and Kubernetes.
• HiveQL — the Hive query language variant supported by Spark SQL.