Streaming Data Architecture and Spark Streaming

9.1 Streaming Data Architecture: The Four Layers

9.1.1 Components of a Streaming Architecture

A streaming data architecture ends up with the following components:

• The collection layer collects the data from the sources — the web servers, sensors, and machines that generate events.
• The data flow layer sits between the collection layer and your processing. It is an intermediate layer, and as we will see, its job is to absorb the differences in speed between the fast data sources and the slower processing stage.
• The storage layer holds the results after processing. You may keep them in memory (main memory, for fast access), persist them on a hard drive for later use, or hand them to the delivery layer.
• The delivery layer passes the results onward — to downstream layers, or to the end user who is looking for the results or the analysis.

A useful mental picture: collection is the "ears" of the system, data flow is the "throat" that regulates how much can pass, processing is the "brain", storage is the "memory", and delivery is the "voice" that reports back. Each of these four roles can be scaled out — more machines, more capacity — without redesigning the others.

9.1.2 The High-Level Flow and the Feedback Loop

Here is the high-level generalized architecture, in the order data moves through it:

1. The collection tier collects data from web browsers, sensors, and automated machines — whatever the sources are. That is its only job: collect the data.
2. Data from the collection tier is forwarded to the analysis tier, where it will be processed. It has to go through the data flow layer on the way.
3. After the analysis, you store the data in memory — memory means main memory — or you persist it in persistent storage.
4. From storage, you push the data to the delivery tier.
5. Finally, the results are displayed on web interfaces, mobile interfaces, and the dashboards on which we are tracking metrics.

One more thing may happen: from the delivery tier you send something back to the analysis tier. Why would you do that? Because it may be an iterative application — an application that refines its answer over several rounds. Some results feed back into the analysis tier, the analysis tier runs again, returns to memory, and the loop continues until the computation converges on a final answer. Clustering is the classic example: assign points to clusters, update the cluster centers, reassign, and repeat until the centers stop moving. Without this feedback loop, such iterative algorithms could never run on streaming data.

Visual intuition: Picture the architecture as a horizontal flow diagram. On the left, draw a fan of source arrows (web browsers, sensors, machines) entering a box labelled Collection. A single thick arrow leaves Collection and enters the Data Flow box — a narrow funnel shape, because the funnel's job is to smooth the flow. From the funnel, an arrow enters the Analysis box (a cluster of worker circles). Analysis connects down to a Memory/Storage box (a cylinder, the database symbol). From storage, an arrow rises to the Delivery box, which fans out again into dashboards, mobile screens, and alerts. Finally, a curved arrow returns from Delivery back to Analysis, closing the feedback loop. The one-sentence takeaway: data flows left to right in one direction, with a single feedback arc reserved for iterative algorithms.

9.1.3 The Collection Layer

The collection tier collects the data from your sources. Those sources could be web browsers, automated machines, or sensors. How does the collection happen? The communication between the collection tier and the sources runs over a TCP/IP network using HTTP. Sensors can post data to an HTTP service periodically — each post carries the latest readings.

The original use case for this tier was website log analysis. The idea: build a strong log analysis system where you want to detect certain errors or deviations in what users are doing. The initial format in which logs were collected was the W3C standard log data format — a fixed, text-based format for web server logs. Nowadays newer formats are available, and the collection tier supports them too: JSON, Avro, and Thrift are all common today. Avro and Thrift are interesting because they are compact binary formats with a schema — the format tells you what each field means, which makes later processing much easier.

Collection happens at specialized servers called edge servers — machines placed close to the data sources. The collection process is usually application-specific: there are specific adapters that talk to your data sensors or to the source machines where the data is coming from. One adapter might speak the protocol of a temperature sensor; another might poll a database; a third might tail a web server's log file. The edge server runs the right adapter for each source.

Real-world: It is quite common today to have an IoT system with edge processing: the edge server collects data from IoT devices and pushes the data deeper into the back end, into the cloud system. The collection tier also integrates with modern data flow systems directly. Older servers may or may not integrate directly with those data flow systems — for them, the edge server remains the bridge.

9.1.4 The Data Flow Layer

The data flow tier is a separation between the collection tier and the processing layer. Why is this required? Because the two systems may work at different rates: the source may generate data very fast, and the processing tier may not be able to process at that speed.

So the responsibilities of the intermediate layer are:

• Accept the messages and events from the collection layer.
• Provide those messages and events to the processing layer.
• Act as a real-time interface to both sides — to the producer (the data source) and to the consumer (your processing tier).
• Guarantee at-least-once semantics: we do not want to lose any data, so every event must reach processing at least once.

Why is the buffer needed even when nothing is broken? Because of spikes. Data sources at times spike — if you are analyzing real-time streaming data on an e-commerce website, big sale days cause the event rate to explode. Without a buffer, you would lose data during those spikes: you want real-time streaming, but your processing tier is not fast enough to cope with the burst. So you put an intermediary layer that holds the messages and guarantees at-least-once semantics.

A note on delivery semantics — there are several you can read about:

• At-least-once: every event is delivered one or more times; nothing is lost, but duplicates are possible.
• At-most-once: every event is delivered at most once; nothing is duplicated, but events can be dropped.
• Exactly-once: every event is delivered exactly once — neither lost nor duplicated. This is the hardest to guarantee and the most expensive.

Spark supports configurable delivery semantics; you will see receivers and acknowledgements again in Section 9.7, where these concepts return in concrete form.

Real-world: This buffering role is exactly what message-queue systems like Kafka play in industry — an intermediary between producers and consumers that absorbs rate mismatches and spikes so no events are lost. Kafka is a distributed, partitioned, replicated commit log: producers append events to it, consumers read from it at their own pace, and the log itself survives machine failures.

9.1.5 The Processing Layer

Third comes the processing — the analytical tier. You are analyzing something, and the processing tier does that. These distributed processing layers are built on the concept of locality of reference.

What do we need in practice? There is a cluster. We must:

1. Partition the data so that parallel processing is possible — split the dataset into chunks, one chunk per node (or per partition).
2. Schedule the job — decide which node runs which task, and in what order.
3. Manage the jobs with some coordinators — a resource manager that hands out memory and CPU.

You have seen all of this in Hadoop — this is exactly the part YARN does. The good news: the framework takes care of almost everything. The heavy lifting of data partitioning, job scheduling, and managing certain jobs is done for you; you write the transformations and the framework worries about where they run.

Which streaming framework do you use? Several are available:

• Apache Storm — a classic continuous-operator stream processor (you will meet that model in Section 9.2).
• Apache Spark Streaming — with Spark you do batch processing and you do stream processing also, on the same engine.
• Apache Kafka Streaming — good if you only want to do certain lightweight computations, directly inside the message system.

A practical split: heavy processing is always done on Spark; certain lightweight computations on streaming data can be done with Kafka Streaming. So the processing layer is typically the cluster that exists, takes the data, partitions it onto a number of nodes, carries out certain transformations, carries out certain actions, and then passes the data into memory.

9.1.6 The Storage Layer

Then comes the storage tier. It keeps the processed data in memory — or you can persist the data permanently, that is also possible. Usually the data stays in memory, because the data is processed once: there is no need to write it to disk if no one will read it again.

But there are use cases where events or outcomes need to be persisted as well. For example, you process something now so that later, some historical analysis can be carried out on the accumulated results. That is the case where you persist the data on the storage tier.

For permanent data, NoSQL data stores are becoming the more popular choice — MongoDB, Cassandra, and the other NoSQL data stores we have covered in detail elsewhere. There is a decision to make: look at your use case and choose an appropriate NoSQL database. Ask what the use case wants:

• Consistency — do reads need to see the latest writes immediately? (MongoDB is strong in document stores with tunable consistency; HBase offers strong consistency.)
• Wide-columnar data — large, sparse tables with many attributes per row? (Cassandra and HBase are the wide-column stores.)
• JSON documents — semi-structured records that vary from row to row? (MongoDB stores BSON documents directly.)

There is not a single database that fits all use cases. The relational database management system (RDBMS), which follows ACID semantics, is rarely used here — its strict transactional guarantees are a poor fit for high-volume, append-only streaming outcomes.

Real-world: In industry, the permanent storage for streaming outcomes is typically a NoSQL store chosen per use case — consistency, wide-column, or document-oriented — rather than an ACID RDBMS. This matches what you have seen in the NoSQL lectures: pick the store by the shape of the data and the guarantees you need.

9.1.7 The Delivery Layer

The next component is the delivery layer. Data is processed and stored in memory; from memory you go to the delivery layer. It is usually a web-based interface, and nowadays mobile interfaces are also becoming popular. There are dashboards built with streaming visualization — they get continuously updated as the underlying events are processed; the dashboards update automatically. You can use standard web technologies to create these interfaces and keep them updated: HTML, CSS, and JavaScript, with WebSockets for pushing updates from the server to the browser without the browser asking. HTML5 is quite popular for building the interfaces. You can also export the outcomes as PDF or SVG formats — that is also a possibility, for reports that must be shared outside the dashboard.

And it is not only dashboard metrics being maintained. There are other use cases where the delivery tier sends alerts: when a certain thing happens, an alert is sent. Say you have log streaming coming in and you are looking for an error keyword; the delivery tier can send an alert whenever that error is recognized. The delivery tier also feeds the data to downstream applications — a fraud-checking service, a recommendation engine, a billing system. So the delivery tier is not just about maintaining dashboards.

Real-world: Real-time dashboards with WebSockets and HTML5, alerting on error keywords in streaming logs, and PDF or SVG export of outcomes are standard delivery-tier patterns in production streaming systems.

9.1.8 Where This Architecture Lives in Practice

Putting the four layers together end to end: a click stream from an e-commerce site is collected at edge servers (collection), routed through Kafka where bursts are buffered (data flow), processed in Spark Streaming which computes, say, per-product page-view counts (processing), the counts persist into Cassandra for later analysis (storage), and live charts on the operations dashboard update within seconds while a rule engine raises an alert if error rates exceed a threshold (delivery).

9.2 The Continuous Operator Model

9.2.1 How Traditional Streaming Processing Was Realized

We have seen the components of a streaming system. Now: how was streaming processing actually realized? It was realized as a continuous operator model — the traditional way to process a stream, and the model against which Spark Streaming later positions itself.

In the traditional streaming processing diagram, there is a source operator which receives the streaming data from the data sources — live logs, telemetry data, IoT devices. You can use some Kafka-style data injection to bring the data in — Kafka sits in front and feeds the stream in. Data is then processed in parallel across the cluster:

1. The source operator sprays the records over the cluster — it distributes each incoming record to one of the worker nodes.
2. The continuous operators process the data — you take the data once, apply map and reduce transformations, and generate the output. This is done on a bunch of records, one record at a time.
3. A sync operator collects the processed records.
4. Finally, the results are given to the downstream systems — HBase or Cassandra are typical examples.

Real-world: Kafka is a typical injection front-end for this model, and HBase or Cassandra are the typical downstream stores that receive the processed results.

9.2.2 Fine-Grained Record-Level Processing

Look at the granularity: one dot in the diagram is one record, so the model works at a very, very fine level — the granularity is at the record level. This is the defining property of the continuous operator model, and it is worth pausing on.

The cluster has a set of worker nodes, each of which runs one or more continuous operators. Each continuous operator processes the streaming data one record at a time — this is the fine-grained control — and forwards the records to the other operators in the pipeline. If one transformation is being done here, maybe some other continuous operators after this apply further transformations; the operators are set up as a pipeline. Finally, the sync operator collects from the pipeline and sends the data to downstream applications like HBase, Cassandra, and so on.

So the kind of system used is: you get the data from the data sources through the source operators; the source operator sprays the data across the cluster record by record; continuous operators do a certain sort of processing — it may be a pipeline of continuous operators; then the sync operator sends the data to the downstream applications.

Visual intuition: Picture the diagram as a conveyor belt running around a cluster. At the head, a spray nozzle (the source operator) sits above the belt; each dot that lands is a single record. The belt passes under a row of worker stations — the continuous operators — and at each station a worker picks up one dot, applies one transformation, and places the transformed dot back on the belt. At the end of the belt, a collection hopper (the sync operator) gathers the finished dots and pours them into downstream stores. The takeaway: every record is handled individually, immediately, as it passes — nothing is grouped, nothing waits.

9.2.3 Why We Care About This Model Today

The continuous operator model is not a museum piece — it is the model behind real systems like Apache Storm, and the model that Spark's designers explicitly contrasted with Spark Streaming's batch-based approach. Knowing exactly how it works is what makes the challenges of Section 9.3 and Spark's answers in Sections 9.4–9.6 intelligible: every advantage Spark claims (fast recovery, load balancing, unification) is defined relative to the record-at-a-time behavior you have just seen.

9.3 Challenges with the Continuous Operator Model

These systems face three challenges — and the third one is the most important for everything that follows.

9.3.1 Fast Failure and Straggler Recovery

The first thing we want from a streaming system is fast failure and straggler recovery. If there is a failure, we want to recover from it as fast as possible.

Why is fast recovery difficult to achieve in a continuous operator model? Because of the static allocation of continuous operators to the worker nodes. The continuous operators are given to the worker nodes to operate, and that allocation is static — it does not change and does not adapt to dynamic workloads. That is one challenge with the continuous operators.

Think about what a failure means here. In the continuous operator model the stream is live and transient: a node dies, and the records that node was holding at that instant are simply gone — there is no on-disk intermediate result to recompute from, because records were processed and forwarded as they arrived. And you cannot just wait out a straggler, because a straggler delays every record that must pass through it. The system must detect the slow or failed node and shift its work automatically, or the entire pipeline stalls.

9.3.2 Load Balancing

Load balancing is also a challenge. There is no dynamic adaptability of the resource allocation based on the workload. The operators are pinned to specific workers when the topology is deployed; the system does not re-examine the workload later and move operators around.

The consequence: an uneven allocation of the processing load between the workers can cause bottlenecks. If one worker's operator happens to sit at the hot spot of the stream — say it receives records for the most popular key in a reduce — that worker saturates while the others sit idle. In a system with static allocation, nothing redistributes the load; the hot worker simply becomes a straggler. This is the load-balancing problem in its purest form: the workload is dynamic, but the allocation is frozen at deployment time.

9.3.3 Unifying Streaming, Batch, and Interactive Workloads

The third point — very, very important — is the unification of streaming, batch, and interactive workloads. The kind of system we have seen works at a very fine-grained, record-by-record level, and these systems might be good for querying the streaming data as it flows. But what about interactive query processing, or batch processing over static datasets? These systems might pose a challenge.

The system might have requirements to:

• Query the streaming data interactively — ask a question and get an answer quickly, not as a pre-wired pipeline.
• Combine the stream with static datasets — join live sensor readings with the static product catalog stored in a database.
• Run batch processing — heavy analyses over the full history, using the same logic as the streaming query.

Interactive query processing and batch processing are difficult tasks in a continuous operator system which is not designed for ad hoc queries. An operator topology is compiled for a fixed pipeline; an interactive query arrives at runtime and does not fit any of the installed operators. This requires a single engine that can combine batch, streaming, and interactive processing — and that is where we find that Spark does exactly that.

Exam note: The unification point was flagged as very, very important — it is the central motivation for the whole rest of the discussion. Expect to be asked why a single engine that handles streaming, batch, and interactive workloads is hard to build on the continuous operator model.

9.3.4 Advanced Analytics and a Common Abstraction

There are also complex workloads that require more than simple transformations:

• Continuous learning and updating of data models — the model must be retrained or updated as new stream records arrive, not once on a fixed dataset.
• Querying the streaming data with SQL — the stream should behave like a table you can query.

We will see the unbounded table: a stream of records is coming, you put it in a structured format using a DataFrame, and once that table is updated automatically you can query it using SQL. Whether it is SQL or PySpark or something else, we should have a common abstraction across these analytic tasks — one mental model, one API family — that makes the developer's job much easier.

These were the challenges that existed with respect to the streaming systems. The next section shows how Spark answers each of them.

9.4 How Spark Meets the Challenges

We have already started with Spark, and we know there is fast recovery from failures and stragglers. Spark addresses these challenges one by one.

9.4.1 Fast Recovery with RDDs and DAGs

We know RDDs and DAGs — we have studied RDDs and distributed shared memory systems, and we said there is a process to deal with the stragglers. There is a DAG being created out of those transformations, and in case an RDD partition is lost you can always reconstruct it.

Why does that solve the recovery problem? Because of the lineage idea:

• Every RDD remembers the transformations that created it — its lineage.
• The sequence of transformations forms a DAG: each RDD is a node, each transformation is an edge.
• If any partition is lost — a node crashed, a straggler fell behind — Spark walks back up the DAG to the surviving inputs and recomputes only the lost partition.

This is the crucial contrast with the continuous operator model. There, a failed node means lost records; the stream is transient and there is nothing to recompute from. In Spark, a failed partition is not lost data — it is a small recomputation. The output is a function of the DAG, and the DAG still exists. And because recovery happens at the granularity of partitions (chunks of a batch), not individual records, the recomputation is fast: only the affected chunks are rebuilt, in parallel across the surviving nodes.

9.4.2 Dynamic Load Balancing and Resource Allocation

Second, there is better load balancing and better use of resources, because Spark has dynamic resource allocation and dynamic load balancing.

In the continuous operator model the operators were pinned to nodes forever. In Spark, tasks are short-lived units of work — one task processes one partition — and the scheduler decides where each task runs. There is no permanent binding between a chunk of data and a machine. If one node becomes loaded, new tasks can be placed elsewhere; if a node idles, work can flow to it. The scheduler watches the actual resource utilization across the cluster and redistributes work accordingly. That is the dynamic load balancing.

There is a second dimension of dynamism: data. Data from the streaming sources can also be combined with a very large range of static data sources available through Spark SQL. So the streaming pipeline is not sealed off from the rest of the data estate — the live stream and the historical tables live under one roof and can be joined in the same job.

9.4.3 One Engine for Every Workload

Third, Spark provides a unification of the different data processing capabilities. Whether you want:

• streaming data,
• streaming with interactive data,
• streaming as well as batch processing,
• or interactive and batch alone —

everything runs on one engine. It is a unified engine that natively supports both batch and streaming workloads, which makes it very easy for developers to use a single framework to satisfy all the processing needs. Whether it is batch, interactive, or streaming, you can use Spark.

9.4.4 Native Libraries for Machine Learning, Graphs, and SQL

Spark also has native integration with advanced libraries, as we have seen:

• Machine learning — you can build and run ML models with Spark's MLlib, right on the streaming or batch data.
• Graph processing — you can integrate graph processing with Spark (GraphX): PageRank, connected components, community detection on large graphs.
• SQL — you want to run SQL queries on top of Spark; that is also possible with Spark SQL.

So Spark gives you a unified engine to process different kinds of workloads, which was difficult to achieve with the continuous operator models. The libraries are not add-ons bolted on the side — they operate directly on the same RDD/DataFrame abstraction the streaming and batch pipelines use, so a model trained on historical data can score live stream records without leaving the engine.

Real-world: This is why Spark became the default platform for many analytics teams: one skill set (Spark) covers the nightly batch warehouse job, the real-time fraud-scoring stream, the ad hoc SQL exploration, and the ML model training — and the team maintains one cluster instead of three.

9.5 Spark Streaming and Discretized Streams

9.5.1 What Spark Streaming Is

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of the live data streams. It is the streaming face of the unified engine from Section 9.4 — the same core, the same fault-tolerance machinery, the same programming model, but tuned for data that keeps arriving.

Data can be ingested from many sources: data can come from Kafka, from Flume, you can have data in an S3 bucket or HDFS, a Kinesis stream, or a Twitter stream being created — data can come from any of these sources. Note that HDFS and S3 are largely about batch processing: if your "stream" is actually a directory of files being written, you can still stream over it, but the file-based sources are naturally batch-flavored. The genuinely live sources — Kafka, Flume, Kinesis, Twitter — are where streaming shines.

The data can be processed using complex algorithms expressed with very high-level functions like map, reduce, join, and window. We typically use the map transformation, reduceByKey, and count — count is an action. There is a certain set of transformations we have seen in batch: filter, and all those things. The key point: we just need to know the function. You call map, that is all — you do not know how it is implemented. You call filter and pass a condition saying on what basis we should filter — that is all you are doing. The window operation we will see with the help of an example today (Section 9.9).

The processed data can be pushed out to file systems, databases, and live dashboards:

• You want to store the data in a file system — good for that.
• You want to store it in a particular NoSQL data store — that is also possible.
• You want something sent to the dashboard so that real-time metric tracking is done — also possible.

And you can apply Spark's machine learning and graph processing algorithms on the data streams as well. This is what Spark offers: the streaming layer reuses everything the rest of Spark offers.

9.5.2 Discretized Streams: From Records to Batches

These streams are termed discretized streams — DStreams — where the live input data streams are divided into batches.

In the diagram there is a receiver; the records are being fed. Now, instead of records being fed to the cluster one at a time, what we are preparing is a batch of RDDs, and we are sending that to the cluster. It is not record-level: each RDD may contain multiple records as well. So we batch a certain number of records and send them. These batches are then processed to generate the final stream of results in batches.

A DStream is a high-level abstraction which represents a continuous stream of data. It can be created either from the input streams from the sources — Flume, Kinesis, and so on — or you can apply high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs. So a DStream is not a new data structure — it is the familiar RDD, repeated in time:

DStream = [ RDD_0, RDD_1, RDD_2, ... ]        (one RDD per batch interval)

So if we compare with the continuous operator model, which was very fine-grained, record-by-record processing, this is a little coarse-grained processing — processing a batch of RDDs. We prepare certain batches and send those batches across the place. Records are processed in batches with short tasks; each batch is an RDD — a partitioned dataset.

Why is batching good for the challenges of Section 9.3? Because each batch is a regular, finite, partitioned dataset: the DAG scheduler can plan it, the task scheduler can balance it, and if a task fails it can be relaunched or recomputed. You will see these benefits in detail in Section 9.6.

Real-world: The ingestion front-ends here are exactly the systems named above — Kafka, Flume, Kinesis, S3, HDFS, and Twitter streams — the standard set of producers for streaming analytics in production. In practice, Kafka is by far the most common of these: Spark Streaming's Kafka connector consumes from Kafka topics, and the same Kafka cluster frequently doubles as the data flow layer from Section 9.1.

9.6 The Benefits of Batching

Batching provides dynamic scheduling of tasks, faster failure recovery, and easier interoperability of batch, stream, and interactive analysis. Each of these answers one of the challenges from Section 9.3.

9.6.1 Dynamic Scheduling Instead of Static Allocation

In the traditional system there is static scheduling of the continuous operators to the nodes, which causes bottlenecks. Consider what happens with an uneven partition of the stream: you are sending certain streams here, certain streams there, but more load lands on one node. Because of the static resource allocation, whatever the capacity is, you keep sending more load to that node — so that node may become a bottleneck (a straggler, in the vocabulary of Section 9.3).

When it comes to Spark Streaming, more load on a partition means longer tasks. But the batch model makes the schedule dynamic. The diagram shows dynamic scheduling of the tasks: say the orange node is getting highly loaded. The scheduler dynamically looks at the resource usage rate and the resource allocation, and it shifts some of the tasks to the other nodes, based on the available resources.

Why is this impossible in the continuous operator model? Because there the "tasks" are long-lived operators wired to specific nodes at deployment time. A batch system creates new tasks for every batch, every few seconds — each scheduling round is a fresh chance to balance the load.

9.6.2 Failure Recovery at Coarse Grain

If a node fails, the failed tasks are relaunched on the other nodes. There is also faster recovery by using multiple nodes for recomputation.

This is possible at a coarse-grain level of processing, not at a fine-grain level. The reason:

• In the record-level model, a failure at a node destroys the in-flight records held by that node — transient data with no backup.
• In the batch model, the failed node held partitions of an RDD. Those partitions are recoverable: their lineage (the DAG of transformations that produced them) is still intact, and their inputs still exist on other nodes.

So recovery is a matter of relaunching the failed tasks and recomputing the lost partitions — and because the recomputation happens in parallel across all surviving nodes (multiple nodes for recomputation), it is fast. These are the benefits of batching that Spark offers.

9.6.3 Interoperability of Batch, Stream, and Interactive Analysis

Batching gives easier interoperability of batch, stream, and interactive analysis. You can run anything, or a combination of these. The DStream — the discretized stream — is a series of RDDs, and every RDD operation you know from batch processing applies to it. Concretely:

• Join a DStream with a static RDD — the live stream can be combined with historical or reference data as if both were ordinary RDDs.
• Convert a DStream's RDDs into a DataFrame — a DataFrame is structured data.
• Query the structured data interactively using SQL — now your live stream is a queryable table (the unbounded table of Section 9.10).

The stream is not a separate universe. Because it is built from the same RDD abstraction as batch, every batch tool — joins, SQL, DataFrames, ML — works on streaming data with no bridge layer. That is the unification promised in Section 9.4, delivered at the data-structure level.

9.6.4 Machine Learning on Streams

You can also apply machine learning library functions on the data stream's RDDs. For example, you can apply a k-means model to label the streaming data: streaming data is coming, you want to label those records, you want to group those records — k-means clustering — you can do that. The MLlib functions accept the stream's RDDs directly, because those RDDs are just RDDs.

Real-world: This is how real-time personalization is built: a model is trained offline on historical data (batch), then loaded and applied to the live stream — scoring each incoming record against the trained model, or updating the model continuously as new labeled records arrive. Fraud detection, recommendation ranking, and IoT anomaly detection all follow this shape: batch-train, stream-score.

9.7 Spark Streaming Sources and Receivers

9.7.1 Sources: Basic and Advanced

Every input DStream is associated with a receiver object, which receives the data from the source and stores it in Spark's memory for processing. The receiver is the connection point: it is the object that actually listens to the source and hands the data to Spark.

There are two families of sources:

• Basic sources — directly available in the StreamingContext API:
• the file system — read as a stream of files (with HDFS and S3 being batch-flavored, as noted in Section 9.5),
• socket connections — a TCP stream of text lines,
• and sources created directly through the StreamingContext API.
• Advanced sources — Kafka, Flume, Kinesis:
• available through extra utility classes (connector libraries),
• these require linking against extra dependencies — you add the connector artifact to your build, and often extra configuration (like the Kafka brokers) at runtime.

The distinction matters practically: basic sources work out of the box, while advanced sources need the matching connector dependency and its configuration.

9.7.2 Reliable and Unreliable Receivers

There are two kinds of receivers:

• A reliable receiver sends an acknowledgement back to the source when the data is received and stored in Spark with replication. The ack tells the source "this record is safely in Spark; you do not need to keep it." If Spark fails before the ack, the source can resend — nothing is lost.
• An unreliable receiver is one that does not send an acknowledgement to the source. It receives the data, but the source has no way to know whether the data arrived and was stored.

9.7.3 How Data Moves from Source to Cluster

So the answer to the student's question: nobody pulls, and nobody pushes per record. The receiver is the active agent. It ingests the stream from the source, organizes the records into batches, materializes each batch as an RDD, and hands the batch to the cluster. The partitioning of the RDD across nodes and the scheduling of tasks are framework responsibilities — the developer never writes that code. (Several students asked a version of this question; the receiver is the single answer for all of them.)

9.7.4 Delivery Semantics

At-least-once semantics, at-most-once semantics, exactly-once semantics — all those things can be maintained. Which one you get depends on the combination of choices:

• the receiver type (reliable or unreliable),
• whether the source supports acknowledgements and replay,
• the output operations and whether they are idempotent.

With a reliable receiver and a replay-capable source you can achieve at-least-once, and with the appropriate output transactionality, effectively exactly-once.

Exam note: the three delivery semantics (at-least-once, at-most-once, exactly-once) are exam-relevant vocabulary. Know the definition of each and which receiver choice supports the no-loss guarantee.

9.8 Transformations and Output Operations on DStreams

9.8.1 Transformations

Spark transformations allow modification of the data from the input stream. Transformations on DStreams mirror the RDD transformations we saw in batch processing. We saw certain transformations on RDDs in batch processing; the similar kind of transformations are possible on an input DStream too: you have map, filter, union, count, reduce, and countByValue — those are possible on the discrete stream.

The key principle: because a DStream is a sequence of RDDs, a DStream transformation is just the corresponding RDD transformation applied to every RDD in the sequence. There is no new machinery — the batch vocabulary carries over unchanged.

A quick reminder of what the familiar ones do:

• map(func) — pass each element of each batch through a function func; returns a new DStream of the results.
• filter(func) — keep only the records on which func returns true.
• union(otherStream) — merge the elements of two DStreams into one.
• count() — count the elements in each RDD of the source DStream; produces a DStream of single-element RDDs.
• reduce(func) — aggregate the elements within each RDD using an associative and commutative function func.
• countByValue() — for a DStream of elements of type K, produce a DStream of \((K, \text{Long})\) pairs where the value of each key is its frequency within each RDD of the source.

There are also the stream-specific building blocks you will see again in later sections: reduceByKey(func) aggregates the values for each key within each batch (Section 9.11 shows exactly why this one costs a shuffle), transform(func) applies an arbitrary RDD-to-RDD function to each RDD of the source DStream — this is the escape hatch that lets you use any batch RDD operation inside a streaming job — and updateStateByKey(func) maintains a state per key across batches, updating the state for each key by applying a function to the previous state and the new values. The window-level transformations (countByWindow, reduceByWindow, reduceByKeyAndWindow, countByValueAndWindow) are covered with the window operations in Section 9.9.

9.8.2 Output Operations

There are also output operations that allow our DStream data to be pushed out to external systems like a database or a file system:

• print() — prints the first ten elements of each batch of data on the driver node running the streaming application; useful for development and debugging.
• saveAsTextFiles(prefix, suffix) — saves the DStream contents as text files; the file name at each batch interval is generated based on prefix and suffix.
• saveAsObjectFiles(prefix, suffix) — saves the DStream contents as SequenceFiles of serialized Java objects.
• saveAsHadoopFiles(prefix, suffix) — saves the DStream contents as Hadoop files.
• foreachRDD(func) — the most generic output operator: it applies a function func to each RDD generated from the stream; that function pushes the data to an external system — for example, writing it over the network to a database. That kind of function you can always write.

Each batch's output is written independently, with the batch timestamp embedded in the file name (the prefix-TIME_IN_MS-suffix pattern), so the outputs of different batches never collide.

So, answering the natural questions: it is not only that the data stays in memory — memory is fast, and we have seen how failures are dealt with. Can we do batch processing with this? We have seen that. Can we do interactive query processing? We have seen that. The certain transformations we saw with the batch processing — can we do the same with a stream of data? That is what we are seeing now. And if we want to save the file on a persistent storage, that is also possible using output operations.

9.9 Window Operations

9.9.1 Windows and Sliding Windows

I talked about the window, so let us look at it. Window operations apply over batches of RDDs: there is an incoming data stream coming; the RDDs are indexed. We make a window of RDDs and want to apply an operation on a window. If three RDDs are there in one window:

\[W_t = \{R_t, R_{t+1}, R_{t+2}\}\]

where \(W_t\) is the window at position \(t\), and \(R_t, R_{t+1}, R_{t+2}\) are the three consecutive RDD batches inside it. The index \(t\) counts batch intervals: \(R_t\) is the batch produced for the time interval \(t\), \(R_{t+1}\) the next one, and so on. The window length is three batches, so a window always spans three consecutive batch intervals.

There is also the concept of a sliding window. This is one size of the window — three RDDs — and now I want to slide it by two. I have included this one and this one, so:

\[W_{t+2} = \{R_{t+2}, R_{t+3}, R_{t+4}\}\]

The slide says how far the window moves each time. Sliding by two from position \(t\) lands at position \(t+2\), and the new window again covers three consecutive batches — but note the overlap: \(R_{t+2}\) is shared between \(W_t\) and \(W_{t+2}\), because the window (three wide) moved only two steps. The window size and the slide are independent choices: window size controls how much history you look at; slide controls how often you recompute.

If you want to analyze certain metrics like a moving average kind of thing, you can always decide how much to move the window — the window size and the slide are configurable. These operations are possible using window operations.

9.9.2 Window-Level Transformations

There are transformations on windows: countByWindow, reduceByWindow, reduceByKeyAndWindow, countByValueAndWindow — all those things which we have seen with the others are also available with these window operations. When you create a batch of RDDs and put them into one window, you can apply an operation at the window level.

The standard window API takes two parameters, expressed in batch intervals:

• windowLength — the duration of the window: how many batches (or seconds) of history each window covers.
• slideInterval — the interval at which the window operation is performed: how often a new window is formed and the operation re-run.

The window-level transformations:

• window(windowLength, slideInterval) — creates a new DStream of windows; the raw material for all the others.
• countByWindow(windowLength, slideInterval) — counts the elements inside each window.
• reduceByWindow(func, windowLength, slideInterval) — reduces the elements within each window using an associative, commutative function func.
• reduceByKeyAndWindow(func, windowLength, slideInterval, numTasks) — the keyed version: reduces the values per key across the whole window; with an inverse function form (reduceByKeyAndWindow(func, invFunc, ...)) it can reuse the previous window's result for efficiency — subtract the expiring batches, add the new ones — instead of recomputing the whole window.
• countByValueAndWindow(windowLength, slideInterval) — the frequency of each distinct value within each window.

The canonical textbook pattern for this is a windowed word count: a socket text stream is split into words, mapped to (word, 1) pairs, and reduced with reduceByKeyAndWindow over a window of 60 seconds sliding every 10 seconds — giving the word frequencies for the last minute, updated every ten seconds.

Visual intuition: Picture a timeline of batch boxes marching left to right, one box per batch interval. A bracket (the window) encloses three consecutive boxes. Every slide step, the bracket jumps right by two boxes and re-closes, always covering exactly three. Where the bracket sits at a given moment is \(W_t\); after the jump it is \(W_{t+2}\). The one-sentence takeaway: the bracket's width is the window length, its jump size is the slide, and both are under your control.

9.10 Structured Streaming: The Unbounded Table

9.10.1 From RDDs to DataFrames to an Unbounded Table

As I mentioned earlier, there is also structured streaming, which works on the Dataset/DataFrame API and turns the stream into a structured form. Let us retrace the progression: we talked about RDDs — an RDD is basically unstructured data: it is a bag of objects, with no schema attached. After that we said that to store structured data you go for the datasets; if you want type checking, you go over the DataFrame. So you build a DataFrame out of the stream and then you say: okay, this is my structured data. You do not have to do batching of that: the stream is continuously coming, and you keep converting it to a DataFrame, which gives it structure, and the data is processed.

The mental model:

1. RDD — unstructured. Rows have no enforced schema; the programmer knows what is inside, the framework does not.
2. Dataset / DataFrame — structured. Columns have names and types; the engine can check types, optimize queries, and run SQL.
3. Structured streaming — take the stream and build a DataFrame from it continuously. Every incoming record is appended to the DataFrame as a new row, in structured format, with no manual batching step.

Once you have given it a structure, it becomes an unbounded table: the stream will come, and you will keep on appending the records to it — you can see the new rows are coming and being added in structured format. So this becomes a kind of an unbounded table — a table that has no fixed end, because new rows keep arriving.

Visual intuition: Picture a spreadsheet that grows on its own. Columns are the fields of your records (say, timestamp, device, metric, value). Rows appear at the bottom continuously, one or several per second, as the stream delivers them. There is no last row until you stop watching. That spreadsheet is the unbounded table; every new event simply appends a new row in the defined column format.

9.10.2 SQL on Streaming Data

Now what is possible: you can use SQL and query on the streaming table. The streaming data can be queried because you converted a stream to a DataFrame — a DataFrame is a kind of structured data — you did not have to do batching, it has become an unbounded table, and you query it using SQL to find out certain analytics. That is running SQL on the streaming data.

So a query like "what is the average metric value per device in the last hour?" can be written in plain SQL and executed against a live stream — the engine re-runs it incrementally as new rows arrive, without the developer writing any streaming code. Whether it is SQL or PySpark or something else, the same structured abstraction serves them all — the common abstraction promised in Section 9.3.4.

This is an important practical point: "unbounded" means unbounded in time — the stream has no fixed end — but not unbounded in space. The table lives in the cluster's memory, so its size is bounded by memory. Applications that need long histories handle this by keeping only summaries (windows, aggregations) in the table and writing raw records to persistent storage.

Exam note: you can get a question where you have to write queries, ranging across all the NoSQL data stores — and SQL on streaming data, as shown here, is part of that query-writing family.

9.11 Shuffles: The Costly Wide Transformation

9.11.1 What ReduceByKey Does

Shuffle is one of the operations that is actually a costly operation. Costly means: an operation which needs the data from multiple nodes — a wide transformation. Let us take the example of reduceByKey. What reduceByKey does: it generates a new RDD where all the values for a single key are combined into one tuple, and the key and the result of executing a reduce function against all the values is associated with that key.

Let us look at an example — we have done this earlier, and it is similar to the reduce operation we did in MapReduce. We have a key-value pair dataset:

\[(1, 2), \quad (3, 6), \quad (3, 4)\]

I want to reduce by key. I will collect all the points with key 1 — it is only one point, so it will be one single thing. I will collect all the points with key 3 — there are two points, so I will combine them. I want to combine the values, and here I am combining them like adding those values: \(6 + 4 = 10\). So the result will be:

\[(1, 2), \quad (3, 10)\]

9.11.2 Why a Shuffle Happens

What is the bigger challenge in doing this kind of operation? Not all the values for a single key necessarily reside on the same partition. This \((3,4)\) may be on another partition; this \((3,6)\) may be on another partition — same key, different partitions. To compute the result we must locate them; we must bring them together to compute the result — only then is it possible. That triggers a shuffle operation to occur, and when a shuffle occurs, a lot of data transfer happens.

The shuffle is Spark's mechanism for redistributing the data so that it is grouped differently across partitions. This typically involves copying the data across executors and machines, which makes the shuffle a complex and costly operation. The data is not generally distributed across partitions in the place needed for a specific operation: during computation, a single task will operate on a single partition. So to organize all the data for a single reduceByKey reduce task to execute, Spark needs to perform an all-to-all operation: it must read the data from all the partitions to find out all the values for all the keys, then bring together those values corresponding to a common key and compute the final result. This is what we call a shuffle, and it is a very costly operation — we will optimize certain things in this.

Why exactly is it so costly?

• All-to-all data movement. Every source partition may hold a piece of every key, so each reduce task can need input from every map-side partition. The data crosses the network between executors — the slowest resource in the system.
• Spill to local disk. Even for in-memory RDDs, the shuffle implementation writes its output to partitioned files on local disk between the stages, and the next stage fetches those files.
• Serialization and fetch. Data is serialized, written, read back, deserialized, and transferred — CPU and I/O costs on every byte.
• Loses locality. The data locality principle from Section 9.1 is violated: the data has to move to where the reduce tasks run, instead of the tasks moving to the data.

9.11.3 Operations That Trigger Shuffles

Operations which can cause a shuffle include:

• repartition() — repartitioning redistributes data to a different number of partitions, which requires moving records across nodes.
• reduceByKey() — grouping values by key brings same-key records together (as just shown).
• groupByKey() — the same grouping, without the reduce step: all values for each key are gathered into an iterable.
• join() — joining two datasets on a key must bring matching keys together across both inputs.

All of these share the same root cause: records that belong together are scattered across partitions, and the engine must gather them before the operation can proceed. Contrast with the narrow transformations — map, filter, flatMap — where each output partition depends only on its own input partition: those never shuffle.

9.11.4 Working Around Shuffles with DAG Stages

How do we optimize? Remember the DAG is computed in stages. You carry out certain transformations, and you place certain transformations in certain stages: transformations which can be run parallelly across the nodes should be put in one stage. Whenever the shuffle happens — the data transfer happens — you put the transformations on the other stage. All transformations which can be carried out without shuffling the data can be run parallelly. And the shuffle, if at all we can postpone it, let us postpone it — it should be put in a second stage if at all possible. That is how we optimize, because if we do the data transformation multiple times that will be very, very costly.

The stage structure is Spark's answer to shuffle cost:

• The DAG of transformations is divided into stages at shuffle boundaries.
• All transformations inside one stage are pipelined: each task does its full chain of map/filter/etc. on its partition, in parallel, with no data movement between them.
• A shuffle forces a stage boundary: the stage before the shuffle writes its partitioned output; the stage after fetches it and continues.

A job like word count — textFile → map(flatMap to words) → reduceByKey → count — becomes two stages: the flatMap and map side run entirely in parallel in stage 1 (the reduce function even runs as a combiner on the map side), and only the reduceByKey grouping needs the shuffle into stage 2. By keeping every non-shuffling transformation in the same stage, the engine guarantees the data is touched once, not repeatedly serialized and re-fetched.

Exam note: expect questions on which operations cause shuffles and why the shuffle is costly — the reduceByKey walkthrough above is the canonical example. Also, given a use case, you may have to write map and reduce programs for it; the things will not be very complex.

9.12 RDD Persistence and Storage Levels

9.12.1 Persisting and Caching an RDD

One of the most important capabilities in Spark is persisting or caching a dataset in memory. When you persist an RDD, each node stores any partitions of it that it computes in memory and reuses them in the other actions on the dataset, which allows future actions to be much faster. You can mark an RDD to be persisted using the persist() or cache() methods. The first time it is computed in an action, it will be kept in the memory on the nodes.

Two clarifications that make the behavior concrete:

• persist() and cache() do not compute anything by themselves. They mark the RDD with a flag: "when this RDD is first computed, keep it in memory". The marking is lazy; the storage happens when the first action forces the computation.
• The benefit is reuse. A transformed RDD used by several downstream actions — say, a cleaned dataset feeding both an aggregation and a join — is computed once and served from memory afterwards, instead of recomputed from its lineage on every action. This is the same reason MapReduce is at a disadvantage here: each MapReduce job reloads its input from disk, while Spark's cross-cluster in-memory cache keeps the working set resident.

The Spark cache is fault tolerant: if any partition of an RDD is lost, it will automatically be computed again using the transformations. It is the same thing as with the DAG: if any partition of an RDD is lost, we can always recreate it using the transformations that originally created it. Persistence and lineage are complementary: persistence makes reuse fast, lineage makes loss survivable — a lost cached partition is simply recomputed from its parents in the DAG, not fetched from a backup.

When would you want this? Suppose spillover is happening — certain RDDs are spilling to disk and you do not want that — or you know you need the same RDD again later. Then you persist: you use a persist function or a cache function, and the first time it is computed in an action it will be kept in the memory on the nodes.

Visual intuition: Picture a printer (the lineage DAG) that reprints any page on demand. Persisting is like pinning a page to the wall: the first time it is printed you keep it, and everyone who needs it reads it from the wall instead of waiting for another print run. If the page falls off the wall (a node dies), the printer can still reprint it — that is the fault tolerance — but the fast path of reading from the wall is what persistence buys.

9.12.2 The Storage Levels

Each persistent RDD can be stored using a different storage level, allowing you to persist the dataset on the disk, or to persist it in memory as serialized Java objects. When you store data as serialized Java objects it requires less space; however, the retrieval may be a little time-consuming. These levels are set by passing a StorageLevel object to persist. The cache() method is a shorthand for using the default storage level — StorageLevel.MEMORY_ONLY.

The storage levels:

• MEMORY_ONLY — store the RDD as deserialized Java objects in memory. This is generally the fastest, most CPU-efficient option: the objects are already in their usable form, so retrieval has no serialization cost.
• MEMORY_ONLY_SER — store in memory as serialized Java objects. This is generally more space efficient than deserialized objects, especially with a fast serializer, but it is more CPU intensive to read — every read must deserialize the byte arrays back into objects. Each partition is stored as one byte array, which also reduces garbage-collection pressure.
• MEMORY_AND_DISK — store as deserialized Java objects in memory; if partitions do not fit in memory, store the partitions on disk and read them whenever they are needed.
• MEMORY_AND_DISK_SER — similar to MEMORY_ONLY_SER, but spill the partitions that do not fit in memory to disk instead of recomputing them on the fly each time they are needed: store in memory in a serialized fashion, and if spillover happens, go to disk.

Each storage level offers a different trade-off between memory usage and CPU: storing serialized objects saves space but costs CPU to read and write.

The serialized levels also benefit from a fast serialization library: Kryo serialization is typically smaller and faster than the default Java serialization for RDD partitions, and compressing serialized partitions (spark.rdd.compress) saves more space at the cost of CPU.

9.12.3 Choosing a Storage Level

Which storage level to choose? If you do not know your data, probably this would be the last one — MEMORY_AND_DISK_SER. The recommended process:

1. If your RDD fits comfortably with the default storage level (MEMORY_ONLY), leave it the way it is. Retrieval is fast, it is the most CPU-efficient option, and the operations on the RDD run as fast as possible because your whole data is in memory.
2. If it does not fit, try MEMORY_ONLY_SER and select a fast serialization library, to make the objects more space efficient but still reasonably fast to access.
3. Do not spill it to disk unless the functions that computed your dataset are expensive, or they filter a large amount of data. If the functions are expensive and something is lost, you have to do all the functions again; if they filter a very large amount of data, the filtration process will happen again. So do not spill to disk unless one of these conditions holds.
4. Use the replicated storage levels if you want fast fault recovery.

Exam note: storage levels and the recommended choice process are classic exam material — know which level is space-efficient vs CPU-efficient and when to spill to disk. Answers should be crisp and straightforward, giving all the reasoning, not large story-writing answers.

9.12.4 Fault Tolerance and Replication

All the storage levels provide full fault tolerance by recomputing the lost data. The replicated ones let you continue running the task on the RDD without waiting to recompute a lost partition. If you have replicated an RDD across multiple nodes, even if a partition is lost, the task — or the transformation — can be shifted to the node where the copy still exists.

So there are two tiers of protection:

• Recomputation (available everywhere): any lost partition is rebuilt from its lineage. This costs time — the DAG path to that partition must re-run.
• Replication (replicated levels only): a second copy lives on another node, so a lost partition is served from the copy instantly, and tasks simply move to the node holding the replica. You pay extra memory for the copy; you gain speed of recovery.

9.12.5 Eviction and Manual Removal

Spark automatically monitors the cache usage on each node and drops out the data partitions in a least recently used (LRU) fashion: the dataset or RDD which was used least recently is dropped; the most recently used stays in memory.

You also have the option to manually remove an RDD instead of waiting for it to fall out of the cache: the unpersist() method is there, which you can use to drop an RDD from the main memory. This is useful when you know an RDD is finished — freeing memory immediately for the next dataset rather than waiting for LRU pressure.

Exam Guidance Summary

Weightage

EC1 is 30 percent, EC2 is 30 percent, and EC3 — the comprehensive exam — is 40 percent weightage, the highest out of all. The EC1 components — the quizzes and assignments — are evaluated and frozen now. Assignment 2 carries 15 percent weightage, and the marks will be scaled.

Focus Areas

• Overall focus: the comprehensive exam could be similar to the mid-semester exam, but the focus would be more on the part after the mid-semester — more focus on the NoSQL stores and writing certain queries.
• NoSQL: the focus is on all the NoSQL stores covered, starting from MongoDB, HBase, and Cassandra. Dynamo was not covered in detail; whatever was covered related to it will be based on that.
• Queries: you can get a question where you have to write queries. The queries could range from all the NoSQL data stores. Also, given a use case, you may have to write map and reduce programs for it — the similar thing which we covered in the mid-semester. The things will not be very complex.
• Pig: maybe you might be asked to write Pig queries — Pig Latin — for a particular thing, or how Pig runs queries. You do not have to execute anything. Do not use any of the execution environments — not the remote labs, not something installed on your laptop — and do not paste a screenshot. Everything should be typed in the answer text box: you write the query, and that is what will be seen.
• Spark: Spark is a major module — like Hadoop and MapReduce was a major one — and similarly, quite a good portion of the comprehensive exam would be from Spark. But it is not going to be very tough: simple things, whatever we have covered, will be there.
• Earlier material: whatever is there for reliability, availability, Hadoop, and MapReduce will still be a part of the comprehensive exam. But the larger focus is on the post-mid-semester part.
• Architecture: if there is something on the architecture level, whatever architecture we have covered is the focus. From the range of databases, look into what was covered — everything may not have been covered. The labs covered were MongoDB, MapReduce, and Spark only; the Pig lab part was not covered. Study whatever is covered in the class and pay attention to that.

Answer Style and Instructions

• Answer style: answers should not be very, very large, story-writing kinds of things. Straightforward answers should be there, as crisp as possible, giving all the reasoning. Sometimes what I have found is: if the students know one word — they have heard about that word — they will write two pages about that. That is not the expectation.
• Instructions: read the instructions carefully and then follow them.

Key Industry Applications

• Buffering message buses: the data flow tier's impedance-mismatch buffering and its at-least-once guarantee are exactly the role industry message-queue systems like Kafka play between producers and consumers, absorbing sale-day spikes on e-commerce sites without losing events.
• Edge computing and IoT: edge servers collecting data from IoT devices and pushing it deeper into backend cloud systems is the real deployment pattern for the collection tier.
• Log analysis at scale: the original use case for streaming collection was website log analysis to detect errors and deviations, with the format evolving from the W3C standard log format to JSON, Avro, and Thrift.
• Dashboards and alerting: delivery tiers in industry ship continuously updated dashboards (HTML5, WebSockets), alert on error keywords in streaming logs, export reports as PDF or SVG, and feed downstream applications.
• NoSQL for permanent storage: MongoDB and Cassandra are the popular choices for persisting streaming outcomes for later historical analysis; the choice depends on the use case — consistency, wide-columnar data, or JSON documents.
• Streaming engines in practice: Apache Storm and Spark Streaming for heavy processing, Kafka Streaming for lightweight computations, with ingestion from Kafka, Flume, Kinesis, S3, HDFS, and Twitter streams.
• Machine learning on live streams: k-means and other ML library functions applied to stream RDDs — for example, labeling and grouping records as they arrive.
• Fraud detection and recommendations: streaming engines in production run fraud detection on online transactions, click analysis for online recommendations, push notifications for location-based retail offers, and emergency-service alerts on abnormal measurements in healthcare — all following the architecture of this lecture: collect, buffer, process, store, deliver.
• Real-time sentiment and market analysis: real-time feeds from social media and stock exchanges are used for sentiment analysis and future price prediction, with tweets pulled in real time, processed, and loaded into persistent storage.