Kafka Streaming Demo and a High-Level View of Spark

This lecture runs the Kafka producer–consumer console demo end to end — architecture recap, live walkthrough, and the Windows environment setup that makes it run — then steps back for a high-level view of Spark: what it is, how its layers fit together, and the programming models it offers.

9.1 Course Roadmap and Assessment Plan

9.1.1 The Four Remaining Modules

Half of the course is over, and everything that remains is about streaming frameworks. Four topics are left on the schedule: Kafka streaming, Kafka–Spark integration, Spark streaming, and streaming algorithms. Notice what the list says about the course's shape: the first half built the foundations of distributed batch processing, and the second half is devoted to moving data continuously instead of in fixed batches.

The instructor plans to give Kafka–Spark integration two to three classes, and three in practice, because that is where the two big frameworks meet — Kafka is the message pipeline that feeds events in, and Spark is the processing engine that consumes them. The remaining classes will focus predominantly on Spark streaming, on Kafka–Spark integration, and then on some streaming algorithms.

9.1.2 The Timeline

The course ends on May 12. This month (March) has two more classes. April has four classes on the 7th, 14th, 21st, and 28th, and May has two classes. Around eight classes remain in total, including one session the instructor missed and needs to compensate with an extra class, so after today's class seven classes remain.

There is no regular class next week, because the makeup exam falls in that slot.

9.1.3 Assessments and Exam Guidance

Both assignments are already published — assignment one and assignment two — and the plan for the second quiz depends on finishing one more piece of content first. The instructor wants the introduction to Flink streaming completed before taking up that quiz. Apache Flink is a real stream-processing engine (a sibling of Spark Streaming with native support for event-time windows and exactly-once state), so the module name resolves cleanly to Flink. After that, the instructor will show how to create an account on the Databricks platform so the assignment problems can be solved there. The quiz is planned for around the first week of May.

9.1.4 Student Questions and Answers

9.2 Kafka Streaming: Architecture Recap and Live Demo

9.2.1 The Core Building Blocks

The Kafka architecture rests on three moving parts: the Kafka broker, the Kafka producer, and the Kafka consumer. Producers and consumers can be of any type — the producer is whatever writes events, the consumer is whatever reads them, and the broker sits between them and stores what passes through. You can think of the broker as the shared mailbox: the producer drops letters in, the mailbox holds them, and the consumer comes to collect. The producer never hands the letter directly to the consumer, which is what lets them be started, stopped, and scaled independently.

Three earlier concepts complete the picture: the Kafka topic, the Kafka partition, and the Kafka offset.

The last piece is what gives Kafka its superpower: because each partition is an append-only log, any consumer can rewind to an older offset and replay the stream from that point. In the queue design most beginners imagine, a message is deleted the moment one consumer reads it. Kafka instead keeps messages around (for a retention window such as 12 hours or 50 GB), and each consumer tracks its own position. That means many consumers can read the same topic independently, each at its own pace, and a failed job can re-read exactly the events it missed instead of losing them forever.

Assumption and scope. This design assumes messages are append-only and never edited once written — the broker only ever adds to a partition. If a system needs to modify already-stored events in place, Kafka is the wrong tool. Also, ordering is guaranteed within a partition only; Kafka makes no ordering promise across different partitions of the same topic.

Pitfalls. A common beginner trap is to believe that deleting a message from one consumer's read deletes it for everyone — it does not; each consumer group tracks its own offset. Another is to think the offset is global across the topic; it is local to a partition, so two partitions each restart their numbering at 0. And a third: reading "from the beginning" (offset 0) replays every retained message, which can flood a consumer that expects only new arrivals.

9.2.2 Worked Walkthrough: The Producer–Consumer Demo

This demo was postponed from the previous class, because that class was spent on the sample paper. Today it runs end to end. The full sequence needs Zookeeper, then the Kafka broker, then a topic, then a producer and a consumer, and the commands come from the Kafka documentation — the instructor's standing advice is that none of these commands need to be memorized; copy them from the documentation.

zookeeper-server-start.bat C:\streaming\confluent\etc\kafka\zookeeper.properties
kafka-server-start.bat C:\streaming\confluent\etc\kafka\server.properties
kafka-topics.bat --create --topic example_event --bootstrap-server localhost:9092
kafka-console-producer.bat --broker-list localhost:9092 --topic example_event
kafka-console-consumer.bat --topic example_event --from-beginning

Step 1 — start Zookeeper. In a command window, run zookeeper-server-start.bat with the complete path to zookeeper.properties. The shell initially failed to recognize the properties file, and the fix was to give the entire path instead of a partial one. Once the file loads, Zookeeper is up. Zookeeper is the coordination service the Kafka cluster depends on — it keeps track of which brokers are alive and which partitions each consumer should read — and it must run first. This is the reason the demo script order is not arbitrary: the broker registers itself with Zookeeper, and consumers read their assignment from Zookeeper, so nothing works until Zookeeper is up.

Step 2 — start the Kafka broker. In a second command window, run kafka-server-start.bat with the full path to server.properties. The broker starts, and the console reports that the Kafka server with id 0 has started. That "id 0" is the broker's identity inside the cluster; a production cluster runs several brokers with distinct ids.

Step 3 — create the topic. Run kafka-topics.bat --create and give the topic a name; instead of the default quickstart name, the topic here is example_event. The flags follow the standard create-topic form, with --bootstrap-server localhost:9092 telling the command which broker to contact. Without the topic, the producer and consumer would have nothing named to attach to.

Step 4 — start the producer. Run kafka-console-producer.bat pointing at the topic. Here the class watched a live error correction: the instructor first typed the flag used for the consumer, then caught the mistake — for the console producer the flag is --broker-list, not --bootstrap-server. Once the correct flag was in place, the producer showed its greater-than prompt (>), which means it is ready to accept input.

Step 5 — start the consumer. In another window, run kafka-console-consumer.bat for the same topic. To see messages that were sent before the consumer started, add the --from-beginning flag, which tells the consumer to read from offset 0 rather than from the newest message onward. The standard console consumer follows exactly this form: the topic name plus the begin-from flag when you want the full history.

Step 6 — send messages. Everything typed in the producer window flows to the consumer window: the instructor typed "Hello" and then "Hello Kafka", and both appeared on the consumer side. That is the whole mechanism in miniature — the producer writes messages to the topic, the broker holds them, and the consumer reads them off.

Real-world: Confluent offers a free version of Kafka, and the same console commands shown here are the quickest way to prove a Kafka cluster is alive before wiring it to real applications. Production teams use the identical five-step sequence — Zookeeper, broker, topic, producer, consumer — as the sanity check after a fresh install.

9.2.3 The Demo Environment: Confluent Folder Layout

The layout of a Confluent Kafka installation explains where every command and configuration file lives. In the installation folder there is a bin folder, and inside it a windows subfolder holding Windows-compatible batch files — the .bat files used above. That is why the PATH entry points into confluent\bin\windows: the command-line tools themselves are the batch files in that subfolder.

The properties files sit elsewhere: in Confluent Kafka they live in the etc folder, under etc\kafka, which holds zookeeper.properties, server.properties, and the producer and consumer properties files. If you use plain Apache Kafka instead of Confluent, the same properties files are under a config folder instead. The layout difference is only a folder name — etc\kafka in Confluent versus config in vanilla Apache Kafka — and every demo command needs the full path to its properties file, whichever distribution you use.

9.2.4 Worked Walkthrough: Windows Installation and Environment Setup

The setup was shown live, and the instructor shared a text document with the installation steps for both Kafka and Spark. The order matters, and each step is an environment variable or a PATH entry.

1. Install Java, then create the JAVA_HOME environment variable pointing at the JDK folder. This was shown in the environment variables dialog, with JAVA_HOME attached to the installed JDK directory. Kafka and Spark are both JVM software, so this variable is the foundation everything else builds on.
2. Download Spark — the Spark download bundles Hadoop's file system — and separately download the Hadoop Win Utilities, commonly called winutils. Compatibility is the key word: for Spark 3.0.0 the compatible winutils version is 2.7.7. Download the zip, keep only the Hadoop version compatible with your Spark, unzip it into a folder like C:\streaming\hadoop, create HADOOP_HOME pointing there, and add it to the PATH.
3. Kafka: create KAFKA_HOME pointing at the Confluent Kafka installation folder, then add the bin folder — specifically confluent\bin\windows — to the PATH.
4. Spark: download Spark, create SPARK_HOME, and add the spark\bin folder to the PATH.
5. PySpark needs Python, since PySpark is the Python flavor of Spark. Install Python with python.exe, or use the Anaconda Python that comes with Anaconda 3. Create the PYSPARK_PYTHON environment variable and point it at the Anaconda Python executable (for example the python.exe inside the Anaconda 3 folder). Then link the Spark 3 Python folder, and add the unit testing library (py4j) that PySpark needs, into the same configuration.

Pitfalls. Version mismatch is the first failure mode: winutils 2.7.7 belongs with Spark 3.0.0, and pairing an arbitrary winutils with a Spark version it was not built for throws filesystem errors at runtime. The second is pointing PYSPARK_PYTHON at the wrong interpreter — if Anaconda is installed but PYSPARK_PYTHON still names a bare python, Spark may use the wrong Python and fail to import its own modules. The third is skipping the PATH steps: tools work only from the folder where the .bat lives until their bin folders are on the PATH.

Real-world: this local Windows setup is the free path to PySpark, and it is exactly what the Databricks platform provides as a managed service for the assignment. The winutils step exists because Hadoop's file system code, which Spark bundles, needs Windows-native binaries on this platform — on Linux and macOS that code runs without any extra download, which is why the winutils step appears only in Windows guides.

9.2.5 Student Questions and Answers

9.3 Spark: A High-Level View

9.3.1 What Spark Is

Spark is a processing engine that sits above the MapReduce techniques used conventionally. Whenever you think of Spark, think of in-memory computation. It is a very generic distributed data processing model — generic in the sense that the same engine handles batch processing, stream processing, and real-time processing. One engine, three workloads: that generality is what makes it the workhorse of modern data platforms.

Two capabilities make it special. First, it supports iterative algorithms, where the same computation passes over the data repeatedly and intermediate results stay in memory. This is the decisive difference from classic MapReduce: a MapReduce job that iterates must write its intermediate output to disk between passes, while Spark keeps the data cached in memory, so machine-learning algorithms that loop over the same dataset hundreds of times run dramatically faster. Second, it supports interactive querying: you can write queries on the streaming data the way you write SQL queries on a stationary data table. That is the uniqueness of Spark.

9.3.2 Spark Architecture: Cluster Managers, Spark Core, and High-Level APIs

The architecture block diagram builds from the bottom up. The bottom layer is the cluster manager, which can be Mesos, YARN, a standalone scheduler, or Kubernetes. The cluster manager's job is the resource negotiation: it decides which machines in the cluster get assigned to run which parts of a Spark program, and it hands over memory and CPU as the job asks for them.

On top of the cluster manager sits the Spark Core — the layer where the Spark program is written. Spark Core is the real, standard name of this layer: it is Spark's engine, the part that schedules tasks, moves data, and keeps the program alive across machine failures. The Spark Core program can be implemented in Python, Scala, or Java.

On top of the core sit the high-level APIs. One is the structured streaming API, which the instructor called very, very powerful, popular, and innovative. The other high-level pieces are real-time processing, Spark SQL, the Machine Learning Library (MLlib), and the Graph library (GraphX). These libraries sit above Spark Core and give you ready-made building blocks — SQL queries, machine-learning models, graph algorithms — instead of raw task-level programming. For the assignment, the class will use the Machine Learning Library.

The instructor's practical note: if you are good in Java, Python, or Scala, that is enough for stream processing — and this course uses Python, which means PySpark.

Real-world: managed Spark platforms such as Databricks provide the same architecture as a service, and Spark integrates with the Hadoop HDFS, with Kafka, and with AWS, so the architecture you learn here is the one used in production deployments.

9.3.3 Programming Models: RDD versus Structured Streaming

Spark programming can be done in two ways. One uses the RDD — the resilient distributed dataset — and the other uses the structured streaming API. The RDD is the old model and is obsolete as well; it is not popular now. The structured streaming API is the most commonly used model currently. The course looks at both: one or two RDD examples for familiarity, and structured streaming in much more detail.

Comparison — when to pick which.

One-sentence rule: use RDD only to understand how Spark thinks; write new work in structured streaming.

9.3.4 Data Ingestion and the Machine Learning Library

Data ingestion into the Spark streaming environment can come through many platforms — Kafka, Kinesis, Twitter, and so on — and the Spark documentation shows these ingestion diagrams. The pattern is always the same: events arrive on a message platform, Spark reads them from it, and the streaming query processes them as they come. Because the sources are replayable (Kafka offsets, Kinesis position), Spark can re-read anything it missed after a crash.

On the machine learning side, MLlib covers the typical tasks: classification, regression, clustering, collaborative filtering, and more. The role of machine learning in PySpark is to analyze the events in real time and make predictions and labeling on these events. The instructor's example: when you are predicting the price of a particular vegetable, you need to look at every column of the data, and the next step is data cleaning — checking each column before it feeds the model.

Real-world: this is the production pattern behind real-time scoring — events arriving through Kafka or Kinesis are classified or labeled as they arrive, which is what companies mean when they say real-time prediction on streaming events.

9.3.5 Student Questions and Answers

9.4 What Comes Next

9.4.1 Before the Next Class

The next class starts Spark streaming with examples. Meanwhile, two things are expected. First, set up the local environment using the installation document that was shared — the next class assumes Kafka and Spark are installed. Second, complete part one of the assignment by the end of this month. The assignment has two parts and the submission link is one link for both, but part one should be ready even if the final submission happens together.

9.4.2 Student Questions and Answers

Exam Guidance Summary

• The mid-semester makeup exam is next week; there is no regular class in that slot, and one student sits the makeup on Sunday.
• The second quiz is planned for around the first week of May, after the introduction to Flink streaming concepts is completed (Apache Flink, the stream-processing engine, resolves the earlier audio ambiguity).
• The exam question about consumers and producers confused several students, while the other two questions were manageable — study Kafka producer and consumer concepts, and the console demo, before the makeup. In particular, remember which flag belongs to which console tool: the producer uses --broker-list, and the consumer reads from the beginning with --from-beginning.
• Assignments one and two are both published with a single submission link; complete part one by the end of this month regardless.
• Databricks account setup will be explained so the assignment problems can be solved on the platform.

Key Industry Applications

• Real-world: Confluent Kafka's free distribution and console scripts for quick cluster verification — the same five-step console sequence (Zookeeper, broker, topic, producer, consumer) is the standard smoke test after any Kafka install.
• Real-world: production ingestion into Spark streaming from Kafka, Kinesis, and Twitter — replayable sources let Spark re-read anything it missed after a failure.
• Real-world: MLlib for real-time event analysis, prediction, and labeling — the pattern behind real-time scoring systems, where incoming events are classified or labeled as they arrive.
• Real-world: Databricks as the managed platform for the assignment, mirroring local PySpark setup — the architecture you set up by hand on Windows is exactly what the managed service provides.
• Real-world: local Windows PySpark requires Hadoop Win Utilities matched to the Spark version (winutils 2.7.7 for Spark 3.0.0), because Spark's bundled Hadoop file system code needs Windows-native binaries.
• Real-world: Spark integrates with Hadoop HDFS, Kafka, and AWS in production deployments, and the in-memory architecture makes iterative machine-learning workloads practical at scale.