Stream Processing and Analytics — Course Overview and Core Concepts

Stream Processing and Analytics

1.1 Course Overview and Learning Plan

1.1.1 The Two Platforms: Kafka and PySpark

This course is a deep exploration of stream processing (processing data continuously as it arrives, rather than in one big offline job) and streaming analytics (running analysis on that moving data) built around two platforms: Kafka and Apache Spark, the latter through its Python flavor, PySpark. Kafka is the streaming backbone — the system that ingests, stores, and moves continuous streams of events — while Spark is the processing engine that analyzes those streams at scale. Everything in the course connects back to one of these two tools.

The instructor brings 25 years of industry experience, with 8 of those years spent specifically on stream processing and analytics, so the course constantly ties concepts back to how these platforms behave in real production environments — not just how they are documented. Expect war stories, architecture trade-offs, and operational detail that textbooks leave out.

All hands-on activities are covered as part of the class sessions themselves. There are no separate webinars planned for this course — the practical work is not an add-on you complete on your own; it happens live, in class, with guidance. Each session runs about one and a half hours, and the course spans 16 lectures.

1.1.2 How the Course Is Delivered

Every session mixes three elements: minimal slide presentations, detailed class notes, and extensive whiteboard work. The presentations exist mostly as scaffolding; the real explanations happen on the whiteboard, where the instructor draws architectures and walks through designs step by step. The emphasis is on live, interactive explanation — if anything is unclear, you can interrupt and ask on the spot.

The course runs in flip mode: recorded sessions are available, and students are expected to go through them first, then come to class to clarify doubts. Recordings are uploaded ahead of class so you can preview material before the live session. Starting from the second lecture onward, materials are uploaded one class in advance.

A small but telling detail about how the course evolves: one new element was added this run compared to previous offerings — an extra class on structured streaming, which covers how you write SQL-like queries directly on streaming data. The instructor explicitly framed this as continuous improvement: keep the important parts, add hands-on value, and never let the workload become hectic.

1.1.3 What to Expect

The course deliberately does not try to cover every streaming framework in depth. A few platforms — such as Flink, Samza, and AWS Kinesis — are explored through assignments rather than lectures, so you gain a working understanding of the ecosystem. But the deep dive, the part taught in class with full attention, is Kafka and Spark. If you take away real working skill on those two platforms, you have taken away the core of the course.

Real-world: the instructor also plans to demonstrate Kafka streaming with some Java code this run — a first-time addition — because hands-on demos of producers and consumers make the abstraction concrete. If the session count allows it, that Java demonstration stays; if not, it is trimmed without losing anything important.

1.2 Course Structure, Modules, and Resources

1.2.1 Module Roadmap

The course is organized into a clear sequence of modules:

1. Data systems — the first couple of modules establish what data systems are and how they behave.
2. Streaming architectures — next, the course covers the principles of designing architectures for streaming applications: how do you design architectures for streaming applications, and what design principles govern them.
3. Streaming frameworks — this is where Kafka arrives. The course starts with Kafka streaming and covers the Kafka architecture in detail.
4. Post mid-semester content — after the mid-semester exam, the focus shifts to streaming data frameworks, streaming analytics, and streaming SQL, predominantly covered in this second half.

The mid-semester exam lands right after the Kafka architecture is covered. Everything up to that point — data systems, streaming architecture principles, and Kafka fundamentals — is the examinable territory for the mid-semester.

1.2.2 Contact Sessions and Flip Mode

The course plan lists 11 visible contact sessions, but there are actually 16 contact sessions in total; the discrepancy is a quirk of how the plan is displayed, not a reduction in content. The course is in flip mode: students work through recorded lectures first, then attend live sessions to resolve doubts and go deeper.

The first few lectures cover the first modules — data frameworks — and the instructor uses the whiteboard for every session, with slide support where needed. The session flow is always the same shape: explain the concept, show the architecture, then demonstrate it hands-on so each idea becomes something you have seen working.

1.2.3 Textbook and Learning Resources

The course follows one primary textbook: Techniques to Analyze and Visualize Data. Although the handout lists T1 and T2 as secondary texts, this book is the one predominantly followed throughout the course. A soft copy of the book is available, so you are not required to purchase a physical copy — the soft copy suffices for the entire course.

Real-world: beyond the textbook, the practical resources are the setup files the instructor shares — industry installation files for Kafka and the script commands for running producers and consumers. These remove the biggest barrier to picking up streaming technologies: the apprehension around installation and orchestration. There is a structured way to set up this environment on a local laptop, and the course follows exactly that path so you can work in parallel with the instructor.

1.3 Evaluation Components

1.3.1 The Weightage Breakdown

The evaluation is built from six pieces. Two assignments worth 10 marks each, a quiz worth 10 marks, the mid-semester exam (EC2) worth 30 marks, and the comprehensive exam (EC3) worth 40 marks. The first three pieces — assignment 1, assignment 2, and the quiz — combine into EC1, which is worth 30 marks. This is a five-credit course, and the continuous evaluation alone (EC1) accounts for a substantial portion of the total marks.

The quiz is made of objective-type questions — true/false and similar formats. There are two quizzes, each carrying five marks, so the 10-mark quiz component is split across two sittings rather than one high-stakes event.

1.3.2 Quiz Mechanics

The quizzes have a distinctive operational pattern. Each quiz is a timed quiz: for 10 questions you get roughly 15 to 20 minutes, depending on the complexity of the questions. All questions are objective type — true/false and similar formats — so the time pressure is about speed of recognition rather than long written answers.

The time window is the key scheduling detail: each quiz stays open for a 3 to 4 day window, typically spanning a full weekend — enabled on a Friday, open through Saturday, Sunday, and Monday. Within that window you pick any single day to take the quiz. It is not a rigid "opens Sunday, closes Monday morning" arrangement; you get the whole weekend.

You are notified through the course portal when a quiz is published, along with the deadline and schedule, so the operational details are always visible. The professor explicitly encouraged questions about this setup because the quiz mechanics are easy to get wrong if you do not know them in advance.

1.3.3 EC2 and EC3 Exam Shapes

The mid-semester exam (EC2) is worth 30 marks and consists of roughly three to four questions. It is a closed-book exam. The comprehensive exam (EC3) is worth 40 marks and is an open-book exam.

1.3.4 Timeline and Calendar

The calendar places each component deliberately across the semester:

• Assignment 1 is declared after the 6th lecture, with a 3-week submission window. The window is flexible — deadlines are mutually agreeable, because working professionals need room to plan around jobs.
• Quiz 1 sits around early March.
• The mid-semester exam lands in March, after the Kafka architecture is covered.
• The comprehensive exam (EC3) closes the course.

All evaluation criteria for assignments are uploaded to the course portal so nothing about grading is hidden.

1.4 Assignment 1 — Deep Exploration of Streaming Architectures

1.4.1 Scope of the Deep Exploration

Assignment 1 is a deep exploration of a streaming architecture. The instructor maintains a list of streaming platforms — Flink, Samza, AWS Kinesis, and similar — and each group is assigned one architecture to study in depth. The assignment is deliberately research-shaped: you take a real streaming system, understand its architecture, and produce an artifact that documents your findings.

The pedagogical logic is worth understanding. The frameworks in this list are real and used in industry, but the course cannot teach every one of them in depth. By assigning them for exploration, you still gain a basic understanding of the broader ecosystem, while the class time concentrates on the deep dive into Kafka and Spark.

1.4.2 Group Work and Submission

The work is organized in groups of four people, with one assignment given to one group. The submission is an artifact — a document or deliverable you email to the instructor.

Group size is not fixed in stone; it is calibrated to the class strength. Historically this course has had around 45 to 50 students, which comfortably supports four-person groups. But if the current batch is smaller — say 10 to 15 students — the group size drops to 2 to 3 people per group rather than forcing tiny groups to split four-person workloads. The exact group configuration is announced once the class count is known, typically by the next session.

1.4.3 Timeline

Assignment 1 is given after the 6th lecture and carries a 3-week window. The deadline is flexible by design — the instructor explicitly said not to worry about deadlines, because the plan is to discuss mutually acceptable timelines. If you need extra days, you ask, and the schedule adjusts. The only constraint is the natural rhythm of the semester: the submission falls roughly three weeks after publishing, around the 24th of March.

1.5 Assignment 2 — Machine Learning with PySpark on Databricks

1.5.1 Scope: ML on Spark

Assignment 2 is hands-on machine learning: you implement machine learning techniques on Spark, using PySpark on the Databricks platform. The assignment asks you to explore a couple of PySpark libraries, develop machine learning models, and present those models as assignment artifacts.

This is where the course philosophy shows: you learn the machine learning algorithms on your own, for the assignment's purposes, while the instructor demonstrates the platform mechanics — how to create Spark DataFrames, how to do the analytics side of PySpark. So the assignment is not teaching you ML from scratch; it is teaching you how to run ML at scale on a streaming-capable engine.

Real-world: the demonstration covers the analytics aspects of PySpark — creating Spark DataFrames and working with them — which is precisely the skill set used in production data engineering work.

1.5.2 Databricks Community Edition

The assignment runs on the Databricks community edition, which is free — you can use the Databricks cloud platform without paying anything. This matters because installing and orchestrating Spark clusters is exactly the kind of setup work that makes people apprehensive about these technologies; Databricks removes that entire class of problems.

Real-world: Databricks was built by the creators of Spark, and it handles everything for you — cluster installation, cluster management, all of it. When you register for the community edition, the platform asks whether you want the AWS flavor, the Azure flavor, or the GCP flavor, and then provides the overall ecosystem around your choice. There is extensive help documentation for usage. Some workloads carry a small cost, but it is not high — and compared to GCP, AWS, or Azure, Databricks has become increasingly prominent because of its features and its user-friendliness, especially for Spark analytics.

1.5.3 One Notebook for Both Assignments

A practical convention simplifies submission: both assignments can be submitted together in a single Jupyter notebook — this has been the established practice and continues this run. The notebook has two parts:

• Part one: the artifacts of assignment 1 — the documentation of the streaming architecture exploration. Jupyter helps you write nice documentation, the way you would present a slide deck, so the entire content goes in as part one.
• Part two: the coding for assignment 2 — the PySpark ML implementation.

One file, both assignments, submitted at the end. That has been the practice, and the course continues with it.

1.6 Stream Processing vs Batch Processing

1.6.1 The Time-Scale Rule

The single most important distinction in this course is stream processing versus batch processing, and the deciding factor is one thing: the time scale you work on. The instructor's bottom line is compact and worth memorizing:

• If you analyze data on a time scale of minutes or seconds, it is stream processing.
• If you schedule your analysis activities hourly, every eight hours, daily, or every two days, it becomes batch processing.

Real-world: in the BFSI (banking, financial services, and insurance) domain, streaming analytics is everywhere. But the same organization often needs both modes: real-time fraud checks on transactions while they happen (stream), and overnight risk analysis over millions of customer records (batch). The time scale of the question determines the mode, not the industry or the technology.

1.6.2 Customizable Batch Size

Kafka and Spark both support customizable batch sizes, and this is what makes them flexible enough for both regimes:

• Micro-batch processing — near-real-time windows that process small batches of events continuously.
• Stream processing — true event-by-event processing.
• Proper batch processing — large jobs scheduled at fixed hours of the day.

Because you can tune the batch size, one platform can serve both the streaming and the batch sides of a business. The practical implication for architecture: your architecture should be able to switch between these strategies as the analysis requirement moves along the time scale.

1.6.3 One Architecture for Both Strategies

A clarifying question from a student produced the richest explanation of the session — the collection-window view of the problem. The answer used an analogy that runs through the entire course (developed fully in the next section): stand on the bank of a river, and you cannot capture the whole river in a bucket. You capture a sample, and the sampling strategy — count-based or time-based — is exactly the window mechanism of stream processing. Once the sample is collected, you apply algorithms to it and draw inferences.

The deeper point is that the stream/batch question is not a binary technology choice. The same event data can be analyzed as a stream (per-second fraud scoring) or as a batch (nightly trend reports), and a well-designed architecture keeps both options open.

When to pick which: let the freshness requirement of the business question decide — seconds need stream, hours need batch, and a well-designed architecture keeps the ability to switch between the two rather than committing to one technology.

1.7 Windows in Stream Processing

1.7.1 The River Bucket Analogy

The central mental model for windows is the river bucket analogy. Imagine you stand near the bank of a river and want to analyze the minerals in the water. You cannot capture the entire river in a bucket — so you collect a sample. The question is how you decide when to stop collecting, and there are exactly two strategies:

1. Collect water up to a particular level in the bucket — a fixed amount of water, whatever that takes.
2. Collect water for a fixed duration — put a stopwatch on, fill for five or ten minutes, and stop when the timer ends.

These two strategies map exactly onto the two window mechanisms of stream processing: event-based (count) windows and time-based (temporal) windows.

1.7.2 Event-Based (Count) Windows

The first strategy — fill the bucket to a level — is the event-based window. You collect events until the window reaches a specified number of events, then you process that group and start the next window. The window is defined by how many events it holds, not by how long it took to collect them. In the analogy, the bucket level is the count; a slow stream takes longer to fill the bucket, a fast stream fills it quickly, but the processing unit is always "these N events together."

1.7.3 Time-Based (Temporal) Windows

The second strategy — the stopwatch — is the time-based window. You collect every event that arrives within a fixed time frame — for example, one second or one minute — regardless of how many events that turns out to be. When the window closes, you process whatever accumulated. In the analogy, the stopwatch is the clock; a busy river fills the bucket faster, a quiet river slower, but the processing unit is always "everything that arrived in this time slice."

1.7.4 Why Windows Matter

Windows exist because infinite data cannot be processed as an infinite batch. You need a finite unit to apply algorithms to, and the window defines that unit. Two properties of windows are worth internalizing:

• You use a logical window for the accumulation of events — the window is a logical grouping, not a physical storage constraint.
• Whether a particular event is selected into a window is a random process — the stream arrives continuously, and the window simply captures whatever falls inside its boundaries.

Once the window closes, the collected events are analyzed with appropriate algorithms and the inferences are drawn — the same analysis step whether the window was count-based or time-based. Choosing the right window length (and type) is a design decision that shapes the latency and accuracy of the whole pipeline: shorter windows give faster answers on less data; longer windows give more stable answers but delayed results.

1.8 Deterministic vs Probabilistic Streaming Algorithms

1.8.1 The Deterministic Baseline

Before streaming, recall what an ordinary algorithm guarantees. A deterministic algorithm produces the same output for the same input, every time. The professor's example is a linear search used to count the frequency of occurrence of an element: run it today at 3 o'clock and you get a count; run the same thing at 4 o'clock and you get the same count. The answer is stable because the input is fully known and the algorithm examines it completely.

Determinism is a property of the input and the method, not the hardware. The count is exact because every occurrence is examined — nothing is sampled, nothing is dropped.

1.8.2 Why Streaming Algorithms Are Probabilistic

Streaming algorithms, by contrast, are probabilistic and non-deterministic. The reason is the sample. You only collect a random sample of the stream — remember the bucket in the river — and the sample may not reflect the entire stream. Randomness is involved in the collection of the sample, so the analysis built on it inherits that randomness.

This is why the algorithms used in data processing over streams — and even over batch samples — are non-deterministic in nature. The same analysis run twice can produce slightly different numbers, not because the code is buggy, but because the underlying sample differs. That is the fundamental trade: exactness is sacrificed for the ability to handle data too large to hold, at the speed the stream demands.

1.8.3 The Logical Window and Random Sampling

The bridge between the two regimes is the logical window. You accumulate events into a logical window, decide an appropriate window length, and within that window the selection of a particular event is a random process — whether an event lands in the window is governed by timing and arrival, not by choice. Consequently, the sampling and analysis techniques used in streaming are probabilistic techniques.

The design consequence: when you design a streaming analytics solution, you are not choosing between "right" and "wrong" algorithms in the classical sense. You are choosing probabilistic algorithms with known error characteristics, and the window length is your dial for controlling accuracy. Understanding which traditional algorithms carry over to streaming — the course spends two to three lectures on exactly this — is a core skill of the second half of the course.

1.9 Real-World Applications of Stream Processing

1.9.1 Credit Card Fraud Detection

Real-world: the canonical streaming use case is credit card fraud detection. Credit card transactions happen continuously, day in and day out, and each transaction is captured as an event — a very small piece of data, a few kilobytes, like a mobile SMS. The question is how to analyze this data in real time.

Here is the everyday experience the architecture explains: if you swipe a card for more than 80,000 rupees, you get a phone call from customer care saying a transaction just happened — press one to confirm, press two to raise an issue. Behind that call, the bank's stream processing engine analyzes transactions as they occur, classifies each one as genuine or fraudulent using machine learning, and triggers the confirmation call the moment a suspicious transaction passes. This is real-time transaction classification with machine learning, running on streaming data.

1.9.2 Ride-Hailing and Stock Trading

Real-world: ride-hailing platforms such as Uber and Ola are streaming businesses. The availability of cabs, traffic congestion in real time, and the decisions built on that information — where to route cars, how to price a ride — all depend on processing streams of location and demand events as they happen. Traffic information loses its value the moment it is stale; streaming is the only mode that works.

Real-world: online stock trading platforms — ICICI Direct is the example used — are streaming applications at heart. When you view a stock, the time series graph updates as a function of every second, even milliseconds. The interesting part is the bracket order: you set a price bracket as a threshold, and the platform monitors the streaming price data continuously. When the price enters the bracket, the purchase triggers automatically — no human in the loop. The events of the stock and its value are analyzed as a function of time, and the action is triggered by the analysis performed on this real-time streaming data.

1.9.3 Insurance and BFSI Analytics

Real-world: insurers such as Max Bupa and Sundaram use analytics at two different time scales. Issuing a policy to a prospective customer starts with analyzing your data and asking questions about your past health history. The indicators behind those questions come from batch processing: analyzing voluminous records of many insurance customers to derive risk markers — if a person carries these kinds of risks, the policy is issued or refused accordingly.

The same insurance domain shows the streaming-versus-batch split in miniature. Term plan policies — lifelong plans with monthly premiums — examine health and lifestyle factors like hypertension and diabetes to arrive at a recommended premium. That analysis runs as batch: you run the analytics script daily or every two days, derive trends, and use those trends for future decisions — including pricing riders on existing policies at a competitive price. The risk assessment is batch; the policy pricing is batch; but the same data, analyzed in seconds, would be stream.

1.9.4 The Common Thread

Across all four domains — banking, ride-hailing, stock trading, insurance — the pattern is identical: small events generated continuously, analyzed within a chosen time scale, with an action triggered by the analysis. The only variable is the window: seconds for fraud and stock prices, minutes for traffic, hours or days for insurance risk trends. That is why the time-scale rule from Section 1.6 is the connective tissue of the entire course.

1.10 Industry Relevance: Kafka, Spark, and Databricks

1.10.1 State-of-the-Art Skills

Real-world: a student asked whether the technologies in this course are actually used in industry or whether the course lags behind. The answer is direct: Kafka and Spark are state-of-the-art technologies — the two most important skills in the data engineering job market today. A couple of years back, the course covered all the frameworks with little time left for practice; the current focus is deliberate.

The positioning matters for career planning:

• For a data engineer role, there are two very important skills: Kafka and Spark.
• Spark can be done through Scala or through Python; this course uses the Python flavor of Spark — PySpark — the most happening variant.
• Spark and Databricks are very prominent in industry right now — Databricks is a major platform in production data work.

The other frameworks the course mentions are still real, but less central — which is exactly why they are assigned for exploration rather than taught in depth. You come out with working depth on the two technologies that matter most, and a survey-level understanding of the rest.

1.10.2 Kafka on Your Laptop

A common fear about Kafka is the setup — installation and orchestration feel heavy, and people become apprehensive about picking up the technology because of it. The course removes that barrier. Kafka's free community edition can be installed locally on your laptop, and the instructor shares the industry installation setup file plus the script commands for running producers and consumers. There is a structured way to set up the environment locally, and the course follows it exactly, so you can follow along in parallel on your own machine.

1.10.3 PySpark and Structured Streaming

On the Spark side, the course focuses on two things: Spark structured streaming and the Spark DataFrame API. Structured streaming is the newer addition to this run of the course — if you are familiar with SQL queries, it answers the question "how do we write queries over streaming data?" This is the "structured data" dimension students sometimes ask about: it is Spark SQL applied to streams.

The second half of the course is predominantly hands-on with Spark: creating Spark DataFrames, the analytics aspects of PySpark, and — across two to three lectures — the details of which traditional algorithms carry over to streaming applications. The machine learning algorithms themselves are self-learned for the assignment, while the instructor demonstrates the platform side.

1.10.4 Hadoop Is Optional

Real-world: a student with local Spark experience asked whether Spark streaming needs Hadoop at the backbone. The answer: Hadoop can be used, but it is not a necessary option — there are multiple options. HDFS is one distributed file system mechanism, but the semantics of distributed computing do not depend on it; the course explains the various configuration setups for a cluster, while focusing on how to set up the ecosystem on a laptop locally.

For learning purposes, Hadoop is not needed at all: knowing the concepts and running a proof of concept locally is entirely sufficient. In production, Hadoop could be one of the parallel options — but the focus of this course is how to best use the streaming platforms themselves. If you look at Databricks, it was developed by the creators of Spark and has everything built in — you never bother about installing clusters; the platform takes care of it, and there is a lot of help documentation. There is a cost attached to some workloads, but it is not high, and compared with GCP, AWS, or Azure, Databricks has become more and more prominent because of its features and user-friendliness for Spark analytics.

1.11 Prerequisites and Preparation

1.11.1 Python Programming

The one prerequisite the instructor states outright is Python programming. Backgrounds vary from batch to batch, so if the majority of the class does not have Python experience, there will be a one-hour refresher — one lecture on the programming basics, described as "not so difficult." You do not need to be a Python expert going in; you need to be able to read and write basic Python, and the refresher covers the gap for those who need it.

1.11.2 Maturity Over Tools

No specific technologies are required beforehand. Once you know a couple of programming languages and how an IDE works, installation and setup are not a problem — that is learnable on the spot. The fundamental requirement is programming fluency in any language: Java is fine, Python is the course language, and the only real problem would be entering the course unfamiliar with any programming language at all.

The stated expectation for students: you already carry industry maturity — understanding, context, and the ability to reason about systems. The course expects that maturity, and nothing else is needed.

1.12 Learning Philosophy and Expectations

1.12.1 Assessments Match the Class

The course's operating principle is stated up front and repeated: assessments are perfectly aligned with what is covered in class. The instructor is explicit that students will not face pressure preparing for the exams, because the exam focus is exactly the class content. There is no extra study and no self-study exercise required — the course is designed for working professionals, for whom putting ten hours a week into a course is unrealistic, and the instructor reasons that too many expectations cause students to lose interest.

1.12.2 Closed Book and Open Book Need the Same Skills

The distinction between EC2 (closed book) and EC3 (open book) matters less than students expect. The instructor's position: even the closed-book exams could be given as open-book exams — there is no real difference, because the evaluation is about whether you know the concept, whether you know how to design an architecture, and whether you have learned the skills. It is not about remembering and reproducing. Whether the book is open or closed, the questions test understanding: evaluate this architecture, design an architecture for this business case, write this algorithm for streaming data.

1.12.3 The Daily Revision Habit

The recommended rhythm is deliberately light: every day, spend 10 to 15 minutes on the notes covered in the last class before coming to the next class. That is sufficient — it doubles as exam preparation. The philosophy is that learning should not feel like pressure; the instructor's own analogy: if someone gives you a delivery target of 5 p.m., you deliver, but with the 5 p.m. target in the back of your mind, some pressure is always on. If the same person says "forget about delivery, give it whenever you are comfortable" and just checks the status at 5 p.m., you have progressed more — you know a greater amount — precisely because the pressure was removed. Deadlines that remove pressure improve learning.

1.12.4 Why Attend at All

The question "why should you attend these classes when all content exists on YouTube and Google" gets a direct answer: you cannot manage without attendance. The lectures are customized specifically to this particular handout and curriculum, and — most importantly — they attach a real-time industry perspective to the content. That perspective is something you will not find elsewhere; it is the unique value of the sessions.

The second argument for attendance is structure. A student summed it up: left to oneself, exploration can go anywhere — you can plan, detail, and get lost in exploration. A concrete learning plan guides you, so you cover the right things in the right order. The class is that plan, executed with a real-time perspective attached.

Exam Guidance Summary

• Weightage: Assignment 1 (10) + Assignment 2 (10) + Quiz (10) = EC1, 30 marks. EC2 (mid-semester) = 30 marks, roughly 3–4 questions, closed book. EC3 (comprehensive) = 40 marks, open book. Five-credit course. Total 100 marks.
• Quiz mechanics: two quizzes of five marks each; objective type (true/false); 10 questions in roughly 15–20 minutes depending on complexity; a 3–4 day window (typically Friday through Monday); no makeup; once started, must finish in one sitting.
• Assignment 1: deep exploration of a streaming architecture (Flink, Samza, AWS Kinesis, similar); groups of four (calibrated to class strength); given after the 6th lecture; 3-week window; flexible deadline; 10 marks; submitted as an artifact.
• Assignment 2: machine learning techniques on Spark via PySpark on Databricks community edition; 10 marks; can be submitted with Assignment 1 as a single Jupyter notebook (part one: assignment 1 documentation, part two: assignment 2 code).
• Mid-semester style: theory-based. Expect architecture evaluation ("here is an architecture — evaluate it") and business case to architecture design ("here is a business case — design the architecture"). Covers data systems, streaming architecture principles, and Kafka architecture up to the exam.
• Post-mid-semester style: coding questions; algorithm design and implementation over streaming data; two to three lectures cover which traditional algorithms carry over to streaming.
• Core conceptual rules: the time-scale rule decides stream vs batch (minutes/seconds = stream; hourly/daily = batch); the window mechanics (event-based vs time-based, river-bucket mapping) are classic theory questions; streaming algorithms are probabilistic and non-deterministic because the random sample may not reflect the entire stream.
• No pressure: assessments align exactly with class coverage; no extra study; 10–15 minutes of daily note review between classes is enough preparation. Closed-book and open-book exams test the same skills — understanding and design, not recall.

Key Industry Applications

• Kafka + Spark (PySpark): state-of-the-art skills for data engineer roles; the two most important technologies in the streaming space; Databricks prominent in industry.
• Credit card fraud detection: events of a few kilobytes analyzed in real time by a stream processing engine; machine learning classifies transactions; a swipe above 80,000 rupees triggers a confirmation call (press 1 to confirm, press 2 to raise an issue).
• Ride-hailing (Uber, Ola): cab availability and traffic congestion analyzed in real time for decision making.
• Stock trading (ICICI Direct): time series graphs updating per second or millisecond; bracket orders auto-trigger purchases when the streaming price enters the threshold bracket.
• Insurance (Max Bupa, Sundaram): batch analytics over voluminous customer records derive risk markers for policy issuance; term plan premiums reflect health factors like hypertension and diabetes; rider pricing set competitively from daily or two-day batch trends.
• Time-scale rule: minutes or seconds → stream processing; hourly, 8-hourly, or daily → batch processing; Kafka and Spark support customizable batch sizes (micro-batch, stream, or scheduled batch), so one architecture serves both regimes.
• The common pattern: across banking, ride-hailing, trading, and insurance — small events generated continuously, analyzed within a chosen time scale, with an action triggered by the analysis; only the window (seconds for fraud and stock prices, minutes for traffic, hours or days for insurance trends) varies.
• Kafka community edition: free local installation on a laptop with shared setup files and producer/consumer script commands — no cloud needed to learn.
• Databricks community edition: free cloud platform from the creators of Spark; AWS, Azure, or GCP flavor chosen at registration; handles cluster setup entirely.