Exam Review and Preparation Guidance

15.1 Exam Blueprint: The Three Sample Questions

The session opened as a review: a walk through the sample questions before the paper itself was shared. The three questions form the exam blueprint — one on Spark Structured Streaming, one on the Count-Min Sketch, and one on Spark transformations and actions. The message that runs through all three: the exam is problem-based, so know how to do each technique, not just how to talk about it.

15.1.1 Question 1 — Spark Structured Streaming and the Timing Diagram

The first sample question is built around Spark Structured Streaming. It expects you to recall what was covered in class: the Spark Structured Streaming model and the timing diagram.

A timing diagram shows how the streaming engine processes micro-batches over time. It is a picture of the streaming loop, and every exam answer on it should walk the same four steps:

1. Arrival — data arrives from the source (a socket, Kafka, a file directory) as a continuous stream of records.
2. Batching — the engine collects everything that arrived since the last interval into a small batch, called a micro-batch.
3. Processing — the micro-batch runs through the same query you would write for batch data (filters, groupings, aggregations).
4. Emission — the result of that batch is written out, the next interval starts, and the loop repeats.

The question will ask you to draw or explain that diagram: what happens at each step of the streaming pipeline. Keep the loop in order — arrival, grouping into a batch, processing, emission — because the marks sit on the sequence, not on decoration.

A solution to this question is provided alongside the sample paper, so there is no need to worry about whether your version is right. The guidance given was direct: refer to the class notes on Spark Structured Streaming, and the timing diagram exactly as it was explained in class.

15.1.2 Question 2 — Count-Min Sketch Table Construction

The second sample question is an algorithm problem on the Count-Min Sketch. The name of the algorithm was also rendered as "quantum mean sketch" and "Count Moon sketch" in places; those are mishearings of the spoken name — the technique taught in class is the Count-Min Sketch, and that is the name to write in the exam.

You should be constructing the Count-Min Sketch table: given a stream of items and a set of hash functions, fill in the sketch table, row by row, and use it to answer frequency-estimation questions. This is a hands-on problem: you are given the items, the hash functions, and an empty table, and you must produce the completed table plus the estimated counts.

The one caveat emphasized in class: when you apply hash functions, sometimes the hash value can be zero. The exam will state explicitly whether the table's index starts from zero or from one — whether to resume from one or start indices from zero — so read that convention from the question itself and apply it consistently. Section 15.5 covers this question in full detail, including a worked table under both conventions.

15.1.3 Question 3 — Spark Transformations, Actions, and the Lifecycle

The third sample question is about Spark transformations and actions, and the entire Spark lifecycle related to them.

A Spark program builds a lineage of transformations — nothing executes at the moment you call a transformation — and only when an action is invoked does the engine actually evaluate the computation. The lifecycle question ties these together: from loading the data, through the DAG of transformations, to the action that triggers job execution.

The order matters more than it looks: the transformation stage is nearly free (the engine just records the operation), while the action stage is where every job's real cost lands. Section 15.4 goes deeper into the lazy behavior this depends on.

15.1.4 Format Facts: Open Book, Numerical, Balanced

The exam is open book. When asked directly, the answer was confirmed: yes, it is open book only.

The student who had previewed the model sample paper observed that the exam looked largely numerical this time and asked whether that was the case. It is: the exam is going to be more numerical this time, but that does not mean there are no theory questions — it will be balanced. Numerical problems will carry most of the weight, with conceptual questions sitting alongside them.

That combination — open book, numerical, balanced, problem-only — sets the study direction for everything that follows: learn to perform the techniques, and carry the concept that sits under each one.

15.2 Where the Weightage Sits: 30% Pre-Mid-Sem, 70% Post-Mid-Sem

15.2.1 What the Mid-Sem Already Tested

The course split into two major halves, and the exam weightage follows that split: roughly 30% pre-mid-semester content and 70% post-mid-semester content. That split is the single most useful number for planning revision — it tells you where marks live, so it tells you where hours should go.

The mid-sem already tested the architecture-side items: message delivery semantics, when to use which architecture, and the differences between batch processing and stream processing. These were the architecture topics, and they are now checked off — the exam will not re-ask them at depth.

One student was confused about message writing mechanisms — the instructor clarified that message writing was among the things discussed in the mid-sem: there are certain aspects, like writing directly or writing indirectly, that were covered then. Some parts can be ignored; just go through the topics listed and it will come back.

The distinction the student was tripping on: message delivery semantics (the guarantees about whether a message arrives, such as at-least-once or at-most-once delivery) and message writing mechanisms (how a producer actually writes messages into the system — directly, or indirectly through an intermediate component) are two different topics. Both were covered before the mid-sem. If either one feels foggy, a quick pass over the pre-mid-sem list is enough to bring it back — this is 30% material, not the place to invest deep revision.

15.2.2 Where to Spend Your Time Now

The architecture aspect was already evaluated in the mid-sem, so do not spend too much time on it — except a few topics. Do not develop too much into pre-mid-sem content: generalized streaming architecture, Lambda architecture, Kappa architecture, message delivery semantics, and how data gets into the system. On those, only look at the basics — the differences between batch processing, stream processing, and real-time processing.

The reason for stating this is direction: "You should know where to focus." The post-mid-sem portion — algorithms and structured streaming — is where the weight sits, and the sample questions reflect that: two of the three sample questions are streaming-related.

15.2.3 The "Dirty Frog" Study Plan

The analogy is a sequencing rule for revision. Picture your topic list as a plate of frogs: some look ugly and taste worse, some are fine. Everyone naturally picks the fine ones first and pushes the ugly one aside — and then the ugly one is all that is left when energy runs out. The professor's rule inverts that instinct: eat the dirty frog first. The "dirty frog" here is the most important topic that feels hardest — the algorithms and structured streaming material — so you face it at your freshest and feel confident early.

Mapping it to this course:

Other topics are not unimportant, but the highlighted ones — the algorithms and structured streaming — should come first so you feel confident early. The extra special classes exist precisely so the techniques can be discussed in detail instead of being rushed in the last class, when people get nervous.

Where the analogy breaks: a frog stays ugly no matter when you eat it. Revision topics do not — with the post-mid-sem material done first, the remaining architecture topics become lighter, so the plan is not about punishment, it is about ordering effort while energy is high.

15.3 The Course at a Glance: Architecture and Processing

The course was divided into two major aspects: architecture and processing. Keeping that split in mind is useful because the weightage split follows it — architecture was mostly tested in the mid-sem, processing (algorithms and structured streaming) carries the post-mid-sem exam.

15.3.1 The Architecture Side

Under architecture, the class covered:

• batch processing, stream processing, and real-time processing, and the differences between them;
• data: the sources of data, and after data ingestion, the common interaction patterns;
• the components of the Lambda architecture, the Kappa architecture, and the generalized architecture;
• message delivery semantics and the importance of long-term storage;
• the message-routing mechanisms (the recording rendered this as "message-rating"; the surrounding discussion of message delivery and long-term storage points to routing or ordering as the intended topic).

The message-routing item is the only one whose phrasing is unclear in the recording; the surrounding discussion is about message delivery and long-term storage, so routing or ordering is the likely intent. If the sample paper or your class notes use a different name for this item, follow those.

15.3.2 The Processing Side

Under processing, the class worked with PySpark:

• the difference between RDD and Structured Streaming;
• various clauses, like group by and where — how you group records, the same way SQL statements group them;
• the word count problem, and the streaming version of word count;
• setting up Kafka path integration;
• as part of the assignment, how to run machine learning experiments in PySpark;
• streaming algorithms.

Another student was confused about Spark's place in the stack — Spark resides in the processing layer. The answer pointed to the links shared from the Spark documentation in earlier classes; refer to those.

15.3.3 Streaming Algorithms Covered

Under streaming algorithms, a few are important: the Bloom filter, the Count-Min Sketch, reservoir sampling, and the decaying window. These algorithms are the core machinery of stream analytics — approximate membership testing (Bloom filter), approximate frequency counting (Count-Min Sketch), unbiased random sampling from a stream (reservoir sampling), and recency-weighted aggregation (decaying window).

Each one answers a different question about a stream, so it helps to keep them separated by question type:

Real-world: the same ideas power production stream-processing pipelines — Bloom filters in database and cache layers, Count-Min sketches in network monitoring and analytics engines, reservoir sampling wherever a fixed-size random sample of an unbounded stream is needed.

15.4 Lazy Evaluation in Spark

15.4.1 The Question and the Answer

A student asked a question that reveals a common gap: the class covered wide and narrow transformations, but the student did not remember whether lazy transformation was studied. The answer connected the two ideas: lazy transformation is nothing but the wide transformation — the evaluation is delayed until the point where it has to be evaluated. Lazy means expression evaluation is delayed till the point where it is needed.

The point of the correction: the student had learned the idea without attaching the name. The name "lazy transformation" was not a separate topic to remember — it is the same deferred-evaluation behavior, and in this course the term is tied to the wide transformation. Once the name is connected to the behavior, the topic stops looking like a gap.

15.4.2 Lazy Versus Wide: What the Terms Mean

Two ideas are being joined here. Wide transformations are transformations that need data from other partitions — shuffles — like groupByKey, join, or distinct. Narrow transformations work on data within one partition — like map or filter — and can be executed without shuffling.

Laziness is a property of the whole Spark lifecycle: building a transformation only records the operation in the lineage; no computation happens until an action forces it. Wide transformations in particular are postponed because they are expensive — the engine waits, collects the lineage, and then evaluates. So "lazy" and "wide" describe different axes, but in the way this course treats them for the exam, the lazy transformation is the wide transformation: the point being that its evaluation is deferred until it is needed.

The two axes, made explicit: narrow vs wide says where the data comes from (same partition or other partitions). Lazy vs eager says when the computation runs (never at call time, only when an action demands it). The professor's exam shorthand collapses these: the lazy transformation is the wide transformation, because both descriptions single out the operations whose evaluation is deferred until they are truly needed.

That single idea is the backbone of the Spark lifecycle question in Section 15.1.3: transformations build a plan, actions trigger execution, and laziness is why building a transformation is cheap but evaluating it is where the work happens.

15.5 The Count-Min Sketch Question in Detail

15.5.1 How the Table Is Built

The Count-Min Sketch answers one question about a stream: how many times has this item appeared? It trades exactness for memory — a small two-dimensional table instead of a full count per item.

Construction: the sketch is a table $C$ of $d$ rows and $w$ columns. Each row $j$ gets its own hash function $h_j$, mapping an item to a column of that row. For every incoming item $x$, you hash it with every hash function and add one to the cell it lands in. Because different items can collide into the same cell, the counts are over-estimates — a cell may hold the counts of several colliding items.

The steps in exam order:

1. Draw the table: $d$ rows (one per hash function $h_j$) and $w$ columns, every cell started at 0.
2. For each incoming item $x$, compute $h_1(x),h_2(x),\dots ,h_d(x)$ — one column per row.
3. Add 1 to each of those $d$ cells, one per row.
4. When asked for the frequency of $x$, hash again with all $d$ functions and take the smallest of the $d$ cell values.

The verbal description of the table in this session: you should be constructing this Count-Min Sketch table. Only thing is that — when you do hash functions, sometimes the hash value might be zero.

15.5.2 The Hash-Value-Zero Convention

This is the specific trap the exam flags: a hash function can output zero. The point was made twice for emphasis: "Sometimes the hash value might be zero. The hash value is zero. We will tell that."

The exam question will tell you which convention applies — whether to resume from one, or whether the start indices start from zero. In other words: if the hash value is 0, does that mean column index 1, or column index 0? Apply the stated convention consistently across the whole table, because a single off-by-one error cascades through every row.

15.5.3 The Update and Estimate Rules

With $C$ the sketch table, $d$ the number of hash functions (rows), $w$ the number of columns, and $h_j$ the hash function of row $j$, the update for an incoming item $x$ is:

$$C[j][h_j(x)]\leftarrow C[j][h_j(x)]+1\text{for }j=0,1,\dots ,d-1$$

where $x$ is the item, $j$ indexes the row, and $h_j(x)$ is the column that $x$ maps to under row $j$'s hash function. This is the standard form: hash the item with every row's hash function and increment each target cell by 1, exactly as the class's table construction describes.

The frequency estimate for an item $x$ takes the minimum over the rows — the cell with the least count bounds the true count from above:

$${\hat{f}}_x=\underset{j}{\min }C[j][h_j(x)]$$

where ${\hat{f}}_x$ is the estimated frequency of $x$, and the minimum over rows is what makes collisions hurt least: the true count is at most this value, and no single collision inflates every row.

The reason the minimum works: every row gives a count that is at least the true frequency, and errors come from collisions with other items. The smallest count across rows is the closest to the truth. Concretely, a row can only gain from collisions — each colliding item adds 1 to a shared cell — so row $j$'s value is the true count plus the collision noise of that row. The minimum picks the row with the least noise.

The standard guarantee bounds how bad the overcount can be. If the sketch has width $w=2/\epsilon$ and depth $d=\log (1/\delta )/\log 2$, then with probability at least $1-\delta$ the estimate for an item whose true count is $f_x$ falls between $f_x$ and $f_x+\epsilon N$, where $N$ is the total number of items entered. Two practical sizes from the course references: width 40 with depth 7 keeps the estimate within 5% of the total with 99% probability; depth 8 with width 128 gives about 1.5% relative error at about 99.6% probability. That is why a small fixed table can stand in for unbounded per-item counters: memory stays $O(dw)$ and each update or query costs $O(d)$ hash computations, no matter how long the stream runs.

15.6 Structured Streaming: The Word Count Problem and Window Concepts

15.6.1 Word Count in Structured Streaming

Structured streaming was covered in detail in the course — the word count problem being the canonical example. The streaming word count mirrors the batch word count: read a stream of text, split it into words, and aggregate counts over the arriving data. The difference is that the aggregation happens over micro-batches as data streams in, not once over a static file.

The emphasis was explicit: focus more on algorithms and structured streaming, because structured streaming was done in detail — like the word count problem and other things; for the class notes, just refer to them.

Algorithms are the core part of structured streaming. You may wonder why the course did not start with algorithms if they are so central. The reason given: unless you have a formal understanding of these techniques, discussing just the algorithms becomes a little dry. So the course laid the foundation first, then built the algorithms on top of it.

15.6.2 Group By and Where Clauses

Within PySpark, the class looked at various clauses, including group by and where. A student asked about the group by clause — the instructor confirmed it was discussed: it is how you group records, like SQL statements group them.

Group by splits the data into groups by key so that aggregations (counts, sums) apply per group; where filters rows. In structured streaming both clauses work on the streaming DataFrame the same way they do in batch — the engine turns the query into the same plan, then executes it incrementally over micro-batches.

The SQL connection is the fastest way to stay oriented: any query you can write for a static table, you can write for the streaming DataFrame — the engine just runs it repeatedly, once per micro-batch.

15.6.3 Window Concepts and the Spark Documentation

Window concepts were explained using the Spark documentation: links were shared while explaining windowing — how streaming aggregations are bucketed over time windows (tumbling or sliding), which is how the timing diagram in the sample question ties to real behavior.

Windowed aggregation answers a timing question that word count alone cannot: "how many occurrences fell inside this time bucket?" Two window shapes matter:

• Tumbling window — fixed-size, non-overlapping buckets. A 1-minute tumbling window yields bucket 0–1 min, 1–2 min, 2–3 min; each event belongs to exactly one bucket.
• Sliding window — fixed-size buckets that overlap. With a 10-minute window sliding every 5 minutes, an event at 12:00 belongs to the 11:50–12:00 bucket and the 11:55–12:05 bucket.

The timing diagram question is the visual of this machinery: events arriving on the left, micro-batches formed at each interval, the windowed query applied, and results emitted on the right — then the next interval begins.

Real-world: windowing is what makes streaming analytics usable in production — dashboards and alerting systems aggregate over time windows rather than per event, exactly the pattern the Spark documentation shows.

15.7 Answer Strategy and Exam-Day Advice

15.7.1 Be Specific, Not Verbose

The single most direct piece of exam technique: don't write some verbose thing — be specific on what is expected, and that will save everybody's time mutually. Answer exactly what the question asks, in the format it expects: a constructed table, a drawn timing diagram, a lifecycle walk-through. Extra prose does not earn extra marks in an open-book, problem-heavy exam.

15.7.2 Regular Versus Makeup

Asked about the makeup exam, the advice was to prefer the regular exam — especially for this course. The student confirmed the regular slot, as in the mid-sem.

15.7.3 Practical Preparation

The sample paper was shared on the course channel during the session — check there if you missed it.

The sample questions come with solutions, and if you find anything you would change in a solution, you can raise it. The solutions are a working reference, not a secret answer key — questioning them is part of understanding.

On preparation overall: spend enough time on the class notes, and focus on the topics that were highlighted. It is not that other topics are unimportant — it is a question of direction and of where the weight sits (Section 15.2). Also, if any area still feels weak — Spark itself, for example — go back to the documentation links shared in class rather than searching the internet blind.

15.8 The Approach Behind the Course

15.8.1 Why Orchestration Fear Is the Real Barrier

The reasoning behind how the course was taught: most working professionals get afraid and frightened when they look at the installation and orchestration side of these systems — Kafka, Zookeeper, and related tools. That fear blocks learning before the concepts even start. So the course was shaped to minimize that complexity: everything was demonstrated on a laptop with simple steps, packaged as a final set of steps in a text file — follow the file and it works without referring to the internet for anything.

Real-world: Kafka's path integration (streaming data into Spark) and Zookeeper's coordination role are exactly the pieces people find intimidating; the course's message is that these tools are complex but there is a way to learn and a way to teach them.

The instructor also admitted to struggling in the beginning: "I also struggled like you guys in the beginning. Then I realized, why can't I make a document which can be used for everybody." The point extends beyond the course — the mechanisms learned here can be reused for any other framework later.

15.8.2 Learning by Exploration: The Assignment's Intent

The assignment — running machine learning experiments in PySpark — was designed for exploration, not just evaluation. All the resources are available on the internet, but exploring them and running your own experiment is itself the learning. "If people are not serious about the quality of the assignment, then I cannot help — it is a platform to learn." Marks will be there for the exploration. One month was given so there is enough time to do it leisurely, because after this course nobody will hand over this kind of platform for learning streaming machine learning. "Once you know it, it is very simple. But if you don't know, it will become a bottleneck."

15.8.3 Known Unknowns and Unknown Unknowns

The distinction about finding things out: finding an unknown known is easy — the unknown is unknown to you but known to somebody else, so it can be found. Finding an unknown unknown is different — keep digging. The course design tries to spare students the sleepless nights spent hunting unknown unknowns and brainstorming without direction. The intention: people should feel that they are learning. Giving marks is not a problem — the purpose is learning.

15.8.4 A Five-Credit Course Is About Your Own Time

The course is a five-credit course, and the point about what that means: apart from the lecture classes, how much time you spend on your own is the main intention behind it — that is why it carries heavy weightage. Choose the course because you have an aspiration to learn, but it should not happen at the cost of sleepless nights: be leisurely, do what is needed, without wasting time on brainstorming and finding unknown unknowns.

This session itself was one of the extra special classes — held to discuss the techniques in detail instead of rushing everything at the end, when people get nervous and the last class becomes a panic.

Exam Guidance Summary

Weightage and focus

• Weightage: roughly 30% pre-mid-semester, 70% post-mid-semester.
• Already tested in the mid-sem: message delivery semantics, when to use which architecture, batch vs stream processing differences, message writing mechanisms (direct vs indirect write).
• Post-mid-sem focus: algorithms (Bloom filter, Count-Min Sketch, reservoir sampling, decaying window) and structured streaming (word count problem, timing diagram). Class notes are the reference.
• Pre-mid-sem revision should be shallow: only the basics of generalized/Lambda/Kappa architectures, message delivery semantics, how data enters the system, and batch vs stream vs real-time differences.

The three sample questions

• Three sample questions: (1) Spark Structured Streaming + timing diagram, (2) Count-Min Sketch table construction, (3) Spark transformations and actions lifecycle. Solutions are provided with the sample paper; the number of pages can be ignored.
• Count-Min Sketch convention: hash values can be zero; the question will state whether indices start from zero or from one — apply it consistently.
• Window concepts: refer to the Spark documentation links shared in class.

Format and technique

• Format: open book; mostly numerical problems; no theory or derivation answers even though derivations were taught; conceptual questions sit alongside numericals; balanced overall.
• Answer technique: be specific, not verbose; construct the expected artifact (table, diagram, lifecycle).
• Regular exam preferred over makeup, especially for this course.

Key Industry Applications

• Real-world: Apache Spark Structured Streaming — production stream processing over micro-batches; the timing diagram pattern is how streaming engines actually schedule and emit results.
• Real-world: Apache Kafka path integration with PySpark — streaming data into a processing pipeline; Zookeeper handles coordination. Installation and orchestration are the practical barrier that the course's step-by-step approach removes.
• Real-world: streaming algorithms — Bloom filter (approximate membership, used in databases and caches), Count-Min Sketch (approximate frequency counts, used in network monitoring and analytics engines), reservoir sampling (fixed-size unbiased random sample of an unbounded stream), decaying window (recency-weighted aggregation for dashboards and trending).
• Real-world: running machine learning experiments in PySpark — the assignment pattern: exploration of real frameworks is the intended learning outcome, and the same mechanisms transfer to any other framework later.
• Real-world: windowing in the Spark documentation — time-based aggregation is the standard production pattern for dashboards and alerting.